Image processing method, image processing apparatus, electronic device, and computer program

Abstract

This application provides an image processing method, apparatus, electronic device, and computer-readable storage medium, the method comprising the steps of: acquiring a pending presentation word; acquiring text features of the pending presentation word and mapping the text features to a generation probability index and description type of the pending presentation word, wherein, in response to the generation probability index being greater than an index threshold, the description type indicating that the pending presentation word does not contain a verb, and the pending presentation word contains multiple clauses, acquiring similar images corresponding to each of the multiple clauses, determining the degree of image difference between the similar images corresponding to each of the multiple clauses; and, in response to the degree of image difference being less than an image difference threshold, using the similar images corresponding to each of the multiple clauses as illustrations for the corresponding clauses.

Description

[Technical Field]

[0001] (Cross-reference to related applications) This application claims priority to the Chinese patent application filed with the China National Patent Office on April 4, 2023, with application number 202310399237.0, the entirety of which is incorporated into this application by reference.

[0002] This application relates to image processing technology, and more particularly to image processing methods, image processing apparatuses, electronic devices, and computer-readable storage media. [Background technology]

[0003] In related technologies, in text-based image generation tasks, the user provides text for a desired description, such as a story text or a martial arts novel text, as a prompt (also called a "prompt"). Based on the prompt, an image generation model is invoked to generate multiple corresponding story images, which are then used as illustrations (also called "illustrations") for the prompt. However, this method of directly generating images based on prompts often results in less effective illustrations.

[0004] On the one hand, the descriptions of suggested words entered by the user may be abstract. For example, the described content may be diverse and complex actions, and it is difficult to perfectly represent these actions in an image, resulting in a low degree of matching between the generated image and the suggested word. On the other hand, suggested words given by the user may contain many clauses, and the elements and content described in these clauses may also differ. Therefore, even with the same suggested word containing multiple clauses, it is possible to generate multiple generated images with completely different content. The differences between these generated images are large, and there is also a possibility that generated elements may be missing, making them unsuitable as illustrations for suggested words and affecting the correlation of the overall generation effect of the generated images. [Overview of the Initiative]

[0005] Embodiments of the present application provide an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product that improve the overall correlation between images and text by identifying the possibility of generating presented words in a scene where images are generated from text.

[0006] The technical solution of the embodiment of the present application is realized as follows.

[0007] Embodiments of the present application provide an image processing method performed by an electronic device, the method being Steps to obtain a word to be processed, A step of obtaining the text features of the pending suggestion word, and mapping the text features to a generative probability index and description type of the pending suggestion word, wherein the generative probability index is used to represent a score that the pending suggestion word can be used to generate an illustration. Steps to obtain similar images corresponding to each of the multiple clauses in response to the fact that the generative probability index is greater than an index threshold, the description type indicates that the pending presentation word does not contain a verb, and the pending presentation word contains multiple clauses, wherein the image-text similarity between the clause and the corresponding similar image is greater than an image-text similarity threshold, The steps include determining the degree of image difference between similar images corresponding to each of the aforementioned multiple sections, The process includes the step of, in response to the image difference being smaller than the image difference threshold, setting a similar image corresponding to each of the plurality of sections as an illustration for the corresponding section.

[0008] Embodiments of the present application provide an image processing apparatus comprising an acquisition module, a mapping module, and a determination module. The acquisition module is configured to acquire a pending word, The mapping module is configured to acquire the text features of the pending presented word and to map the text features to the generative possibility index and description type of the pending presented word, the generative possibility index being used to represent a score that the pending presented word can be used to generate an illustration. The acquisition module is further configured to acquire similar images corresponding to each of the multiple clauses in response to the fact that the generative possibility index is greater than the index threshold, the description type indicates that the pending presentation word does not contain a verb, and the pending presentation word contains multiple clauses, and the image-text similarity between the clause and the corresponding similar image is greater than the image-text similarity threshold, The decision module is configured to determine the degree of image difference between similar images corresponding to each of the plurality of sections, The decision module is further configured to use similar images corresponding to each of the multiple sections as illustrations for the corresponding sections in response to the image difference being smaller than the image difference threshold.

[0009] Embodiments of the present application provide an electronic device, said electronic device, Memory for storing computer-executable instructions or computer programs, The present invention comprises a processor that executes the image processing method provided in the embodiment of the present invention when executing the computer-executable instructions or computer programs stored in the memory.

[0010] Embodiments of the present application provide a computer-readable storage medium storing computer-executable instructions or computer programs, wherein, when the computer-executable instructions or computer programs are executed by a processor, the processor causes the processor to execute the image processing method provided in the embodiment of the present application.

[0011] An embodiment of the present application provides a computer program product including computer-executable instructions or a computer program, and when the computer-executable instructions or the computer program are executed by a processor, the processor is caused to execute the image processing method provided in the embodiment of the present application.

[0012] The embodiment of the present application has the following beneficial effects.

[0013] By determining a generation possibility index of a prompt, which is a score that can be used for generating an illustration, and making a determination by combining the generation possibility index and the description type of the prompt, it is possible to identify whether the prompt contains a verb and whether the prompt is suitable for generating an illustration, and perform related processing on the prompt based on the identification result. By checking the prompt from dimensions such as generation possibility and description type, it is possible to avoid a decrease in the quality of the generated illustration due to the fact that the prompt does not meet the processing conditions for generating an image from text. By determining the similar images of the segments in the prompt respectively and evaluating the image difference degree of the optimal similar image of each segment, the illustrations of multiple segments of the prompt are selected. Based on the image selection of the prompt, a more targeted and detailed image selection for the segments of the prompt is realized, thereby improving the overall correlation of the generated image corresponding to the prompt.

Brief Description of the Drawings

[0014] [Figure 1] It is a schematic diagram showing the configuration of the architecture of an image processing system according to an embodiment of the present application. [Figure 2] It is a schematic diagram showing the configuration of an electronic device according to an embodiment of the present application. [Figure 3A] ~ [Figure 3K] It is a flowchart of an image processing method according to an embodiment of the present application. [Figure 4] It is a schematic diagram of the principle of a closed-loop system for continuous illustration generation according to an embodiment of the present application. [Figure 5] It is a schematic diagram of generating an image of an original phrase according to an embodiment of the present application. [Figure 6] It is a sequence diagram of images secondarily generated after phrase segmentation according to an embodiment of the present application. [Figure 7] It is a schematic diagram of an original generated image sequence according to an embodiment of the present application. [Figure 8] It is a schematic diagram of generated images related to a sequence according to an embodiment of the present application. [Figure 9] It is a diagram showing an example of an image generation model architecture according to an embodiment of the present application. [Figure 10] It is a diagram showing the process of evaluating the correlation of generated images according to an embodiment of the present application. [Figure 11] It is a schematic diagram of the principle of performing sequence correlation evaluation on generated images according to an embodiment of the present application. [Figure 12] It is a diagram showing the process of identifying inappropriate phrases as generation results according to an embodiment of the present application. [Figure 13] It is a diagram showing an example of determining the difference of generated images according to an embodiment of the present application.

Embodiments for Carrying out the Invention

[0015] To make the object, technical solution and advantages of the present application clearer, hereinafter, the present application will be described in more detail with reference to the drawings. The described embodiments do not limit the present application, and all other embodiments that can be obtained without creative efforts by those skilled in the art are included in the protection scope of the present application.

[0016] In the following, the description of "some embodiments" describes a subset of all possible embodiments. Understandably, "some embodiments" can be the same subset or a different subset of all possible embodiments and can be combined with each other without contradiction.

[0017] Understandably, the embodiments of this application involve user data, and if the embodiments of this application apply to a specific product or technology, user permission or consent is required, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0018] The terms "first / second / third" and so on in the specification of this application do not limit a specific order, but rather distinguish similar objects. It is understandable that, since "first / second / third" can be changed in a specific order or sequence where appropriate, the embodiments of this application described herein may be performed in an order other than that illustrated or described herein.

[0019] In the embodiments of this application, the terms “module” or “unit” refer to a computer program or part of a computer program having a predetermined function, which operates together with other relevant parts to achieve a predetermined objective, and which can be achieved in whole or in part by using software, hardware (e.g., processing circuits or memory), or a combination thereof. Similarly, one or more modules or units can be realized using one processor (or more processors or memory). Furthermore, each module or unit may be part of an overall module or unit that includes the function of that module or unit.

[0020] It should be noted that, when applied to a specific case, the collection and processing of relevant data in this application (e.g., obtaining pending terms) should strictly comply with the requirements of the relevant national laws and regulations, obtain informed consent or individual consent from the data subject, and the subsequent use and processing of the data should be carried out within the scope of the laws, regulations, and authorizations of the data subject.

[0021] Unless otherwise specified, all technical and scientific terms used in the embodiments of this application have the same meaning as those commonly understood by those skilled in the art. The terms used in the embodiments of this application are adopted solely for the purpose of describing the embodiments and are not intended to limit this application.

[0022] Before describing the embodiments of this application in detail, the nouns and terms used in the embodiments of this application will be explained below.

[0023] 1) Single Image Generation and Sorting: For any given word, an image generation model is invoked to generate multiple generated images for that word. These multiple generated images are then evaluated to determine evaluation metrics for each generated image. The generated images are then sorted based on these evaluation metrics to select the optimal generated image that matches the given word.

[0024] 2) Sequence Generation and Sorting: For multiple presented words, single image generation and sorting is performed on the first presented word to obtain the optimal generated image. Based on the optimal generated image for the first presented word, the generated images of subsequent non-first presented words are sorted again to obtain their corresponding optimal generated images. This process is repeated until all presented words have corresponding optimal generated images, and finally a sequence diagram of the multiple presented words is obtained.

[0025] Embodiments of the present application provide an image processing method, an image processing apparatus, an electronic device, a computer-readable storage medium, and a computer program product that improve the overall correlation between images and text by identifying the possibility of generating presented words in a scene where images are generated from text.

[0026] Referring to Figure 1, Figure 1 is a schematic diagram showing the architecture of the image processing system 100 provided in an embodiment of the present application, which includes terminals (exemplifying terminals 500-1 and 500-2), a network 300, and a server 200, wherein the terminals (exemplifying terminals 500-1 and 500-2) are connected to the server 200 via the network 300, and the network 300 may be a wide area network, a local area network, or a combination thereof.

[0027] Terminals (examples include terminals 500-1 and 500-2) run applications (APPs) for various document editing application scenarios, such as instant messaging APPs, reading APPs, video APPs, document editors, or other software programs with document editing capabilities. When a user inputs suggested words that need to be used to generate images for a document being edited on the APP, the terminals receive the input suggested words and send them to the server 200 via the network 300. The server 200 receives the suggested words sent from the terminals, first obtains multiple generated images of these suggested words, determines the illustration for the suggested words from the multiple generated images, then extracts the text features of the suggested words and maps them to a generative probability index and description type. If the generative possibility index is greater than the index threshold, and the description type indicates that the presented word does not contain a verb and contains multiple clauses, then similar images corresponding to each of the multiple clauses are obtained, where the image-text similarity is greater than the similarity threshold. Next, the image difference of these similar images is determined, and similar images with an image difference less than the image difference threshold are used as illustrations for the clauses. Finally, the illustrations for the presented word and the illustrations for the multiple clauses are sent back to the terminals (terminals 500-1 and 500-2 are examples) via network 300 and inserted into the document currently being edited.

[0028] In some embodiments, a user inputs suggested words into a document editing app on the device, and the device can directly process these input suggested words. First, it obtains multiple generated images of these suggested words and determines the illustration for the suggested word from the multiple generated images. Next, it extracts the text features of the suggested words and maps them to a generative possibility index and description type. If the generative possibility index is greater than the index threshold and the description type indicates that the suggested word does not contain a verb and contains multiple clauses, it obtains similar images for each of the clauses where the image-text similarity is greater than the similarity threshold. Next, it determines the image difference of these similar images, and selects similar images with an image difference less than the image difference threshold as the illustration for the clauses. Finally, it displays the illustrations for the suggested words and clauses directly in the app's document editing interface.

[0029] For example, in a scene where several novels or screenplays are being edited, document editor software running on a terminal receives novel phrases from the user in real time, which need to have corresponding images generated. Then, based on the novel phrases, it rapidly generates a large number of generated images to obtain illustrations for the presented words. After that, it processes the presented words to determine similar images of multiple clauses containing the presented words as clause illustrations, and finally, it returns the illustrations for the presented words and clauses to the illustration interface of the document editor software.

[0030] In some embodiments, the server 200 shown in Figure 1 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), big data, and artificial intelligence platforms. The terminals shown in Figure 1 (exemplifying terminals 500-1 and 500-2) may be, but are not limited to, smartphones, tablet computers, notebook computers, desktop computers, smart speakers, smartwatches, smart TVs, or in-vehicle terminals. The terminals and servers may be connected directly or indirectly via wired or wireless communication, and the embodiments of this application are not limited to these.

[0031] The embodiments of this application are achievable through artificial intelligence (AI) technology, which is a theory, method, technique, and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve the best results. In other words, artificial intelligence is a comprehensive field of computer science that seeks to understand the nature of intelligence and to produce new intelligent machines that can react in a manner similar to human intelligence. Artificial intelligence involves studying the design principles and implementation methods of various intelligent machines so that they have the functions of perception, reasoning, and decision-making.

[0032] Taking the server provided in the embodiment of this application as an example, for instance, a server cluster can be deployed in the cloud to open up artificial intelligence cloud services (AI as a Service: AiaaS) to users or developers. The AIaaS platform divides several common AI services and provides them as independent or packaged services in the cloud. This service model is like opening an AI theme mall, where all users or developers can access and use one or more artificial intelligence services provided by the AIaaS platform through an application programming interface.

[0033] For example, a cloud server encapsulates the image processing program provided in the embodiment of the present invention. The user invokes the image processing service within the cloud service via a terminal (which runs apps such as instant messaging apps and reading apps), thereby invoking the image processing program encapsulated on the server deployed in the cloud and receiving text input from the user. First, multiple generated images of this text are obtained, and illustrations for the input text are determined from the multiple generated images. Next, text features of the text are extracted and mapped to the generative possibility index and description type of the presented word. If the generative possibility index is greater than the index threshold and the description type indicates that the presented word does not contain a verb and contains multiple clauses, similar images corresponding to each of the clauses are obtained, the image-text similarity is greater than the similarity threshold. Next, the image difference of these similar images is determined, and similar images with an image difference less than the image difference threshold are returned as illustrations for the clauses and inserted into the document editing interface.

[0034] Referring to Figure 2, which is a schematic diagram showing the configuration of an electronic device 400 according to an embodiment of the present application, the electronic device 400 may be implemented as the server 200 shown in Figure 1, or as the terminals shown in Figure 1 (terminals 500-1 and 500-2 are exemplified). The electronic device 400 shown in Figure 2 includes at least one processor 410, memory 450, and at least one network interface 420. Here, each component within the electronic device 400 is coupled via a bus system 440. Understandably, the bus system 440 is used to enable connection communication between these components. In addition to the data bus, the bus system 440 includes a power bus, a control bus, and a status signal bus. However, for clarity of explanation, various buses are represented as the bus system 440 in Figure 2.

[0035] The processor 410 may be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), a programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, where the general-purpose processor may be a microprocessor or any conventional processor.

[0036] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, and the like. The memory 450 optionally includes one or more storage devices located physically separate from the processor 410.

[0037] The memory 450 may include volatile memory or non-volatile memory, or it may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random-access memory (RAM). The memory 450 described in the embodiments of this application includes any suitable type of memory.

[0038] In some embodiments, the memory 450 may store data to support various operations, as illustrated below, and examples of this data include programs, modules, and data structures, or subsets or supersets thereof.

[0039] Operating System 451 is configured to implement various basic services and handle hardware-based tasks, including system programs such as a framework layer, core library layer, and drive layer, which handle various basic system services and perform hardware-related tasks.

[0040] The network communication module 452 is configured to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB).

[0041] In some embodiments, the apparatus provided in the embodiments of the present invention may be implemented in software. Figure 2 shows an image processing apparatus 453 stored in memory 450, which may be software in the form of a program or plug-in. This apparatus comprises an acquisition module 4531, a determination module 4532, and a mapping module 4533. Since these modules are logical, they can be arbitrarily combined or further divided depending on the function to be implemented. The function of each module will be described later.

[0042] In some embodiments, a terminal or server can implement the image processing method provided in the embodiments of the present application by executing various computer-executable instructions or computer programs. For example, computer-executable instructions may be microprogram-level instructions, machine instructions, or software instructions. Computer programs may be native programs or software modules of an operating system, native applications (APPs), i.e., programs that need to be installed on an operating system to run, or applets that can be embedded in any APP, i.e., programs that can be run simply by downloading them to a browser environment. In summary, the computer-executable instructions may be of any form, and the computer programs may be applications, modules, or plug-ins of any form.

[0043] The image processing method provided in the embodiments of this application will be described with reference to exemplary applications and embodiments of the electronic devices provided in the embodiments of this application.

[0044] Referring to Figure 3A, Figure 3A is a flowchart of an image processing method according to an embodiment of the present application, and the implementing body may be an electronic device. The explanation will be given with reference to the steps shown in Figure 3A.

[0045] In step 101, retrieve the word that is awaiting processing.

[0046] In some embodiments, the pending presentation words may be text entered by a user or a computer program (i.e., an artificial intelligence-based text generation program), and may be phrases from a novel or script, phrases containing multiple clauses, or multiple phrases. These presentation words are used to describe several target themes, which may be several themes with a specific theme or style within the text of a novel or script, such as a martial arts theme, a history theme, a gourmet theme, or a travel theme. Next, an image generation model is invoked to generate multiple generated images corresponding to the pending presentation words, where these generated images are similar to the pending presentation words and are used to describe the same target themes, such as a martial arts theme, a history theme, a gourmet theme, or a travel theme.

[0047] In step 102, the text features of the pending presented words are obtained, and the text features are mapped to the probability index and description type of the pending presented words.

[0048] In the process of selecting an illustration for each pending suggestion in the input text, it is difficult to represent all the content of all the clauses in a pending suggestion in an illustration, as each clause may contain multiple clauses and contain different content. Therefore, further processing can be performed on the pending suggestion in the following way: namely, by performing a generative possibility identification on the pending suggestion, the pending suggestion is divided into multiple clauses, and an illustration corresponding to each clause is generated.

[0049] In some embodiments, referring to Figure 3B, the "obtaining text features of the awaiting presented word" in step 102 shown in Figure 3A can be achieved by the following steps 1021A to 1022A, which will be explained in detail below.

[0050] In step 1021A, the pending word is converted into a mark sequence.

[0051] In some embodiments, performing generative possibility identification on a pending presented word requires processing the text features of the pending presented word using a single word generative possibility identification model, which can be achieved by extracting the text features of the pending presented word, calling a semantic understanding model based on the extracted text features of the pending presented word to perform an embedding process on the pending presented word, and obtaining an embedding mark sequence corresponding to the pending presented word. Here, the semantic understanding model may be a Bidirectional Encoder Representation from Transformers (BERT) model.

[0052] In step 1022A, a semantic understanding model is invoked based on the mark sequence to perform encoding and obtain the text features of the presented words awaiting processing.

[0053] Continuing from the above embodiment, after obtaining the embedding mark sequence of the awaiting presented word, the semantic understanding model is called to perform encoding on the embedding mark sequence of the awaiting presented word to obtain the text features of the awaiting presented word.

[0054] In some embodiments, referring to Figure 3C, the "mapping text features to the potential generation index and description type of awaiting presented words" in step 102 shown in Figure 3A can be achieved by the following steps 1021B to 1022B, which are described in detail below.

[0055] In step 1021B, the convolutional network within the first text classifier is invoked to perform a convolution operation on the text features to obtain the first convolutional feature, the multi-classification layer within the first text classifier is invoked to map the first convolutional feature to the first probability of multiple candidate generative indicators, and the candidate generative indicator corresponding to the highest first probability is used as the generative indicator for the word awaiting processing.

[0056] After obtaining the text features of the pending presented word, a word generation possibility discrimination model can be called to predict the text features. Here, the word generation possibility discrimination model is used to determine whether the corresponding pending presented word is suitable for image generation by the image generation model, that is, whether the pending presented word is an inappropriate word for image generation. Specifically, it includes two text classifiers, each used to predict whether the description of the generated image corresponding to the pending presented word matches the description of the presented word, and whether the pending presented word is an action description word. The convolutional network of the first text classifier is called to perform a convolution operation on the text features of the pending presented word to obtain the first convolutional feature, where the convolutional network may have multiple convolutional layers, such as two layers. Next, the multi-classification layer of the first text classifier is called to map the first convolutional feature to the first probability of multiple candidate generation possibilities indices, and the candidate generation possibility indice with the largest first probability is used as the generation possibility index for the pending presented word.

[0057] In some embodiments, the candidate generative probability index may be three, for example, 0, 1, and 2, i.e., the first text classifier is a 3-classification prediction. Here, if the candidate generative probability index is 0, it means that the descriptions of multiple generated images corresponding to the pending presented word are predicted to be all different from the descriptions of the pending presented word, meaning that the pending presented word is not suitable for illustration generation. If the candidate generative probability index is 1, it means that the descriptions of only a small portion of the multiple generated images corresponding to the pending presented word are predicted to be the same as the descriptions of the pending presented word, similarly meaning that the presented word is not suitable for illustration generation. If the candidate generative probability index is 2, it means that almost, and indeed all, of the descriptions of multiple generated images corresponding to the pending presented word are predicted to be the same as the descriptions of the pending presented word, meaning that the presented word is suitable for illustration generation. On the other hand, the generative probability index is used to represent the score at which the pending presented word can be used for illustration generation, and here the score criterion may be a score value where the candidate generative probability index is directly the generative probability index. A higher score on the generation probability index (for example, a score of 2) indicates that the pending suggested word has greater reference value for use in generating illustrations.

[0058] In step 1022B, the convolutional network in the second text classifier is invoked to perform a convolution operation on the text features to obtain the second convolutional feature, the multi-classification layer in the first text classifier is invoked to map the second convolutional feature to the second probability of multiple description types, and the description type corresponding to the highest second probability is set as the description type of the awaiting presented word.

[0059] The probability index for generating text features of the awaiting presented word is determined, and the convolutional network in the second text classifier is invoked to perform a convolution operation on the text features to obtain a second convolutional feature, where the convolutional network may have multiple convolutional layers, such as two layers. Next, the multi-classification layer in the first text classifier is invoked to map the second convolutional feature to the second probability of multiple description types, and the description type corresponding to the highest second probability is set as the description type of the awaiting presented word.

[0060] In some embodiments, the description types of pending presentation words include types that include actions and types that do not, and are used to determine whether a pending presentation word is an action description phrase; that is, the second text classifier is biclassified. If the predicted description type includes a verb, it means that the pending presentation word is an action description phrase, and since action description phrases are generally difficult to fully represent on an image, the failure rate of generated images is high. That is, if a pending presentation word includes a verb, it means that the pending presentation word is not suitable for illustration generation. If the predicted description type does not include a verb, it means that the pending presentation word is not an action description phrase, and further subdivision of the pending presentation word is performed to obtain multiple clauses, for which images can then be secondary generated.

[0061] According to the embodiment of the present application, by identifying awaiting presented words using a single word generation possibility identification model, it is possible to identify awaiting presented words that have a low generation possibility and include actions, perform a rationality evaluation on the awaiting presented words, and further provide processing feedback on the awaiting presented words. This solves the problem that a single image generation attempt using awaiting presented words is irrational, and avoids a decrease in overall image generation effectiveness caused by the low generation possibility of awaiting presented words and the omission of elements.

[0062] Continuing to refer to Figure 3A, in step 103, in response to the fact that the generative possibility index is greater than the index threshold, the description type indicates that the pending presented word does not contain a verb, and the pending presented word contains multiple clauses, similar images corresponding to each of the clauses are obtained.

[0063] Here, the fact that the generative possibility index is greater than the index threshold, the description type indicates that the pending presented word does not contain a verb, and the pending presented word contains multiple clauses can be considered word conditional for unsuitability in identifying the generative possibility of a presented word in a scene where an image is generated based on text. If the above conditions are met, it means that the pending presented word is an unsuitable word, i.e., it is not a word suitable for generating an illustration.

[0064] After performing phrase generation possibility identification on the pending presented word, if the predicted generation possibility index is greater than the index threshold (e.g., an index threshold of 1), the predicted description type indicates that the pending presented word does not contain a verb, and the pending presented word contains multiple clauses, then it is necessary to split the pending presented word to obtain its multiple clauses.

[0065] From all the generated images of the pending presented word, the most similar image for each clause is searched for. Specifically, for each clause of the pending presented word, the image-text similarity between the clause and all the generated images of the pending presented word is calculated. For example, the image-text similarity may be cosine similarity. The single generated image with the highest image-text similarity from all the generated images is determined as the candidate similar image for that clause. In this way, a candidate similar image corresponding to each clause can be determined, and if the image-text similarity between the clause and the corresponding candidate similar image is greater than the image-text similarity threshold, the corresponding candidate similar image is set as the similar image for the clause.

[0066] In some embodiments, if the generative probability index is greater than the index threshold (e.g., the exponential threshold is 1) and the predicted description type indicates that the pending presentation contains a verb, then it means that the corresponding pending presentation is an action description phrase, and since action description phrases are generally difficult to perfectly represent on an image, the failure rate of the generated image is high, meaning that the pending presentation is unsuitable for illustration generation, and therefore the pending presentation is deleted.

[0067] In some embodiments, if the generative probability index is smaller than the index threshold (e.g., an index threshold of 1), it means that the pending suggestion word is not suitable for illustration generation. Considering that different pending suggestion words are extracted sequentially from the input text, for example, if the input text is divided into multiple pending suggestion words according to periods at the phrase level, then the illustrations for the pending suggestion words can be stored directly in the text's illustration sequence in the order they are generated.

[0068] Referring to Figure 3A, in step 104, the degree of image difference between similar images corresponding to multiple sections is determined.

[0069] After determining similar images corresponding to each clause of the presented word awaiting processing, the degree of image difference between these similar images is then determined, where the degree of image difference can be measured by the variance of the images.

[0070] In some embodiments, referring to Figure 3D, step 104 shown in Figure 3A can also be achieved by steps 1041 to 1042, which are described in detail below.

[0071] In step 1041, for each similar image, the grayscale mean of the pixels in each row within the similar image is determined, and the grayscale mean of the pixels in each row is combined as an image feature of the similar image.

[0072] In some embodiments, grayscale processing is performed on each similar image to obtain a corresponding grayscale map. For each grayscale map of the similar image, the average value of the pixel points in each row of the similar image is calculated sequentially, the average value of the pixel points in each row is recorded, and these are combined as image features of the similar image.

[0073] In step 1042, the variance of the image features of the similar images corresponding to each of the multiple sections is determined, and the variance is used as the degree of image difference between the similar images corresponding to each of the multiple sections.

[0074] Continuing from the above example, variance calculation is performed on all obtained mean values, and the resulting variances are the feature values ​​of the similar images. After calculating the feature values ​​of each similar image, they are compared with each other, and the difference between the variances of any two similar images (abbreviated as the variance difference) is determined. The value with the largest variance difference is taken as the total variance of the multiple similar images, and the total variance is taken as the image difference between the similar images corresponding to each of the multiple nodes.

[0075] Continuing to refer to Figure 3A, in step 105, in response to the image difference being less than the image difference threshold, similar images corresponding to each section are used as illustrations for the corresponding sections.

[0076] If the image difference between similar images corresponding to multiple clauses is less than the image difference threshold, it means that the differences between the multiple clauses in the pending presentation word are small, and the corresponding descriptive content is similar, so there is no need to split and generate them. Therefore, similar images corresponding to multiple clauses are used directly as illustrations, and the illustrations are stored in the illustration sequence corresponding to the corresponding input text, according to the order of the clauses in the pending presentation word.

[0077] In some embodiments, if the image difference between similar images corresponding to multiple sections of a pending presentation word exceeds an image difference threshold, it means that there is a large difference in the content described in the multiple sections of the pending presentation word, and the described content may also be different, so the current pending presentation word is judged to be an inappropriate phrase. In other words, the similar images corresponding to the sections within the pending presentation word are not related, so it is necessary to further divide the sections within the pending presentation word. At this point, the process returns to step 101 shown in Figure 3A, and the image processing method provided in the embodiments of this application is executed again for the pending presentation word corresponding to the multiple sections, thereby finally determining the illustration for the pending presentation word.

[0078] Here, if the image difference between similar images corresponding to multiple clauses is greater than or equal to the image difference threshold, this can be considered a condition for a word to be unsuitable for identifying the possibility of generating a presented word in a scene where images are generated based on text. If the image difference between similar images corresponding to a clause is greater than or equal to the image difference threshold, it means that the clause is an inappropriate word, i.e., it is not a word suitable for generating an illustration.

[0079] In some embodiments, referring to Figure 3E, the image difference threshold can be obtained by performing steps 106 to 109 below before performing step 105 shown in Figure 3A, which will be specifically described below.

[0080] In step 106, obtain a sample set of image-text pairs.

[0081] In some embodiments, before acquiring similar images corresponding to multiple sections, it is necessary to determine a single image-text similarity threshold to select the generated image that is most similar to a section as the section's similar image. At the same time, in order to improve the sequence correlation of these section-like similar images, it is also necessary to determine the image difference of these similar images, and therefore a single image difference threshold must be set. The method for determining the image-text similarity threshold and the image difference threshold can be achieved by collecting a large number of image-text pairs to form an image-text pair sample set, and then searching and retrieving from there, where the image-text pair includes a sample presentation word and a sample similar image.

[0082] In step 107, the recall rate of the image-text pair sample set at the current threshold point is determined in ascending order of a plurality of predetermined threshold points.

[0083] A certain number of image-text pairs (e.g., 10,000 pairs) are obtained, and threshold points can be set in ascending order of a set of predetermined threshold points, for example, with a step length of 0.1 units between 0 and 1. Next, at each different threshold point, a similarity index is calculated between the sample presented word and the sample similar image of each of these 10,000 image-text pairs, where the similarity index may be cosine similarity. The specific calculation method will be explained in step 20213 below. Among these image-text pairs, image-text pairs whose similarity index is greater than or equal to the threshold point are designated as reproduced image-text pairs, and the number of reproduced image-text pairs is statistically calculated to determine the ratio of the number of reproduced image-text pairs with a similarity index greater than the threshold point to the total number of image-text pairs (10,000 image-text pairs), and this ratio is taken as the recall rate at the corresponding threshold point. In step 108, in response to the fact that the recall at the current threshold point is equal to or greater than the recall threshold, the current threshold point is determined as the image-text similarity threshold.

[0084] If the recall rate at the current threshold point is equal to or greater than the recall threshold, the current threshold point is determined as the image-text similarity threshold, where the recall threshold can be determined according to the number of image-text pair sets, for example, it may be 80%. That is, if the recall rate at the current threshold point is 80% or higher, this threshold point is taken as the image-text similarity threshold.

[0085] In step 109, the variance of the image features of the sample-similar images within the reproduced image-text pair is determined, and the variance of the image features of the sample-similar images within the reproduced image-text pair is used as the image difference threshold.

[0086] After obtaining the reproduced image-text pair, the variance of the image features of the sample-similar images within the reproduced image-text pair is calculated. The variance of the image features is used to represent the difference between images; a larger variance indicates a greater degree of image difference, while a smaller variance indicates a smaller degree of image difference, i.e., a higher degree of image similarity. After determining the variance, the variance of the image features of the sample-similar images within the reproduced image-text pair is used as the image difference threshold.

[0087] For example, suppose there are 10,000 image-text pairs and the recall rate is determined to be 80%, in which case the search is stopped. In this case, it means that the image-text similarity of 8,000 image-text pairs is greater than the current threshold point, and this threshold point is used as the image-text similarity threshold to obtain 8,000 recalled image-text pairs. The variance of the sample similar images corresponding to these 8,000 recalled image-text pairs is calculated, and this variance is used as the image difference threshold.

[0088] According to the embodiment of the present invention, in addition to selecting an illustration for the image generated from the first presented word, a word generation possibility identification model is used to combine the distribution status of the generated image with the description of the presented word and the generated image, thereby automatically identifying presented words with low illustration quality and providing processing feedback. Specifically, by performing a generation possibility evaluation on the presented word and determining the description type of the presented word, it is possible to identify whether the presented word is a detailed description and whether it is suitable for generating an illustration. As a result, the presented word can be automatically split or deleted, and by processing the clauses that have been split from the presented word, a rational evaluation of the clause granularity of the presented words awaiting processing and the selection of images with finer granularity are realized, resulting in more rational generated images and improved correlation of the overall generation results of the presented words awaiting processing.

[0089] In some embodiments, referring to Figure 3F, after performing step 101 shown in Figure 3A, steps 201 to 203 can also be performed, which are described in detail below.

[0090] In step 201, in response to the fact that the pending presented word is an appropriate word, multiple generated images of the pending presented word are obtained.

[0091] Here, an appropriate word is a word that does not satisfy the inappropriate word condition, and the inappropriate word condition includes one of the following: the probability of generation is greater than the index threshold, the description type indicates that the pending presented word does not contain a verb and the pending presented word contains multiple clauses, the image difference is greater than or equal to the image difference threshold, and the probability of generation is greater than the index threshold and the description type indicates that the pending presented word contains a verb.

[0092] In some embodiments, as shown in Figure 3G, the "acquisition of multiple generated images of the presented word awaiting processing" in step 201 shown in Figure 3F can also be achieved by the following steps 2011 to 2015, which will be explained in detail below.

[0093] In step 2011, encoding is performed on the pending presented words to obtain the text features of the pending presented words and the image features corresponding to the text features of the presented words.

[0094] In some embodiments, after obtaining a pending presented word, the text features of the pending presented word can be obtained by calling the text encoder of the CLIP (Contrastive Language-Image Pre-Training) model to perform encoding on the pending presented word. This is because the CLIP model is trained on image-text pair samples. When the CLIP model is used for prediction, it outputs matching image-text pairs based on prediction samples corresponding to the input. If the input is text only, it can output the corresponding text features and simultaneously generate a single random image feature, but this image feature does not correspond to the text features and may or may not have random pixel features.

[0095] In step 2012, noise is added to the image features to obtain noisy image features.

[0096] Continuing from the above example, the image generation model is invoked to perform noise addition on randomly generated image features. That is, random noise elements are added to the image features in stages, and after several time steps, the image features become completely random noise image features.

[0097] In step 2013, text features and noise image features are fused to obtain fused features.

[0098] Next, the text features of the awaited presented words and completely random noise image features are fused to obtain a fused feature. Since the text features of the awaited presented words contain elements of the target theme that can be identified by the image generation model, they can be used in the denoising process to guide the generation of image features that have the corresponding target theme elements.

[0099] In step 2014, denoising is performed on the fused features to obtain the reconstructed image features.

[0100] After fusing the text features of the awaiting presented words into the image generation model, the image generation model performs denoising on the fused features formed by fusing the text features. Specifically, it gradually removes noise elements within the fused features, combines elements of specific themes or styles contained in the text features, and generates reconstructed image features having target theme elements corresponding to the text features through a denoising process with the same time steps as the noise addition process.

[0101] In step 2015, a decoding process is performed on the reconstructed image features to obtain multiple generated images.

[0102] After obtaining reconstructed image features through a denoising process, the generated image is obtained by performing a decoding process on the reconstructed image features. The generated image is obtained by denoising the noisy image features, and because the image generation model has different recognition capabilities for presented word text features, this process is random, and the generated image is not unique.

[0103] By repeatedly executing steps 2011 to 2015 above, different generated images can be obtained; that is, multiple different generated images can be obtained based on the pending prompt. In actual applications, the number of generated images corresponding to each pending prompt can be specified, for example, it could be 10, meaning that 10 generated images are generated based on one pending prompt.

[0104] Continuing to refer to Figure 3F, in step 202, an illustration for the word awaiting processing is determined from multiple generated images.

[0105] In some embodiments, the pending presentation word may be text entered by a user or a computer program (i.e., an artificial intelligence-based text generation program), and may be a phrase from a novel or script, or a phrase containing multiple clauses. In other words, since the pending presentation word may contain multiple presentation words, the final illustration corresponding to the pending presentation word is a sequence diagram with sequence consistency. Therefore, it is necessary to determine the generated image corresponding to each presentation word while ensuring sequence consistency of the generated images for the multiple presentation words within the pending presentation word. That is, the final illustration for the pending presentation word is obtained by performing sequence image selection for multiple presentation words.

[0106] In step 203, save the illustration for the word being presented and awaiting processing.

[0107] According to the embodiment of the present application, if the pending suggested word does not satisfy the inappropriate word condition, it means that the pending suggested word is a suitable word for generating an illustration, and the corresponding illustration is generated by steps 202 and 203. Compared to a solution that generates illustrations in advance based on text and then removes inappropriate image illustrations, this can save the associated computational resources. If the pending suggested word satisfies the inappropriate word condition, it means that the pending suggested word is an inappropriate word unsuitable for generating an illustration, and the subsequent steps 202 and 203 are not executed, thus saving the associated computational resources.

[0108] As described above, the processing method is explained for cases where the probability index for generating a pending suggestion word is greater than the index threshold, the description type indicates that the pending suggestion word does not contain a verb, and the pending suggestion word contains multiple clauses. The processing method is also explained for cases where the probability index for generating a pending suggestion word is greater than the index threshold, the description type indicates that the pending suggestion word contains a verb, and at the same time, the processing method is explained for cases where the probability index is less than or equal to the index threshold. If a pending suggestion word does not satisfy any of the above conditions, it means that the pending suggestion word is suitable for illustration generation, and the pending suggestion word and the corresponding illustration are retained. For each pending suggestion word extracted from the text, the above solution is used to determine whether each pending suggestion word is suitable for illustration generation, and accordingly, deletion, splitting, or retention is performed. By deleting suggestions that are not suitable for image generation and splitting overly long suggestions, the inclusivity of elements in the generated image is ensured, and the overall correlation between the text and the illustration sequence is improved.

[0109] In some embodiments, referring to Figure 3H, step 202 shown in Figure 3F can be achieved by the following steps 2021 to 2025, which are described in detail below.

[0110] In step 2021, the image-text similarity between each of the multiple generated images and the awaiting presented word is determined, and the generated image corresponding to an image-text similarity greater than the image-text similarity threshold is designated as the retained image.

[0111] In some embodiments, the pending suggestion word is text input by a user or a computer program (i.e., an artificial intelligence-based text generation program), and the pending suggestion word is the first suggestion word extracted from the text, i.e., the first suggestion word within the pending suggestion word, first, the image-text similarity between multiple generated images and the pending suggestion word is determined, and the generated image corresponding to the highest image-text similarity is selected as the optimal generated image, which becomes the illustration for the first suggestion word, i.e., the illustration for the pending suggestion word.

[0112] In some embodiments, the pending suggestion word is text input by a user or a computer program (i.e., an artificial intelligence-based text generation program), and the pending suggestion word is not the first suggestion word extracted from the text, for example, if it is the second suggestion word among the pending suggestions, first, the image-text similarity between multiple generated images and the pending suggestion word is determined, and then the generated image corresponding to an image-text similarity greater than the image-text similarity threshold is set as the retained image. Here, the image-text similarity threshold is predetermined and may be obtained by searching a large number of image-text pairs.

[0113] In some embodiments, referring to Figure 3I, the "determining the image-text similarity between each of the multiple generated images and the awaiting presented word" in step 2021 shown in Figure 3H can be achieved by the following steps 20211 to 20213, which are described in detail below.

[0114] In step 20211, image features corresponding to each of the multiple generated images are obtained.

[0115] In some embodiments, if the pending suggestion word is text input by a user or a computer program (i.e., an artificial intelligence-based text generation program), and the pending suggestion word is a suggestion word other than the first one extracted from the text, it is necessary to determine the image-text similarity between multiple generated images and the pending suggestion word, that is, it is necessary to calculate the similarity between the image features of the generated images and the text features of the pending suggestion word. First, it is necessary to obtain the image features of multiple generated images, and the image encoder of the CLIP model's image branch is called on the multiple generated images to perform encoding processing on the multiple images and obtain the image encoding features corresponding to the multiple generated images.

[0116] In step 20212, noun elements within the presented word awaiting processing are identified, and encoding is performed on the noun elements to obtain the present word element features.

[0117] Continuing from the above embodiment, a pending presentation word may contain multiple noun elements, and these noun elements need to be reflected in the generated image and, consequently, in the final sequence diagram. The main types include people, tools, and environments. Since image-text similarity represents the extent to which the semantic information of the noun elements within the presentation word is presented in the generated image, it is necessary to identify each noun element within the pending presentation word for each pending presentation word. This identification method can be implemented using a pre-trained cross-modal model and a multi-label classifier. On the other hand, for the generated images corresponding to the pending presentation words, it is also necessary to identify and record whether the noun elements corresponding to the pending presentation words appear in each generated image, and to store the identification record results in a database, in order to use as a criterion for subsequent sequence image selection.

[0118] Exemplary, the system identifies a noun element within a given word and determines whether it is a person, an object, or an environment. If so, it records whether the noun element appears in the corresponding noun element and the corresponding generated image, in the format of [Given word identifier (ID), Image identifier (ID), Element category-Element (ID)-Element name-Whether the element appears in the image]. For example, if the noun elements "person, cat, dog" are identified from the given word, the format of the record would be "i-th given word, j-th image, object-element 1-cat-appears, object-element 2-dog-appears, person-element 1-person-appears".

[0119] After identifying the noun elements within the awaiting presented word, other parts of the presented word, such as adjectives, adverbs, and prepositions, are directly removed. The text encoder of the CLIP model's text branching is then invoked on the identified noun elements to perform encoding on multiple noun elements within the awaiting presented word, thereby obtaining presented word element features corresponding to the multiple noun elements.

[0120] In step 20213, for each generated image, the cosine similarity between the image features of the generated image and the presented word element features is determined, and the cosine similarity is used as the image-text similarity between the generated image and the presented word awaiting processing.

[0121] Continuing from the above embodiment, multiple noun elements of the awaiting presented word are obtained, and corresponding image coding features and presented word element features are extracted from multiple generated images. Then, for each generated image within the multiple generated images, the similarity between the image coding features of the generated image and the presented word element features is calculated. Here, the similarity may be cosine similarity, and the specific calculation formula is as follows.

number

[0122] In Equation 1 above, similarity represents the cosine similarity, θ represents the cosine angle of the cosine similarity, A represents the image coding features of the generated image, B represents the present word element features, and A i represents the image features of the i-th generated image, and B i This represents the i-th presented word element feature.

[0123] The cosine similarity between the image encoding features of each generated image and the presented word element features is taken as the image-text similarity between multiple generated images and the presented words awaiting processing. Next, all generated images are sorted based on their image-text similarity, and then, based on a predetermined image-text similarity threshold, the generated images corresponding to image-text similarity greater than the threshold are designated as retained images. Here, there may be multiple retained images for presented words that are not the first to appear, and a specific number of retained images may be specified, for example, 10.

[0124] Continuing to refer to Figure 3H, in step 2022, in response to the fact that the retained image contains a noun element within the awaited presented word and that at least one past element in the database contains a noun element, the past element features of the noun element are queried from the database.

[0125] A pending suggestion is text input by a user or computer program (i.e., an artificial intelligence-based text generation program). If the pending suggestion is the first suggestion extracted from the text, the illustration for the pending suggestion, i.e., the optimally generated image of the first suggestion, is designated as the past illustration and stored in the database in association with the noun element within the first suggestion. In this case, the noun element of the first suggestion is called the past element, the first suggestion is designated as the past suggestion, and the element features of these noun elements within the past suggestion, i.e., the suggestion element features extracted by the CLIP model, are stored in the database as past element features according to the corresponding recording format.

[0126] For example, in the database, the optimal association format between the generated image and the noun element in the first presented word may be: tool-element1-cat-[presented word identifier-image identifier-image feature], tool-element2-dog-[presented word identifier-image identifier-image feature], person-element3-person-[presented word identifier-image identifier-image feature].

[0127] The format for saving past element features may be element1-cat-[presentation word identifier-image identifier-cat features], and the format for saving past illustrations may be past image--[presentation word-image identifier-element category-image features].

[0128] If the pending suggestion is text input by a user or computer program (i.e., an artificial intelligence-based text generation program), and the pending suggestion is a suggestion other than the first one extracted from the text, then for each retained image of the other suggestion, it is possible to query whether the retained image contains a noun element from the pending suggestion, based on the identification record results in the database. If the retained image contains a noun element from the pending suggestion, the past element features corresponding to the noun element are then queried from the database, i.e., the suggestion element features of the noun element of the past suggestion extracted by the CLIP model.

[0129] In some embodiments, referring to Figure 3J, the "querying past element features of noun elements from the database" in step 2022 shown in Figure 3H can be achieved by the following steps 20221 to 20222, which are described in detail below.

[0130] In step 20221, common elements between the retained image and the noun element are identified.

[0131] For each retained image, if the retained image contains a noun element from the awaiting presented word, the past element features corresponding to the noun element are then queried from the database. Here, the query process requires identifying common elements between the retained image and the noun element of the awaiting presented word; that is, determining whether the noun element in the retained image is the same as a past element stored in the database, and verifying whether the noun element appearing in the retained image is already recorded in the database.

[0132] In step 20222, query the database for past element features corresponding to common elements.

[0133] If it is determined that there are common elements between the noun elements of the retained image and past elements in the database, i.e., that the noun elements of the retained image are recorded in the database, then the past element features corresponding to these common elements are determined, retrieved, and used in subsequent calculations of element similarity.

[0134] For example, the first presented word (past presented word) awaiting processing contains the three noun elements "person, cat, dog," and is stored in the database associated with the corresponding optimal generated image. The features of the corresponding noun elements are also stored in the database as past element features. At this point, based on the identification record results in the database, it can be determined that the three noun elements "person, cat, dog" also appear in the retained image of the second presented word. Therefore, it can be determined that the common elements in the retained image and the noun elements (past elements) of the first presented word in the database are "person, cat, dog." In other words, it is determined that the noun elements "person, cat, dog" that appeared in the retained image are already recorded in the database. Next, the past element features of "person, cat, dog" are determined, retrieved, and used in the subsequent calculation of element similarity.

[0135] According to the embodiment of the present invention, when performing image selection for a pending suggestion that is not the first to be processed, it is always determined whether a past element exists in the database for the retained image corresponding to the pending suggestion. This ensures that each pending suggestion is associated with a corresponding past element in the image selection process, that the illustrations for each pending suggestion have element relationships, and that the correlation between illustrations is improved.

[0136] Continuing to refer to Figure 3H, in step 2023, the element similarity between the retained image and past element features is determined, the image features of past illustrations of previously presented words are queried from the database, and the image similarity between the retained image and past illustrations is determined based on the image features of the past illustrations and the image features of the retained image.

[0137] In some embodiments, referring to Figure 3K, the "determining the element similarity between the retained image and past element features" in step 2023 shown in Figure 3H can be achieved by the following steps 20231 to 20232, which are described in detail below.

[0138] In step 20231, the element similarity between the retained image and each of the past element features of different types is determined.

[0139] In some embodiments, it is determined that there are common elements between the noun elements of the retained image and the past elements in the database. After obtaining the past element features corresponding to these common elements, considering that these common elements may have different types such as people, tools, and environments, this can affect the generation effect of the presented words awaiting processing. For example, if the first presented word contains past elements such as people, tools, and environments, the next presented word is very likely to contain the environment past element from the previous presented word. Conversely, the corresponding person and tool elements are likely to be different from the person and tool past elements of the previous presented word. Therefore, when calculating the element similarity between the retained image and the past element features, it is necessary to check the trend of element types, and thus it is necessary to calculate the element similarity between the retained image and each of the past element features of different types. Here, the method for calculating element similarity may be to first query the image features corresponding to the retained image from the identification record results stored in the database, and then calculate the cosine similarity between the image features of the retained image and the corresponding past element features to obtain the element similarity between the retained image and the past element features. The method for calculating the cosine similarity here is the same as in step 20213 shown in Figure 3I, and will not be explained again here.

[0140] In step 20232, the element similarity between the retained image and each of the past element features of different types is weighted and added together to obtain the element similarity between the retained image and the past element features.

[0141] Building upon the above embodiment, in order to reflect the trend of element similarity between the retained image and past elements of different types, a weight corresponding to the element similarity of each type of past element can be set based on the update frequency of the corresponding type of past element. Then, the element similarity between the retained image and each of the past element features of different types is added to the corresponding weight to obtain the element similarity between the retained image and the past element features.

[0142] For example, if environment, person, and tool types such as forest, person, and cat exist simultaneously in both the retained image and past elements in the database, first, the cosine similarity between the image features of the retained image and the "forest," "person," and "cat" features is calculated and assigned to a, b, and c, respectively. At this point, the update frequency of the past elements is determined, and if it is confirmed that the environment type element "forest" appears in multiple pending presented words, a large weight such as 0.5 can be assigned. In contrast, the change frequency of the "person" and "cat" elements is high, and they may only appear in one or two pending presented words, so small weights such as 0.4 and 0.1 can be assigned to them, respectively. Next, based on the weights of 0.5, 0.4, and 0.1 assigned to "forest, person, and cat," a, b, and c are weighted and added together to obtain the element similarity between the retained image and "forest, person, and cat," and recorded as "0.5a + 0.4b + 0.1c."

[0143] After determining the element similarity between the retained image and past element features, the image features of past illustrations for past presented words, i.e., the image features of the optimal generated image corresponding to the first presented word in the database, are queried from the database. Then, based on the image features of the past illustrations and the image features of the retained image, the image similarity between the retained image and the past illustrations is calculated. Here, calculating the image similarity may also be done by calculating the cosine similarity between the image features, and the cosine similarity method is the same as in step 20213 shown in Figure 3D, which will not be repeated here.

[0144] Continuing to refer to Figure 3H, in step 2024, the element similarity and image similarity are weighted and added together to obtain the total fusion score of the retained images.

[0145] For each retained image, the image-text similarity between the retained image and past illustrations, and the element-similarity between the retained image and past elements in the database are determined. Then, the average of the image-text similarity and element-similarity of the retained image is calculated, and this average is taken as the sequence similarity of the current retained image.

[0146] Next, for each retained image, the image-text similarity between the corresponding retained image and the awaiting presented word has already been determined in step 2021 shown in Figure 3H. By setting corresponding weights for each, for example by setting the same weights (0.5, 0.5, etc.), a weighted summation process is performed on the image-text similarity and sequence similarity of the retained images to obtain the total fusion score of the retained images. Here, the weights can be adjusted as appropriate based on the number of retained images or the number of noun elements in the awaiting presented word, and are not limited to the same weights.

[0147] In step 2025, the retained image corresponding to the highest fusion total score is determined to be the illustration for the word awaiting processing.

[0148] After determining the total fusion score of each retained image corresponding to the pending suggested word, these retained images are sorted based on their total fusion scores to obtain a final sort result. The retained image with the highest total fusion score among the sorted images is selected as the illustration corresponding to the pending suggested word. Next, the illustration for that pending suggested word is saved to the database as a new past illustration, and processing of the next pending suggested word continues until all pending suggested words in the input text have corresponding illustrations generated, selecting the illustration for the next pending suggested word.

[0149] In some embodiments, if the pending suggestion is text entered by a user or a computer program (i.e., an artificial intelligence-based text generation program), and the pending suggestion is not the first suggestion extracted from the text, then each time an illustration corresponding to a suggestion is selected, that illustration is saved in the database as a new past illustration, and each suggestion also contains a different noun element. For the three types of past elements stored in the database—people, tools, and environments—when selecting an illustration corresponding to a subsequent pending suggestion, if the same past element in the database appears again, the past element feature corresponding to the past element in the database is updated. Conversely, for past illustrations in the database, each time a pending suggestion is processed and a corresponding illustration is generated, the past image feature corresponding to the past illustration in the database is updated. Considering that the processing order of past suggestions and subsequent suggestions is contiguous, the database is dynamically updated based on at least one past suggestion extracted from the input text, and the generated images and past illustrations corresponding to the past suggestions. In some embodiments, a method for dynamically updating past element features in the database involves, when selecting an illustration corresponding to a subsequent pending suggestion, if the same past element in the database reappears in the retained image corresponding to the pending suggestion, then, using predetermined corresponding weights, the element features of the noun element appearing in the retained image and the past element features of the same-named past element in the database are weighted and added together, and the resulting updated past element features replace the previous past element features. Considering that the changes in noun elements included in multiple pending suggestions are not significant, and that many suggestions may describe a limited number of noun elements such as environments, people, and tools, the updating of past element features in the database is relatively gradual, and therefore, the predetermined weights for past element features and the noun element features of the current retained image are relatively well-balanced. For example, they may be 0.6 and 0.4.

[0150] For example, if a noun element such as "forest, person, cat" appears in the retained image, and the past element features stored in the corresponding data also include "forest, person, cat," then the noun element features in the current retained image are added to the database with a weight of 0.4. That is, "Past element features of "forest, person, cat" in the database × 0.6 + Element features of the noun element "forest, person, cat" in the retained image × 0.4 = New past element features of "forest, person, cat" in the database."

[0151] In some embodiments, the method for dynamically updating past illustration features in the database selects an illustration corresponding to a subsequent pending suggestion. Each time a pending suggestion is processed and a corresponding illustration is generated, the image features of the retained image and the image features of past illustrations in the database are weighted by predetermined corresponding weights, and the resulting updated past illustration features replace the image features of past illustrations before the update. Considering that each pending suggestion ultimately selects the retained image with the highest fusion total score as its illustration, the updates to past illustrations in the database are very frequent, and therefore the updates to the image features of past illustrations in the database are relatively fast. For this reason, the image features of past illustrations in the database are given high weights, while the image features of the current retained image are given low weights. For example, these may be 0.8 and 0.2.

[0152] For example, after processing one pending presented word, the retained image with the highest fused total score is selected as the illustration corresponding to the pending presented word, and the image features of the current retained image are added to the database with a weight of 0.2. That is, "Past illustration features in the database × 0.8 + Image features of the retained image × 0.2 = New past illustration features in the database".

[0153] According to the embodiment of the present invention, sequence processing is performed on the generated image for each suggested word in the input text, and a dynamically updated database is constructed as a reference. This constrains the selection of retained images corresponding to the suggested words, and by selecting the optimal illustration for each corresponding suggested word, the generated results are re-sorted with consistency of related elements, and the final sorted result of the sequence diagram is obtained. Therefore, the consistency of the preceding and succeeding illustrations generated for each suggested word can be ensured, and the relevance of the illustrations corresponding to the suggested words can be improved.

[0154] The following describes exemplary application examples of the embodiments of this application in actual application scenarios.

[0155] The embodiments of this invention can be applied to illustration generation scenes in wuxia novels and are used to assist in generating illustrations corresponding to novel text entered by the novel user. In the task of generating sequential illustrations for sequential phrases in wuxia novels based on an image generation model, it is necessary to generate images for multiple consecutive phrases. The biggest challenge in the sequential illustration creation task is maintaining the consistency of the images in context, i.e., a new generated image must be related to the content of the previous generated image. Furthermore, in wuxia novels, a single phrase often contains many clauses, and since the description of each clause is different, multiple generated images with completely different content can be generated from the same phrase, and outputting inappropriate generated images tends to degrade the final illustration effect.

[0156] Based on this scenario, the embodiment of the present invention adaptively generates and adjusts novel sequence illustrations by generating images from text based on an image generation model and mining the images and text content, thereby forming a closed-loop sequence illustration system for generating illustrations of sequence phrases in a novel. In the sequence for generating images from the first phrase, the system automatically identifies phrases with low illustration quality and provides processing feedback by combining a text generation possibility identification model with the distribution status of generated images, based on the text description and the generated images. Subsequently, secondary processing such as deletion, retention, and splitting automatically controls the addition and deletion of effective generation point locations, ultimately forming a closed-loop system for sequential illustration generation.

[0157] Referring to Figure 4, which is a schematic diagram of a closed-loop system for generating continuous illustrations according to an embodiment of the present invention, the closed-loop system first inputs each word or phrase of the novel script (words 1, 2, and 3 in Figure 4) into an image generation model, generates K images (e.g., 10) for each word or phrase, processes feedback based on the generation result sequence, selects the optimal generated image for each word or phrase, and performs identification of inappropriate words in the generated images. That is, based on the generated images and generated words or phrases, words or phrases that satisfy the above inappropriate word or phrase conditions are identified as "inappropriate words or phrases in the generated images," abbreviated as inappropriate words or phrases, and secondary processing is performed on the inappropriate words or phrases. Specifically, this includes deletion, retention, and splitting. For example, word 1 is retained, word 2 is split into 2-1, 2-2, and 2-3, and word 3 is deleted. Next, for words or phrases that require secondary generation, the image generation model is called to perform image generation, and then the generated image sequence processing is performed to obtain the final sequence diagram.

[0158] For example, the system can perform multiple iterations of operational steps, such as identifying inappropriate words in generated images within a closed-loop system, generating images after quadratic division of words, and processing the generated image sequence. For instance, complex words can be divided into clauses until no inappropriate words remain in the generated image, and these clauses can then be further divided into multiple subordinate clauses with multiple subject-predicate structures.

[0159] In some examples, as shown in Figure 4, based on the principle of a closed-loop system, inappropriate phrase recognition is performed on one phrase (e.g., phrase 3), namely "The boy stuck the wooden tag in front of the mound of dirt, struck it several times with his fist, and drove the wooden tag deep into the dirt," to identify it as an inappropriate simple phrase and delete phrase 3. On the other hand, inappropriate phrase recognition is performed on another phrase (e.g., phrase 2), namely "It was already close to dawn when the two arrived at the small liquor store at the foot of the mountain, and the boy, without regard for the details, woke the store owner from his futon, threw a large silver ingot at him, and ordered pheasant and mushroom soup," to identify it as an inappropriate complex phrase and need to be split. That is, phrase 2 is split into three sections, then the image generation model is called to generate images for the three sections, and the generated image sequence processing is performed to obtain the final sequence diagram.

[0160] For illustrative purposes, referring to Figure 5, which is a schematic diagram of the image generation of the original phrase according to the embodiment of the present invention. For phrase 2 shown in Figure 4, "When the two arrived at the small liquor store at the foot of the mountain, it was already close to dawn, and the boy, without regard for details, woke the shopkeeper from his futon, threw a large silver ingot at him, and ordered pheasant and mushroom soup," the image generation model is called directly based on the original phrase to generate an image. As shown in Figure 5, it is clear that only the element described in phrase 2, "the small liquor store at the foot of the mountain," appears in the image, and none of the other descriptive elements appear. At this time, based on the closed-loop system shown in Figure 4, it is necessary to perform phrase identification of the original phrase 2 as an "inappropriate complex phrase" in the generated image, and to perform secondary phrase segmentation on the original phrase 2, specifically, it is divided into the following three sections: "the small liquor store at the foot of the mountain," "When the two arrived, it was already close to dawn," and "the boy ordered pheasant and mushroom soup."

[0161] Next, the three nodes are each secondarily generated using an image generation model, and a generation result sequence processing is performed on all generated images secondarily generated from the three nodes to obtain the three corresponding generated images.

[0162] Referring to Figure 6, Figure 6 is a sequence diagram of images secondarily generated after phrase segmentation according to an embodiment of the present invention. As shown in Figure 6, each of the three images contains all the descriptive elements corresponding to the three sections segmented from the original phrase 2. Next, a sequence processing is performed on the three images. That is, the three generated images are merged to obtain a sequence diagram formed after phrase segmentation, and this sequence diagram is used as the final sequence diagram of the original phrase 2.

[0163] Furthermore, the closed-loop system improves the frame relationships between each generated image, i.e., the sequence relevance between generated images. Referring to Figure 7, which is a schematic diagram of the original generated image sequence according to an embodiment of the present invention, first, for the first descriptive phrase, "The short sword emitted a white, cold light, and the sweaty girl let out a scream of terror," single image generation and sorting are performed, that is, multiple generated images are generated based on the descriptive phrase, and then the multiple generated images are sorted based on an evaluation index to select the best image, such as the first image on the left of Figure 7. Next, for the second descriptive phrase, "The girl in the navy blue swordsman's uniform turned around and looked with a lonely expression at the broken maple leaf that had fallen to the ground," multiple generated images are generated in the same manner, and then the best image, such as the second image on the right of Figure 7, is selected. It can be seen that the clothing of the figures in the two images does not match, which indicates that the frame relationships between the two generated images are poor and that there is no relevance.

[0164] Referring to Figure 8, which is a schematic diagram of generated images related to a sequence according to an embodiment of the present application, according to the method provided in the embodiment of the present application, for the first descriptive phrase "The short sword emitted a white, cold light, and the sweaty girl let out a scream of terror", single image generation and sorting is performed to select the optimal generated image, such as the first image on the left side of Figure 8. For the second descriptive phrase "The girl in the navy blue swordsman's uniform turned around and looked with a lonely expression at the broken maple leaf that had fallen to the ground", single image generation and sorting is performed to generate an image, and then (image) sequence generation and sorting is performed to select the optimal image of the sequence that is more similar to the second descriptive phrase and is also related to the person and elements in the generated image of the first descriptive phrase. Specifically, the second image on the right side of Figure 8 is an example of this. Comparing the sequence diagram composed of the two images in Figure 7 and the sequence diagram composed of the two images in Figure 8, it can be seen that the preceding and succeeding frame relationships are stronger in the latter than in the former, that is, the sequence relationships between the generated images are higher.

[0165] Below, with reference to Figure 4, we will describe the specific implementation process of a closed-loop system for generating continuous illustrations.

[0166] As shown in Figure 4, for multiple descriptive phrases in a novel (e.g., phrase 1, phrase 2, phrase 3), it is necessary to call an image generation model to process the descriptive phrases in order to generate multiple images. Considering that the image generation model is trained on specific image-text pairs, the generation effect for different phrases in different text environments will vary considerably. For example, in the case of a stable-diffusion model, the training images for the model may include works of art, photographs shared on websites, etc.

[0167] In some embodiments, it is necessary to fine-tune the image generation model as a stable diffusion model to satisfy the need to generate images of a specified theme type of novel, thereby training the image generation model to generate images of a corresponding theme type based on the input text content of a theme type. Training samples for the image generation model can be collected from movies and television dramas. Specifically, for example, given movies and television dramas of several theme types such as martial arts or historical themes, frames are randomly extracted from these movies and television dramas to obtain relevant images, and a total of 100 images are extracted from movies and television dramas to be used as training images for the image generation model. At the same time, it is also necessary to collect presentation texts to guide the fine-tuning of the image generation model, and these presentation texts are also provided from movies and television dramas of corresponding theme types, such as martial arts or historical themes.

[0168] Referring to Figure 9, Figure 9 shows an example of an image generation model architecture according to an embodiment of the present application. A cross-modal representation model, trained on image-text pairs, is employed to extract features from the presented word text, where the cross-modal representation model may be a CLIP model. The input to the CLIP model is a sample pair consisting of an image and text, and the specific architecture consists of a text branch and an image branch. Here, the text branch is a text encoder, which may specifically be a transformer encoder structure, and is used to perform encoding processing on the input text to obtain corresponding text features.

[0169] As shown in Figure 9, the main structure of the image generation model consists of three parts. The first part is the pixel space, which contains a Variational AutoEncoder (VAE) located within the pixel space. The second part is the latent space, which contains a diffusion model (Diffusers) and a denoising U-Net model located within the latent space. The third part is the training module, where the training samples exemplify the output of semantic maps, text, representations, and images.

[0170] The initial input to the image generation model is a training image. Each training image x is input to the encoding network ε of a variational autoencoder, and an encoding process called embedding is performed to obtain the corresponding encoded feature Z. Next, the encoded feature Z is mapped to the latent space, and a noise addition process is performed using a diffusion model. That is, the diffusion process shown in Figure 9 is performed. After diffusion noise addition processing over T time steps, the final noise hidden space feature Z is obtained. T Here, the specific process of adding diffuse noise to the latent space is to perform a forward diffuse processing step of the diffuse model on the encoded feature Z, and as the time step T increases, random noise is successively added to the encoded feature Z, until a completely random noise hidden space feature Z is obtained. T The goal is to obtain.

[0171] Subsequently, Hidden Space Feature Z T This is the U-Net model for noise reduction (i.e., the ε shown in Figure 9). θ The image is input to the denoising process, and it is expected that the denoising process will restore the features of the image. For the text corresponding to the collected movie and TV drama images, the CLIP model text branch text encoder (τ in Figure 9) is used. θ By performing encoding processing using ), the corresponding text features can be obtained, and these text features are input to the U-Net model via a transformation module, where the transformation module is the controller.

[0172] In some embodiments, there are two ways to use text features as the input of the U-Net model, and the selection of the method can be realized by the conversion module. The first method is to combine the text features and the latent space feature Z T (the combination shown in FIG. 9) to obtain the combined fusion features and use them as the initial input of the U-Net model. The second method is to first use the latent space feature Z T as the initial input of the U-Net model, and then combine each of the text features with the element input of each sampling layer of the U-Net model to obtain the final input of the corresponding sampling layer.

[0173] The embodiments of the present application use the second method to explain the noise removal process of the U-Net model. That is, first, the latent space feature Z T is used as the initial input of the U-Net model, and then each of the text features is combined with the element input of each sampling layer of the U-Net model to obtain the final input of the corresponding sampling layer, thereby fusing the text features into the image features. The U-Net model is divided into a downsampling part and an upsampling part. The two parts are composed of several sampling layers. The sampling layers of the two parts correspond to each other, and each sampling layer is a cross-attention module (QKV module). Here, the input and output between the cross-attention modules are connected by a skip connection module. By using the attention constraint of the text features in the noise removal process, the final generated image can include the effect of the text features, and the theme type of the generated image can be the same as the theme type of the prompt text.

[0174] Illustratively, as shown in Figure 9, the U-Net model has a total of four QKV modules from right to left, with the two on the right being the downsampling portion and the two on the left being the upsampling portion. Therefore, from right to left, the first QKV module corresponds to the fourth QKV module, and the second QKV module corresponds to the third QKV module. During training of the U-Net model, the output of the first QKV module is input to the second QKV module, and is also associated with the output of the fourth QKV module via a skip connection. Specifically, the outputs of the first QKV module and the fourth QKV module are joined end to end to form the final output of the fourth QKV module. At the same time, the output of the second QKV module is input to the third QKV module, and is further joined with the output of the third QKV module via a skip connection to form the final output of the third QKV module. This completes the denoising process for one time step of the U-Net model.

[0175] As shown in Figure 9, text features and hidden space features Z T The fused feature, formed by combining the two, is then routed through the fourth QKV module of the U-Net model to the hidden space feature Z. T-1 This is obtained. That is, the above process is repeated T-1 times through one time step, i.e., x(T-1) shown in Figure 9, where x represents the number of times denoising is performed. As a result, after a total of T denoising processes by the U-Net model, the finally restored encoded feature Z is obtained, and then the restored encoded feature Z is input to the decoding network D of the variational autoencoder to perform the decoding process and obtain the restored image Obtain JPEG2026514279000003.jpg4170.

[0176] The purpose of fine-tuning the image generation model is to train it to adopt the style associated with illustrations in wuxia novels. In the image generation model fine-tuning process, only the parameters of the U-Net model for denoising in the latent space of the second part need to be trained; all other variational autoencoders and CLIP model parameters for encoding images and text are already trained and do not need to be updated in the image generation model training process. In the image generation model training process, all the training images for fine-tuning are input into the image generation model as full samples and trained for a total of 1000 rounds (steps), with one training round referred to as one iteration.

[0177] In each iteration, due to the limited memory resources of the training equipment, it is not possible to input all samples into the image generation model at once for training. Therefore, all samples are divided into batches for training, and bs samples are input into the image generation model as one batch for training.

[0178] In some embodiments, the loss function for the fine-tuning process of the image generation model employs Mean Squared Error (MSE) loss. That is, the MSE loss between the reconstructed image and the original training image with the added noise distribution is calculated, and specifically, it is expressed as the Mean Squared Error between the fine-tuning training image with noise input and the reconstructed image with the noise-predicted output, and the formula is as follows:

number

[0179] In equation 2 above, y i represents the pixel value of the i-th pixel point in the image, and y i p n represents the predicted pixel value of the i-th pixel point in the image, and n represents the number of full samples.

[0180] Based on the loss value in each round of iterations, a stochastic gradient descent algorithm is used to update the parameters of the U-Net model being trained, returning the data to the U-Net model within the image generation model.

[0181] In some implementations, when training an image generation model, the second part of the diffusion model (Diffusers) process can employ the dreambooth fine-tuning process, where dreambooth is a framework for training the diffusion model to generate images from text. During training, two items, "instance_dir" and "instance_prompt," must be specified on the framework. Here, instance_dir is the path to the instance data, i.e., the directory where the fine-tuning training images are located, and instance_prompt is used to set the theme or style category that requires training, such as wuxia style or historical style. For example, setting the instance_prompt item to "wuxia style" would suffice.

[0182] One round of iterations is completed when training for all N / bs batches is finished. In the training process, an initial learning rate of 0.0005 is adopted, and every 10 rounds of iterations the learning rate becomes 0.1 times the original rate. The decision to continue training is made based on whether the loss function decreases, and the process terminates when the loss value stops decreasing or when a specified number of iterations, e.g., 1000 rounds, is reached. This concludes the introduction of the fine-tuning process for image generation models.

[0183] The image generation model, after fine-tuning is complete, can be used for prediction and generates generated images with the same theme type as the input text and random noise images or random noise images, based on the encoded features of the input text and random noise images with a specific theme type.

[0184] Continuing to refer to Figure 4, after inputting phrases 1, 2, and 3 from the novel script into the closed-loop system, the finely tuned image generation model described above is invoked to generate the corresponding generated images. In the image generation process, the number of generated images can be specified; for example, 10 generated images can be fixedly generated for each phrase. Next, a generation result sequence processing is performed on the 10 generated images for each phrase to select the optimal generated image for each phrase.

[0185] In some embodiments, in order to perform sequence processing on the generated images, a correlation evaluation can be performed on each generated image of a word, i.e., the image-text similarity can be evaluated, and then a sequence correlation evaluation can be performed on the generated images. Here, performing a correlation evaluation on the generated images determines a similarity index between the word and the generated image, and this similarity index represents the degree to which the semantic information of the word is presented on the image.

[0186] Referring to Figure 10, Figure 10 is a diagram showing the process of evaluating the correlation between generated images according to an embodiment of the present invention. First, based on the input phrase and the corresponding generated image, a CLIP model is called and encoding processing is performed on the input phrase to obtain the CLIP text features of the input phrase and the CLIP image features of the generated image. Next, the similarity between the CLIP text features of the input phrase and the CLIP image features of the generated image is calculated to perform a similarity evaluation between the input phrase and the generated image.

[0187] Specifically, the process for performing similarity evaluation between an input phrase and multiple generated images is as follows:

[0188] (1) For the presented words awaiting processing (i.e., input words), key elements are extracted from the presented words. Specifically, the noun elements of the presented words (e.g., real nouns) are retained, and adjectives and adverbs within them are removed.

[0189] (2) Use the CLIP model to extract image features from each generated image.

[0190] (3) Use the CLIP model to extract text features of each noun element within the presented word. For example, if the presented word is "One person is with a gentle cat and a dog," remove the adjective "gentle" and extract the three key elements "cat, dog, person" from the presented word. Thus, a total of three noun elements (features) are retained, which we will refer to here as presented word element features.

[0191] (4) For a given presented word element feature, calculate the cosine similarity between all generated image features and the presented word element feature, and return the cosine similarity as the similarity index. The formula for calculating the cosine similarity is as follows:

number

[0192] In equation 3 above, similarity represents the cosine similarity, θ represents the cosine angle of the cosine similarity, A represents the image features of the generated image, B represents the features of the presented word elements, and A i represents the image features of the i-th generated image, and B i This represents the i-th presented word element feature.

[0193] After determining the similarity index between the input word element features and each generated image, all generated images can be sorted based on the similarity index, and the single generated image with the highest similarity index is considered the optimal image for the input word.

[0194] On the other hand, since it is necessary to ensure consistency in the generated images for multiple input terms, it is necessary to perform sequence correlation evaluation on the generated images for multiple input terms. Here, the specific evaluation process for sequence correlation can be seen in Figure 11, which is a schematic diagram of the principle for performing sequence correlation evaluation on generated images according to the embodiment of the present application. The specific process of sequence correlation evaluation will be described below with reference to Figure 11.

[0195] Step S1 is performed to split a descriptive phrase (containing multiple presented words) from a novel.

[0196] For example, by dividing each presented word, the noun elements contained therein can be identified, and it can be determined whether they are of an element type such as person, tool, or environment. If so, the noun elements are recorded and stored in the sequence element library 1101. In the sequence element library 1101, the recording format of the divided multiple presented words may be "[presented word identifier (ID), image identifier, element category - element identifier - element name - whether the element appears in the image]", for example, "i-th presented word, j-th image, tool - element 1 - cat - appears, tool - element 2 - dog - appears, person - element 1 - person - appears".

[0197] The purpose of recording whether each noun element within a presented word appears is to provide a criterion for sequence judgment of subsequent presented words. Here, the identification of people, tools, and environments can be done using existing business cross-modal multi-label models (or pre-trained cross-modal models (e.g., CLIP models) and multi-label classifiers). The cross-modal model has two input branches: image branching and text branching. This model can also take joint features of images and text as input, or take only image or text features as input and output labels such as knife, sword, horse, etc.

[0198] Referring to Figure 11, first, for the first presented word of a descriptive phrase in a novel, step S2 is executed, calling the image generation model to perform model processing on the presented word and generating multiple images.

[0199] After step S2, step S3 is executed to score the image correlation of the generated images.

[0200] Image-text similarity evaluation is performed on these generated images of the presented words to determine a similarity index between the presented words and each generated image. In other words, the image correlation of the generated images is scored.

[0201] In some embodiments, after step S3, step S6 is performed to select an image.

[0202] Based on the similarity index, the image with the highest similarity index is selected from these generated images and designated as the optimal generated image for the first presented word, denoted by the number j. Next, this optimal generated image is associated with the noun element of the first presented word and saved in the sequence element library. Simultaneously, the noun element of the first presented word is treated as a past element, its features are recorded, and saved in the sequence element library. Here, the association format within the sequence element library can be tool-element1-cat-[presented word identifier-image identifier-image features], tool-element2-dog-[presented word identifier-image identifier-image features], person-element3-person-[presented word identifier-image identifier-image features]. On the other hand, the format for saving the features of past elements can be element1-past element-[presented word identifier-image identifier-past element features].

[0203] After obtaining the optimal generated image for the first presented word, the optimal generated image for the first presented word is stored as a past image in the sequence element library 1101, where the recording format of the past image is "past image - element - [presented word - image identifier - element category - image features]".

[0204] In some embodiments, a descriptive phrase in a novel contains multiple presenting words, and if a descriptive phrase in a novel has a second presenting word, step S2 is performed similarly for the second presenting word, i.e., an image generation model is invoked to perform model processing on the presenting word and multiple images are generated. Then, step S3 is performed, i.e., the image correlation of the generated images is scored.

[0205] Next, step S4 is performed on the generated image of the second presented word to score the cumulative correlation of each element. Then, step S5 is performed to merge and resort the scores.

[0206] Steps S4 and S5 are implemented in the following manner: after performing a similarity evaluation, sorting is performed based on the magnitude of the similarity index; then, a selection is performed using a pre-set similarity index threshold; if there are no generated images with a similarity index greater than the index threshold, it means that the presented word is not suitable for the image, and the generated images with a similarity index greater than the index threshold are determined to be retained images. Next, element information is searched for each retained image. That is, the occurrence status of noun elements in each retained image is determined based on the identification results of noun elements of the presented word in the sequence element library.

[0207] Referring to Figure 11, after performing an element information search on the retained image of the second presented word, if it is determined that a noun element appears in the second presented word, and if the noun element from the second presented word appears in the corresponding retained image, then it is queried whether that noun element exists in the past elements (noun elements of the first presented word) stored in the sequence element library. If it exists, the cumulative correlation of each element is scored based on the retained image. If it is queried that these noun elements do not exist in the sequence element library, it is skipped.

[0208] In some embodiments, the process for scoring the cumulative correlation of each element based on the retained image is as follows: if these noun elements exist in the sequence element library, first, the features of the relevant past element in the sequence element library are obtained; then, corresponding weights are set according to the type of past element; the image-text similarity between the past element features of each element type (environment, person, tool) and the current image (retained image) is calculated; and finally, a weighted summation process is performed on the similarity of each element type based on the weights to obtain the element similarity of the retained image.

[0209] As an example, the corresponding weights are set to 0.5, 0.4, and 0.1 depending on the type of past element (environment, person, tool), and if there is no past element of the corresponding type, the weight of that type is set to 1. In this case, if the similarity between the retained image and the past element features of each type (environment, person, tool) is a, b, and c, respectively, then the element similarity of the final retained image is "0.5a + 0.4b + 0.1c".

[0210] After determining the element similarity of the retained image, the system uses the association between past elements in the sequence element library and the optimal generated image (past image) to search for the corresponding features of the optimal generated image. Then, it calculates the similarity between the current retained image and the past image, which is then used as the past image similarity. In other words, it calculates the similarity between the image features of the optimal image (past image) generated from the previous presented word and the current image (the retained image of the second presented word).

[0211] Referring to Figure 11, after scoring the cumulative correlation of each element based on the retained images in step S4, step S5 is performed to fuse and resort the scores for each retained image. First, all similarities (element similarity, past image similarity) of the retained images are averaged to obtain the sequence similarity of the current retained image, thereby determining the sequence similarity of each retained image of the presented word.

[0212] For each retained image, a corresponding sequence similarity weight and element similarity weight (e.g., 0.5, 0.5) are set, and a weighted addition process is performed on the image-text similarity and sequence similarity of the retained images to obtain the total fusion score of the retained images. The total fusion score of all retained images for the second presented word is calculated, and these retained images are re-sorted based on the total fusion score to obtain and return the final sorted result. Next, an image is selected from the sorted retained images, i.e., the retained image with the highest total fusion score among the sorted images (corresponding to step S6, Select an image) is set as the illustration for the first presented word. Next, the illustration for the second presented word is saved as a new past image in the sequence element library 1101.

[0213] Continuing to refer to Figure 11, after selecting an image for one presented word within a descriptive phrase in the novel each time, the sequence element library needs to be updated to process the next presented word within the same descriptive phrase. The reason for updating the sequence element library is that after selecting an image for one presented word each time, a corresponding new noun element or new illustration may be generated, which are stored in the sequence element library as past element features and past images, respectively. Therefore, the sequence element library is updated dynamically.

[0214] Specifically, the approach to updating the sequence element library is as follows: For each (noun) element within the tools, people, and environments in the sequence element library, if the same noun element reappears later, a dynamic update strategy is employed to update the characteristics of that noun element in the library. For past images in the sequence element library, each time a suggested word is processed and a new optimal image is generated as an illustration, a dynamic update method is used to update the information of the new optimal image in the past image characteristics of the sequence element library.

[0215] Here, the specific formula for dynamically updating the sequence element library is "new_feat = old_feat × w1 + new_image_feat × (1 - w1)". In other words, "new feature of past element in the sequence element library = past element feature in the sequence element × weight + current new image feature × (1 - weight)".

[0216] By pre-setting corresponding weights, the current best noun element features or image features of an image can be incorporated into the sequence element library with the corresponding weights. Considering that past images are updated very frequently while past elements are updated relatively slowly, relatively balanced weights such as 0.6 and 0.4 can be set for past elements in the sequence element library. In this case, the new feature of a past element in the sequence element library is calculated as: Past element feature in the sequence element library × 0.6 + Noun element feature with the same name as the past element feature in the best image × 0.4 = New feature of a past element in the sequence element library.

[0217] For example, the new features of a human element in the sequence element library are calculated as follows: (Human) past element features in the sequence element library × 0.6 + (Human) element features in the current retained image (optimal image) × 0.4 = (Human) element features in the sequence element library.

[0218] Regarding past images in the sequence element library, considering that the single retained image (optimal image) with the highest total fusion score is ultimately selected as the illustration for each presented word, the updates to past images in the database are very frequent, and therefore the updates to image features of past images in the sequence element library are relatively fast. Accordingly, high weights are assigned to the image features of past images in the sequence element library, and correspondingly lower weights, such as 0.8 and 0.2, are assigned to the image features of the current retained image.

[0219] As an example, after processing one presented word, the retained image with the highest fusion total score is used as the illustration for the corresponding presented word, and the image features of the current retained image are added to the database with a weight of 0.2. That is, "Past image features in the sequence element library × 0.8 + Image features of the retained image × 0.2 = New past image features in the sequence element library".

[0220] In some embodiments, an illustration corresponding to the second presented word in a descriptive phrase of a novel is selected, the sequence element library is updated, and then steps S2 to S6 above are performed for the third presented word in a descriptive phrase of a novel, and sequence image selection is performed until all presented words in a descriptive phrase of a novel have corresponding illustrations, and the described sequence image selection results are obtained.

[0221] According to the embodiment of this application, sequence processing is performed on the generated images of each presented word in the text entered by the user, a library of previously generated sequence elements is constructed, sequence correlation evaluation is performed on the generated images of the presented words, and then the images are re-sorted to select them, avoiding inconsistencies between previous and subsequent results. Using the sequence element library constructed by zero-sample denoising as a reference, the generated results are re-sorted for the consistency of related elements to obtain the final sorted result. This fine-grained image selection method is more targeted and useful for presenting elements within descriptive phrases in novels. Unlike general image selection based on the coarse granularity of images based on embedded similarity, the embodiment of this application designs an image selection algorithm with finer element granularity, enabling a delicate evaluation of each noun element and generated image within the presented words.

[0222] Continuing to refer to Figure 4, after performing the generation result sequence processing on the generated images corresponding to each of the phrases 1, 2, and 3, the optimal image corresponding to each phrase can be obtained. Due to differences in the image and text environments, not all texts can generate images successfully. For example, the descriptive element "clappers" in the phrase "The night watchman passes by, beating clappers," and the descriptive element "fiddle" in the phrase "He is fiddling with two red beans in his hands."

[0223] For some texts that are unsuitable for generating illustrations within the input phrase, for example, in a Chinese environment, descriptive techniques such as figurative sentences, parallel sentences, and particles, as well as a wide variety of actions, are difficult to perfectly represent on an image. In the embodiments of this application, relevant data is collected and used to train a phrase generation possibility identification model to identify whether the input phrase is suitable for generation. Referring to Figure 12, Figure 12 is a diagram showing the process of identifying inappropriate phrases as generation results according to the embodiments of this application. First, an image evaluation is performed on the generated image of the input phrase. Here, the image evaluation process is a process of evaluating the similarity between the input phrase and the generated image. For a specific evaluation process, please refer to Figure 10, which will not be repeated here. Next, the phrase generation possibility identification model is called to identify whether the input phrase is suitable for generation. In the image evaluation process, clause splitting is performed synchronously on the input phrase. After clause splitting, clause generation possibility identification is performed.

[0224] A word / phrase generative potential discrimination model is used to determine whether a given word or phrase is suitable for image generation by an image generation model.

[0225] In some embodiments, a single text classification model can be trained as a word generation capability discrimination model, where the text classification model is used to determine whether a given text is suitable for image generation by an image generation model, and determining whether a given text is suitable for image generation by an image generation model requires determining whether the text can generate labelable data by the image generation model.

[0226] Specifically, training a text classification model requires preparing text material as samples. First, all novel texts are collected, and each novel text is split at the sentence level, which includes splitting into S1 sentences at the period level and S2 sentences at the comma level. This yields a total of S3 texts, where S3 is the sum of S1 and S2. These S3 texts are input into an image generation model to generate corresponding generated images. Each text generates 10 generated images with a different random seed (the random seed is used to control the generation process of the image generation model, which generates images of the corresponding theme style based on the theme style of the text), resulting in a total of 10 × S3 generated images. Next, two labels are added to these generated images, where label 1 represents the degree of match between the description of the generated image and the description of the text, and label 2 represents whether the text is an action description text, which will be explained in detail below.

[0227] For label 1, the 10 generated images for each of the 3 texts are evaluated. Specifically, if the descriptions of 3 or more of the 10 generated images for a given text match the description of the text, the text is marked with a 2, indicating that the text is suitable for image generation by the image generation model. If the descriptions of 1 or 2 of the generated images match the description of the text, the text is marked with a 1. If there are no generated images that match the description of the text, the text is marked with a 0, indicating that it should be executed.

[0228] In some embodiments, the criteria for determining whether the description of the generated image matches the description of the text is that all specific objects or elements (excluding abstract objects or elements) described in the text appear, or, if five or more specific objects or elements described in the text appear, 90% of the number of objects or elements appear.

[0229] For example, in the text "His clothes were as black as ink," the descriptive element "ink" is an abstract object, and if black clothes appear in the generated image, it can be considered to match the descriptive object in the text. As another example, the text "The man dragged the box into the alley, stopped next to a taxi beside a fruit stand, and spoke to the driver through the half-open window of the taxi" contains a total of seven elements: "man, box, alley, fruit stand, taxi, window, driver." In this case, if six of these elements (i.e., 7 x 90%) do not appear in the generated image, it will be considered not to match the descriptive object in the text.

[0230] In the case of label 2, it is directly determined whether each of the S3 texts is an action description text. If it is an action description text, it is marked with 1; otherwise, it is marked with 0.

[0231] In some embodiments, the criterion for determining whether text is action description text is whether the text (phrase) jointly describes a single event or scene with multiple subtexts (sections).

[0232] For example, in the text, "He pinched one end of the thread with the index finger and thumb of his withered left hand, and held it to the small eye of the needle in the lamplight, slowly stretching the end of the thread. The first time it came off, the second time the end of the thread became distorted, and the third time the end of the thread split, creating a small branch that blocked the eye of the needle. He tried again and again, but finally gave up," it can be seen that the text employs multiple subtexts to jointly describe the scene, and therefore it can be determined that the text is an action description text.

[0233] The above process generates label 1 and label 2 for each of S3 texts. These labeled texts can be used as training samples to train a text classification model, which includes two text classifiers used to identify whether the text content is suitable for illustration generation and whether it is action description text, respectively.

[0234] In some embodiments, two text classifiers can be implemented using a convolutional neural network and a multi-class prediction layer. First, all training samples containing labels 1 and 2 are input into a BERT model and encoded to obtain the text encoding features of each training sample. Next, the text encoding features of all training samples are input into a two-layer convolutional neural network with a 1x1 convolutional kernel to perform convolution. Finally, the obtained convolutional features are input into a multi-class prediction layer to perform predictions.

[0235] The structures of the two text classifiers are shown in Tables 1 and 2, respectively.

[0236] [Table 1]

[0237] [Table 2]

[0238] As can be seen from the two tables above, the first layer of the model is the input layer, and its specific structure is a two-layer convolutional neural network with a 1x1 convolutional kernel. After inputting the text features of the training samples into the model's input layer and performing a convolution, it outputs two convolutional text features of size 1x512. Next, these two output text features are input to the model's second layer (fusion layer) and a convolution is performed to output a convolutional text feature of size 1x1024. Here, the specific structure of the fusion layer is a convolutional neural network with a 1x1 convolutional kernel. Next, the 1x1024 convolutional text feature is input to the multi-class prediction layer for prediction. Here, the activation function of the prediction layer is the softmax function, and the predicted values ​​output from the softmax function of the prediction layer are mapped to one-hot encoded format and output. That is, as shown in the table, two classification prediction results, Class 1 and Class 2, are output.

[0239] Identifying whether text is suitable for illustration generation is determined based on the degree of match between the image description generated by the text and the text description, and therefore corresponds to Label 1. Since Label 1 has three possible label values ​​("0, 1, 2"), the text classifier treats the predicted values ​​of Label 1 in the training samples (i.e., Class 1 in the table above) as a 3-classification and outputs a 1x3 3-bit one-hot encoding. Similarly, identifying whether text is an action description phrase is determined by whether multiple subtexts (sections) jointly describe a single event or scene, and corresponds to Label 2. Since Label 2 has only two possible label values ​​("0, 1"), the text classifier treats the predicted values ​​of Label 2 in the training samples (i.e., Class 2 in the table above) as a 2-classification and outputs a 1x2 2-bit one-hot encoding.

[0240] For example, if the Class 1 prediction output by a training sample is a label value of 0, it means that the predicted value of the first bit of the one-hot encoding within the output Class 1 is greater than 0.5, and the predicted values ​​of the second and third bits are very small, so the classification result that the prediction layer softmax function ultimately outputs can be expressed as "100". Similarly, if the Class 1 prediction output by a training sample is a label value of 1, it means that the predicted value of the second bit of the one-hot encoding within the output Class 1 is greater than 0.5, and the predicted values ​​of the first and third bits are very small, so the classification result that the prediction layer softmax function ultimately outputs can be expressed as "010". If the Class 1 prediction output by a training sample is a label value of 2, it means that the predicted value of the third bit of the one-hot encoding within the output Class 1 is greater than 0.5, and the predicted values ​​of the second and third bits are very small, so the classification result that the prediction layer softmax function ultimately outputs can be expressed as "001". At the same time, if the Class 2 prediction result output by a certain training sample is label 0, the final classification result output by the prediction layer softmax function can be expressed as "10", and if the Class 2 prediction result output by a certain training sample is label 1, the final classification result output by the prediction layer softmax function can be expressed as "01".

[0241] In some implementations, all training samples are input into the text classifier for training. The training process consists of a total of 60 iterations, and in each round of iterations, due to the limited memory resources of the training equipment, it is not possible to input all training samples into the text classifier at once. Therefore, all training samples are divided into batches for training, and the text classifier parameters are updated by inputting bs samples (batch_size) as one batch. Specifically, the text classifier parameters are updated G / bs times (G is the total number of training samples) in each iteration, meaning that bs training samples are selected from the G training samples each time to perform training predictions, and then two classification cross-entropy loss functions are calculated. That is, after calculating the classification loss for each training sample, the average classification loss of all training samples is calculated to obtain the classification loss for each batch, and then the loss function is fed back into the text classifier network using the stochastic gradient descent algorithm to update the network parameters.

[0242] Here, the classification cross-entropy loss function is expressed as follows:

number

[0243] In equation 4 above, p ic represents the predicted probability that the prediction result of training sample i belongs to category c, and y ic This indicates whether the predicted label value of sample i is c, and if it is c, then y ic = 1, otherwise it is 0. N is the sample data for each model update, i.e., batch_size, and M is the number of predicted classifications; in the case of 3-class prediction, M is 3.

[0244] Continuing to refer to Figure 12, the word generation possibility discrimination model can be used for prediction after training is complete. That is, it identifies input words, determines whether the current input words are unsuitable for image generation, and requires further processing operations such as deleting unsuitable words or splitting complex words that are unsuitable for generation into clauses. Here, the process of calling the word generation possibility discrimination model to identify input words involves two decision processes: an initial decision and a further decision, which will be explained in detail below.

[0245] First, in the initial judgment, the input phrases are input into the phrase generation possibility discrimination model for identification. The model classifies these input phrases into three categories and obtains a classification prediction value (generation possibility prediction value) for each input phrase, namely 0, 1, or 2. This corresponds to label 1 during training of the phrase generation possibility discrimination model and is used to represent the degree to which the input phrase is suitable for illustration generation. At the same time, the phrase generation possibility discrimination model predicts whether the input phrase is an action description phrase and obtains a corresponding action prediction value, namely 0 or 1. This corresponds to label 2 during training of the phrase generation possibility discrimination model and is used to represent whether the input phrase is an action description phrase.

[0246] In some embodiments, the initial decision criteria for the word recognition capability model are as follows: If the predicted generative probability value of an input word is 0 or 1, it is determined that an image cannot be generated from the input word, and the input word should be deleted. If the predicted generative probability value of an input word is 2 and the corresponding predicted action value is 1, it is determined that the input word has a high failure rate in generation, and the input word should also be deleted. If the predicted generative probability value of an input word is 2, the predicted action value is 0, and the input word contains multiple clauses, it is determined that further processing is required for the input word, i.e., clause splitting should be performed.

[0247] Referring to Figure 12, considering that in a Chinese environment there are many sentences containing multiple clauses that can be separated by commas, the input phrases will contain a large number of phrases that require clause splitting. On the other hand, there are consecutive clauses within the phrases to be split that describe partially similar content, and these parts do not need to be split. For example, the phrase "The pine forest stretches on and on, and the green pines spread to the top" may be split into two clauses after the initial judgment, but since the content described by the two resulting clauses is the same, the generated image is likely to be the same as well, and in reality, such sentences do not need to be split quadratically. Therefore, it is necessary to perform further judgment on the results of the above initial judgment for some input phrases, that is, it is necessary to perform clause generation possibility identification on the clauses that have been split from the input phrases.

[0248] In some embodiments, the specific process for making further decisions on clauses separated from the input phrase is as follows:

[0249] (1) After initial judgment, the input phrase is divided into multiple sections. The generative possibility identification model is called to perform generative possibility identification on each divided section, thereby obtaining the classification prediction value (generative possibility prediction value) for each section. Sections with a generative possibility prediction value of 2 are retained and recorded as generative possibility sections.

[0250] For example, after making an initial judgment on the original input phrase, it is divided into 10 clauses. Then, a generative possibility identification model is called to perform generative possibility identification on each of these 10 divided clauses, and a predicted generative possibility value is obtained for each clause. Next, clauses with a predicted generative possibility value of 2 are retained and recorded as generative possibility clauses. For example, if there are 5 clauses with a predicted generative possibility value of 2, 5 generative possibility clauses are determined.

[0251] (2) Each potential generation clause is denoted as i. Next, the image generation model is invoked to calculate the similarity between all generated images generated from the input phrase and potential generation clause i. The process for calculating the similarity between potential generation clause i and each generated image can be found in Figure 8 and will not be repeated here. This allows us to obtain a similarity index between potential generation clause i and each generated image. Next, we determine whether the largest similarity index is greater than the similarity index threshold (denoted as thr1). If the largest similarity index is greater than the similarity index threshold thr1, we retain the generated image corresponding to the largest similarity index, denoted as j, and match it with potential generation clause i to obtain an image-text pair, denoted as (i,j).

[0252] Continuing from the example above, each of these five generative possibilities can be given 10 similarity indices by calculating the (cosine) similarity of each of these possibilities with all the generated images (e.g., 10) generated from the input phrase. Next, it is determined whether the maximum value of the 10 similarity indices is greater than the similarity threshold thr1. If it is greater, the generated image corresponding to the largest similarity indices is associated with the generative possibility to obtain an image-text pair. If it is not greater, it means that all of these 10 similarity indices are less than the similarity threshold thr1, and the generative possibility cannot find a corresponding generated image.

[0253] (3) According to step (2), all generative clauses i can obtain multiple image-text pairs (i,j) under the condition that the similarity index threshold thr1 is satisfied.

[0254] Continuing from the example above, for each of the five possible generation clauses, after searching for corresponding image-text pairs from all the generated images of the input phrase, only four possible generation clauses ultimately find corresponding generated images, resulting in four corresponding generated images. In other words, there are a total of four image-text pairs.

[0255] (4) Perform variance calculation on the corresponding generated images within these image-text pairs (i,j). Variance represents the similarity of features between multiple images, and comparing the similarity of generated images means comparing the closeness of their variances; the smaller the difference in variance, the more similar the images are. If the variance is greater than the variance threshold thr2, it means that the differences in the generative possibility clauses corresponding to these generated images are very large, and further division and generation are necessary.

[0256] For illustrative purposes, referring to Figure 13, which shows an example of determining the differences between generated images according to an embodiment of the present invention. Four executable clauses corresponding to four image-text pairs specifically describe, "They walked on and on, and it was already near dawn when they arrived at a small liquor store at the foot of the mountain. They shook the shopkeeper who was standing by the door, and the boy in the red clothes showed him the sword he held," and the four corresponding generated images are shown in Figure 13. When a variance calculation is performed on these four images, it is found that the variance is greater than the variance threshold, and the content displayed in each generated image is different. In this case, it is ultimately determined that the input phrase needs to be split and generated separately.

[0257] In some embodiments, the method for calculating the variance of multiple generated images may involve first calculating the variance of each generated image, and then determining the total variance of the multiple generated images. The calculation process is specifically as follows: First, grayscale processing is performed on each generated image to obtain a grayscale map. For each grayscale map of each generated image, the average value of the grayscale values ​​of the pixel points in each row of the generated image is calculated sequentially, and the average value of the grayscale values ​​of the pixel points in each row is recorded. Next, the variance is calculated on all the obtained average values, and the resulting variance is the feature value of the generated image. After calculating the feature value (variance) for each generated image, they are compared with each other, the difference in variance between any two generated images is checked, and the one with the largest difference in variance is taken as the total variance of the multiple generated images.

[0258] In some embodiments, the method for calculating the variance of multiple generated images may be to directly calculate the variance of multiple images, and the specific calculation process is as follows: First, grayscale processing is performed on each generated image to obtain a grayscale map, the sum of the grayscale values ​​of all pixel points in each grayscale map is determined, and then the variance of these sums of grayscale values ​​is directly calculated to obtain the variance of the multiple generated images.

[0259] In some embodiments, the method for determining the similarity index threshold thr1 can be obtained by calculating recall and searching based on a large number of image-text pairs. Specifically, a certain number (e.g., 10,000 pairs) of image-text pairs are collected, then threshold points are set with a step length of 0.1 units between 0 and 1, and then the similarity index for each image-text pair within these 10,000 pairs is calculated at each different threshold point. Here, the specific method for calculating the similarity index can be found in the image-text similarity calculation process shown in Figure 8, and will not be repeated here. Next, the number of image-text pairs whose similarity index is greater than the threshold point is statistically calculated to determine the proportion of the number of image-text pairs whose similarity index is greater than the threshold point to the total number of image-text pairs (10,000 image-text pairs), and this proportion is taken as the recall at the corresponding threshold point. When the recall at a certain threshold point reaches 80%, the search process is stopped, and this threshold point is taken as the similarity index threshold thr1.

[0260] In some embodiments, the method for determining the variance threshold thr2 can be obtained based on the similarity index threshold thr1. After determining the similarity threshold thr1, it is possible to identify image-text pairs within 10,000 image-text pairs that are greater than the similarity threshold thr1, calculate the variance of the corresponding images within these image-text pairs, and use this variance as the variance threshold thr2.

[0261] For example, if the set threshold is 0.6, the similarity index for each image-text pair in 10,000 image-text pairs is calculated. Then, statistically, it is determined that 8,000 image-text pairs have a similarity index greater than the threshold, and that the proportion of image-text pairs with a similarity index greater than the threshold is 80% of the total number of image-text pairs (10,000 image-text pairs). At this point, the search is stopped, 0.6 is set as the similarity index threshold thr1, and then the variance of the corresponding images in these 8,000 image-text pairs is calculated based on the similarity index threshold thr1. That is, the variance of the 8,000 images is calculated, and the finally calculated variance result is set as the variance threshold thr1.

[0262] Continuing to refer to Figure 12, after making further decisions on the segmented clauses within the input phrase, that is, after performing clause generative possibility identification, the phrase generative possibility identification model can complete the identification of phrases with inappropriate generation results for the input phrase and determine which input phrases' generated images should be retained, which input phrases should be deleted because they are unsuitable for illustration generation, and which phrases' generated images should be further segmented and regenerated because they are inappropriate.

[0263] Continuing to refer to Figure 4, the word generation possibility discrimination model is called to perform word recognition of inappropriate results for the input words (word 1, word 2, word 3), and then a decision can be made such as "keep word 1," "split word 2," or "delete word 3." For "split word 2," word 2, "When the two arrived at the small liquor store at the foot of the mountain, it was already close to dawn, and the boy, without regard for the details, woke the shopkeeper from his futon, threw a large silver ingot at him, and ordered pheasant and mushroom soup," is split into three sections: "small liquor store at the foot of the mountain," "When the two arrived, it was already close to dawn," and "the boy ordered pheasant and mushroom soup." Next, the image generation model is called to perform secondary generation for each of the three sections, and then the generation result sequence processing is performed on the generated images corresponding to the three sections to obtain the three corresponding generated images, which become the final sequence diagram of word 2.

[0264] This allows the system to automatically drive an image generation model to perform secondary generation. For divisible phrases, the image generation model is invoked to generate secondary images, then it is determined whether the generated images are unsuitable. This process continues until no unsuitable phrases are identified. Next, the generation result sequence processing is performed to obtain the optimal image. The optimal image of the retained phrases is then combined with the optimal image generated secondary from the divisible phrases to finally obtain a sequence diagram of all phrases.

[0265] According to the embodiment of the present invention, a closed-loop generation system is constructed based on the generation-image text generation feasibility evaluation, identification feasibility evaluation, and regeneration after adjustment of generated phrases. This system automatically divides and deletes descriptions of novel scripts, selects phrases and long sentences unsuitable for image generation, divides these long sentences to obtain multiple sections, then performs secondary image generation at a finer granularity for the sections, and performs sequence processing to select the optimal image. This realizes rational evaluation of the granularity of sections and fine-grained image selection for presented words, eliminating the need for manual interaction each time an image is selected. This effectively solves the problem of irrational single-step image generation for presented words and the presence of missing generated elements in the generated image, improving the correlation of the overall image generation effect. Meanwhile, in the image selection process, a library of previously generated sequence elements is constructed, and using the sequence element library constructed by zero-sample noise reduction as a reference, the noun element matching of the generated images is re-sorted to obtain the final sort result, avoiding the problem of mismatch between preceding and succeeding pixels in the generation re-sort result of the sequence correlation of presented words.

[0266] Hereinafter, an exemplary structure in which the image processing apparatus 453 provided in the embodiments of the present application is implemented as a software module will be described. In some embodiments, as shown in FIG. 2, the software module in the image processing apparatus 453 stored in the memory 450 includes an acquisition module 4531, a mapping module 4533, and a determination module 4532. The acquisition module 4531 is configured to acquire a processing-wait prompt word. The mapping module 4533 is configured to acquire the text features of the processing-wait prompt word and map the text features to a generation possibility index and a description type of the processing-wait prompt word. The generation possibility index is used to represent a score indicating that the processing-wait prompt word can be used for illustration generation. The acquisition module 4531 is further configured to acquire similar images corresponding to each of the plurality of clauses in response to the generation possibility index being greater than an index threshold, the description type indicating that the processing-wait prompt word does not contain a verb, and the processing-wait prompt word containing a plurality of clauses. The image text similarity between the clause and the corresponding similar image is greater than an image text similarity threshold. The determination module 4532 is configured to determine the image difference degree between the similar images corresponding to each of the plurality of clauses. The determination module 4532 is further configured to use the similar images corresponding to each of the plurality of clauses as illustrations for the corresponding clauses in response to the image difference degree being less than an image difference degree threshold.

[0267] In some embodiments, the determination module 4532 is further configured to continue to divide the plurality of clauses into a plurality of new processing-wait prompt words in response to the image difference degree being greater than or equal to an image difference degree threshold.

[0268] In some embodiments, the acquisition module 4531 is further configured to convert the processing-wait prompt word into a mark sequence, call a semantic understanding model based on the mark sequence to perform encoding processing, and obtain the text features of the processing-wait prompt word.

[0269] In some embodiments, the acquisition module 4531 is configured to further perform the following steps: call a convolutional network in a first text classifier to perform a convolution operation on at least one noun element to obtain a first convolutional feature; call a multiclass layer in the first text classifier to map the first convolutional feature to a first probability of a plurality of generative index candidates; and set the generative index candidate corresponding to the highest first probability as the generative index of the awaiting presented word, the generative index being used to indicate that the awaiting presented word is available to generate an illustration score; and call a convolutional network in a second text classifier to perform a convolution operation on at least one noun element to obtain a second convolutional feature; call a multiclass layer in the first text classifier to map the second convolutional feature to a second probability of a plurality of description types; and set the description type corresponding to the highest second probability as the description type of the awaiting presented word, the description types including verb-containing types and verb-non-verb-containing types.

[0270] In some embodiments, the acquisition module 4531 is further configured to delete a pending suggestion in response to the possibility index being greater than the index threshold, the description type indicating that the pending suggestion does not contain a verb, and the pending suggestion contains multiple clauses.

[0271] In some embodiments, the acquisition module 4531 is further configured to store illustrations of pending suggestion words in the text illustration sequence in the order they were generated, in response to the possibility of generation being below an index threshold, and different pending suggestion words are extracted sequentially from the text.

[0272] In some embodiments, the decision module 4532 is further configured to perform the following processes for each similar image: determine the grayscale mean of the pixels in each row of the similar image, and combine the grayscale mean of the pixels in each row as image features of the similar image; determine the variance of the image features of the similar image corresponding to each of the multiple nodes, and use the variance as the degree of image difference between the similar images corresponding to each of the multiple nodes.

[0273] In some embodiments, the decision module 4532 is further configured to perform the steps of: obtaining an image-text pair sample set, wherein the image-text pair sample set comprises a plurality of image-text pairs, each image-text pair comprising a sample presentation word and a sample-similar image; determining the recall of the image-text pair sample set at the current threshold point in ascending order of a plurality of predetermined threshold points, wherein the recall is the ratio of the number of reproduced image-text pairs to the total number of plurality of image-text pairs, and the image-text similarity between the sample presentation word and the sample-similar image in the reproduced image-text pair is greater than or equal to the current threshold point; and determining the current threshold as the image-text similarity threshold in response to the recall at the current threshold point being greater than or equal to the recall threshold, determining the variance of the image features of the sample-similar images in the reproduced image-text pair, and using the variance of the image features of the sample-similar images in the reproduced image-text pair as the image difference threshold.

[0274] In some embodiments, the acquisition module 4531 is further configured to acquire multiple generated images of a pending presented word in response to the pending presented word being a suitable word, to determine an illustration for the pending presented word from the multiple generated images, and to save the illustration for the pending presented word, wherein a suitable word is a word that does not satisfy an inappropriate word condition, and the inappropriate word condition includes one of the following: the potential generation index is greater than the index threshold, the description type indicates that the pending presented word does not contain a verb, and the pending presented word contains multiple clauses, the image difference is greater than or equal to the image difference threshold, and the potential generation index is greater than the index threshold and the description type indicates that the pending presented word contains a verb.

[0275] In some embodiments, the decision module 4532 is configured to further determine the image-text similarity between each of a plurality of generated images and the pending presented word, to designate a generated image with an image-text similarity greater than an image-text similarity threshold as the retained image, to query the past element features of the noun element from the database in response to the retained image containing a noun element in the pending presented word and at least one past element in the database containing a noun element, to determine the element similarity between the retained image and the past element features, to query the image features of past illustrations of past presented words from the database, to determine the image similarity between the retained image and the past illustration based on the image features of the past illustration and the image features of the retained image, to obtain a total fusion score for the retained image by weighting the element similarity and image similarity, and to designate the retained image corresponding to the highest total fusion score as the illustration for the pending presented word.

[0276] In some embodiments, the decision module 4532 is further configured to acquire image features corresponding to each of a plurality of generated images, identify noun elements within the awaiting presented words, perform encoding on the noun elements to obtain presented word element features, determine the cosine similarity between the image features of each generated image and the presented word element features, and use the cosine similarity as the image-text similarity between the generated image and the awaiting presented word.

[0277] In some embodiments, the decision module 4532 is further configured to determine the image-text similarity between each of the multiple generated images and the awaiting presentation word, and to use the generated image with the highest image-text similarity as the illustration for the awaiting presentation word.

[0278] In some embodiments, the decision module 4532 is configured to perform the steps of: identifying common elements between a retained image and a noun element; querying a database for past element features corresponding to the common elements, wherein the database includes past element features of past presented words.

[0279] In some embodiments, the determination module 4532 is further configured to perform the steps of: determining the element similarity between a retained image and each of different types of past element features, wherein the types of past elements include people, environments, and tools; and weighting and summing the element similarities between the retained image and each of the different types of past element features to obtain an element similarity between the retained image and the past element features.

[0280] In some embodiments, the decision module 4532 is further configured to treat noun elements within a pending presented word as past elements, store past elements and corresponding past element features in a database, treat illustrations of pending presented words as past images, and store past images in a database.

[0281] In some embodiments, the decision module 4532 is further configured to update the database by weighting the element features of noun elements appearing in the retained image with the past element features of past elements with the same name in the database, and replacing the pre-update past element features with the resulting updated past element features, if the pending suggested word is not the first suggested word extracted from the text; and by weighting the image features of the retained image with the image features of past illustrations in the database, and replacing the pre-update image features with the resulting updated image features.

[0282] In some embodiments, the acquisition module 4531 further performs encoding on the pending presented word to obtain the text features of the pending presented word and the image features corresponding to the text features of the pending presented word. The system is configured to obtain noisy image features by applying noise to image features, to obtain fused features by fusing text features and noise image features, to obtain reconstructed image features by applying noise reduction to the fused features, and to obtain multiple generated images by applying decoding to the reconstructed image features.

[0283] Embodiments of the present application provide a computer program product including a computer-executable instruction or computer program, the computer-executable instruction or computer program being stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instruction or computer program from the computer-readable storage medium and executes it, thereby causing the electronic device to perform the image processing method described in the embodiment of the present application.

[0284] Embodiments of the present application provide a computer-readable storage medium in which computer-executable instructions or computer programs are stored, and when executed by a processor, the computer-executable instructions or computer programs cause the processor to execute an image processing method provided in embodiments of the present application, for example, the image processing method shown in Figures 3A to 3K.

[0285] In some embodiments, the computer-readable storage medium may be RAM, ROM, flash memory, magnetic memory, optical disk, or CD-ROM, and may be a variety of devices including one or any combination of the above-mentioned memories.

[0286] In some embodiments, computer-executable instructions can take the form of programs, software, software modules, scripts, or code, and can be written in any form of programming language (including compiled or interpreted languages, declarative or procedural languages), and can be arranged in any form, including independent programs, modules, components, subroutines, or other units suitable for use in a computing environment.

[0287] For example, a computer-executable instruction may be configured to run on a single electronic device, on multiple electronic devices located at a single location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0288] As described above, in addition to selecting illustrations for the generated images based on the initial presented words, a word generation possibility identification model is used to combine the generated image distribution status with the presented word description and generated image to automatically identify presented words with low illustration quality and provide processing feedback. Specifically, a generation possibility evaluation is performed on the presented words, and the description type of the presented word is determined to identify whether the presented word is a detailed description and whether it is suitable for illustration generation. Next, presented words awaiting processing are automatically divided and deleted, and words and long sentences unsuitable for image generation are selected. These long sentences are divided to obtain multiple sections, and then secondary image generation is performed on the sections at a finer granularity, and sequence processing is performed to select the optimal image. This effectively solves the problem that single-step image generation from presented words is irrational and that generated images have missing generated elements. Furthermore, rational evaluation of the section granularity of presented words awaiting processing and image selection at a finer granularity are realized, resulting in more rational generated images and improved correlation of the overall generation results of presented words awaiting processing. On the other hand, in the image selection process, a library of previously generated sequence elements is constructed, and using the sequence element library constructed by zero-sample denoising as a reference, the noun element consistency of the generated images is re-sorted to obtain the final sort result, thus avoiding the problem of inconsistency between the preceding and succeeding results in the generated re-sort result of the sequence correlation of presented words.

[0289] The foregoing is merely an example of the present application and is not intended to limit the scope of protection. Any modifications, equivalent substitutions, and improvements made in the spirit and within the scope of the present application shall be included within the scope of protection.

Claims

1. An image processing method performed by an electronic device, Steps to obtain a word to be processed, A step of obtaining the text features of the pending suggestion word, and mapping the text features to a generative probability index and description type of the pending suggestion word, wherein the generative probability index is used to represent a score that the pending suggestion word can be used to generate an illustration. Steps to obtain similar images corresponding to each of the multiple clauses in response to the fact that the generative probability index is greater than an index threshold, the description type indicates that the pending presentation word does not contain a verb, and the pending presentation word contains multiple clauses, wherein the image-text similarity between the clause and the corresponding similar image is greater than an image-text similarity threshold, The steps include determining the degree of image difference between similar images corresponding to each of the aforementioned multiple sections, An image processing method comprising the step of, in response to the image difference being smaller than an image difference threshold, setting a similar image corresponding to each of the plurality of sections as an illustration for the corresponding section.

2. The aforementioned image processing method is: The further step includes, in response to the image difference being greater than or equal to the image difference threshold, subsequently dividing the plurality of clauses into a plurality of new pending presentation words, The image processing method according to claim 1.

3. The step of obtaining the text features of the aforementioned pending presented word is: The steps include converting the aforementioned pending word into a mark sequence, The step includes calling a semantic understanding model based on the mark sequence and performing encoding processing to obtain the text features of the processing-awaited presented words, The step of mapping the text features to the probability index and description type of the awaiting presented word is: The steps include: calling a convolutional network within the first text classifier to perform a convolution operation on the text features to obtain a first convolutional feature; calling a multi-classification layer within the first text classifier to map the first convolutional feature to the first probability of a plurality of candidate generative indicators; and using the candidate generative indicator corresponding to the highest first probability as the generative indicator for the word awaiting processing. The steps include: calling a convolutional network in a second text classifier to perform a convolution operation on the text features to obtain a second convolutional feature; calling a multi-classification layer in the first text classifier to map the second convolutional feature to the second probability of a plurality of description types, and setting the description type corresponding to the highest second probability as the description type of the pending presented word, wherein the description types include types that include verbs and types that do not include verbs; The image processing method according to claim 1.

4. The aforementioned image processing method is: The further step includes deleting the pending suggestion in response that the generation probability index is greater than the index threshold and the description type indicates that the pending suggestion contains a verb. The image processing method according to claim 1.

5. The aforementioned image processing method is: In response to the fact that the potential generation index is less than or equal to the index threshold, the further step includes storing the illustrations of the pending suggestion words in the text illustration sequence in the order they were generated, wherein different pending suggestion words are sequentially extracted from the text. The image processing method according to claim 1 or 2.

6. The step of determining the degree of image difference between similar images corresponding to each of the aforementioned multiple sections is: For each of the aforementioned similar images, The process involves determining the grayscale average value of pixels in each row within the similar image, and combining the grayscale average values ​​of pixels in each row as image features of the similar image. The process includes: determining the variance of the image features of similar images corresponding to each of the aforementioned multiple sections, and using the variance as the degree of image difference between the similar images corresponding to each of the aforementioned multiple sections; The image processing method according to any one of claims 1 to 5.

7. The aforementioned image processing method is: A step of obtaining a sample set of image-text pairs, wherein the sample set of image-text pairs includes a plurality of image-text pairs, and each image-text pair includes a sample presentation word and a sample similar image. A step of determining the recall rate of the image-text pair sample set at the current threshold point in ascending order of a plurality of predetermined threshold points, wherein the recall rate is the ratio of the number of recalled image-text pairs to the total number of the plurality of image-text pairs, and the image-text similarity between the sample presentation word and the sample-like image in the recalled image-text pair is greater than or equal to the current threshold point. The steps include determining the current threshold point as the image-text similarity threshold in response to the fact that the recall rate at the current threshold point is equal to or greater than the recall threshold, The steps include determining the variance of the image features of the sample-like images within the reproduced image-text pair, and using the variance of the image features of the sample-like images within the reproduced image-text pair as the image difference threshold, The image processing method according to claim 6.

8. The aforementioned image processing method is: The process further includes the steps of obtaining multiple generated images of the pending presentation word in response to the pending presentation word being an appropriate word, determining an illustration for the pending presentation word from the multiple generated images, and saving the illustration for the pending presentation word, wherein the appropriate word is a word that does not satisfy the inappropriate word condition, and the inappropriate word condition includes at least one of the following: the probability of generation is greater than the index threshold, the description type indicates that the pending presentation word does not contain a verb, and the pending presentation word contains multiple clauses, the image difference is greater than or equal to the image difference threshold, and the probability of generation is greater than the index threshold, and the description type indicates that the pending presentation word contains a verb. The image processing method according to any one of claims 1 to 7.

9. If the pending presentation word is not the first presentation word extracted from the text, the step of determining the illustration for the pending presentation word from the multiple generated images is: The steps include determining the image-text similarity between each of the plurality of generated images and the pending presented word, and setting the generated image corresponding to an image-text similarity greater than the image-text similarity threshold as the retained image, The steps include: In response to the fact that the retained image contains a noun element within the awaiting presented word and that at least one past element in the database contains the noun element, querying the database for the past element features of the noun element; The steps include: determining the element similarity between the retained image and the past element features; querying the database for the image features of the past illustrations of the past presented words; and determining the image similarity between the retained image and the past illustrations based on the image features of the past illustrations and the image features of the retained image; The steps include obtaining a total fusion score for the retained images by weighting and adding the element similarity and the image similarity, The step of determining the retained image corresponding to the highest fusion total score as the illustration for the pending presented word, The image processing method according to claim 8.

10. The step of determining the image-text similarity between each of the plurality of generated images and the processing-awaited presented word is: The steps include: obtaining image features corresponding to each of the multiple generated images; The steps include: identifying noun elements within the awaiting presented word, performing encoding processing on the noun elements, and obtaining presented word element features; The process includes the steps of determining the cosine similarity between the image features of each generated image and the present word element features, and using the cosine similarity as the image-text similarity between the generated image and the present word awaiting processing. The image processing method according to claim 9.

11. If the pending presentation word is the first presentation word extracted from the text, the step of determining the illustration for the pending presentation word from the plurality of generated images is: The process includes determining the image-text similarity between each of the plurality of generated images and the pending presentation word, and using the generated image corresponding to the highest image-text similarity as an illustration for the pending presentation word. The image processing method according to claim 9.

12. The aforementioned past element feature is a feature of the past element within the aforementioned past-presented word, The step of querying the database for the past element features of the noun element is: The steps include identifying common elements between the retained image and the noun element, A step of querying the database for past element features corresponding to the common element, wherein the database includes the past element features of the previously presented word, The image processing method according to claim 9.

13. The step of determining the element similarity between the retained image and the past element features is: A step of determining the element similarity between the retained image and each of the past element features of a different type, wherein the type of past element includes people, environments, and tools. The process includes the step of obtaining an element similarity between the retained image and the past element features by weighting and adding the element similarities between the retained image and each of the past element features of different types. The image processing method according to claim 9.

14. The aforementioned image processing method is: If the pending suggestion word is the first suggestion word extracted from the text, the database is used. The noun element within the processing-awaited presentation word is treated as the past element, and the past element and its corresponding past element features are stored in the database. The process further includes the step of updating the database by saving the illustration of the pending word as a past image and saving the past image to the database. The image processing method according to claim 9.

15. The aforementioned image processing method is: If the aforementioned pending suggestion is not the first suggestion extracted from the text, the database is, The element features of noun elements appearing in the aforementioned retained image are weighted and added together with the past element features of past elements with the same name in the aforementioned database, and the updated past element features obtained replace the previous past element features. The update further includes the step of performing a weighted addition of the image features of the retained image and the image features of the past illustrations in the database, and replacing the image features before the update with the updated image features obtained. The image processing method according to claim 9.

16. The step of obtaining multiple generated images of the aforementioned pending presentation word is: The steps include: performing an encoding process on the pending presented word to obtain the text features of the pending presented word and the image features corresponding to the text features of the pending presented word; The steps include: performing noise addition processing on the aforementioned image features to obtain noisy image features; The steps include: a step of fusing the aforementioned text features and the aforementioned noise image features to obtain a fused feature; The steps include: performing noise reduction processing on the aforementioned fused features to obtain reconstructed image features; The process includes the step of performing a decoding process on the reconstructed image features to obtain a plurality of generated images. The image processing method according to any one of claims 1 to 15.

17. An image processing device, It comprises an acquisition module, a mapping module, and a decision module, The acquisition module is configured to acquire a pending word, The mapping module is configured to acquire the text features of the pending presented word and to map the text features to the generative possibility index and description type of the pending presented word, the generative possibility index being used to represent a score that the pending presented word can be used to generate an illustration. The acquisition module is further configured to acquire similar images corresponding to each of the multiple clauses in response to the fact that the generative possibility index is greater than the index threshold, the description type indicates that the pending presentation word does not contain a verb, and the pending presentation word contains multiple clauses, and the image-text similarity between the clause and the corresponding similar image is greater than the image-text similarity threshold, The decision module is configured to determine the degree of image difference between similar images corresponding to each of the plurality of sections, The decision module is further configured to set similar images corresponding to each of the plurality of sections as illustrations for the corresponding sections in response to the image difference being less than an image difference threshold, in an image processing apparatus.

18. It is an electronic device, Memory for storing computer-executable instructions or computer programs, An electronic device comprising: a processor that executes the image processing method according to any one of claims 1 to 16 when executing the computer-executable instructions or computer programs stored in the memory.

19. A computer-readable storage medium storing computer-executable instructions or computer programs that cause the processor to perform the image processing method described in any one of claims 1 to 16 when executed by the processor.

20. A computer program product comprising a computer-executable instruction or computer program that, when executed by a processor, causes the processor to perform the image processing method described in any one of claims 1 to 16.

Images