Image generation method and device, equipment and storage medium

Abstract

The invention discloses an image generation method and device, equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: acquiring multi-mode input information, wherein the multi-mode input information comprises input information of at least two modes; obtaining a description text of the multi-modal input information, wherein the description text is a text for describing the information content of the multi-modal input information; obtaining at least one cue word corresponding to the description text, wherein the cue word is used for indicating the image content of the image to be generated; obtaining an image corresponding to at least one cue word from an image text library, wherein the image corresponding to the cue word is used for graphically displaying text content represented by the cue word; and generating an output image corresponding to the multi-modal input information according to the at least one cue word and the image corresponding to the at least one cue word through a text graph model. According to the method and the device, the multi-modal input information is adopted, and the image corresponding to the cue word is adopted to control the generation of the output image, so that the image generation quality is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image generation method, device, equipment and storage medium. Background Art

[0002] The text-generated image model is a generative model based on deep learning that can generate original and realistic images based on text descriptions.

[0003] In the related technology, the user inputs a prompt word into the text-based image model, and then the text-based image model performs a Chinese-English translation of the input prompt word, generates an image corresponding to the prompt word, and finally performs yellow reverse detection on the generated image to detect whether the generated image contains illegal content, and intercepts images containing illegal content.

[0004] However, the yellow anti-interception rate of the image generated by the above method is high, and the image generation quality is poor. Summary of the invention

[0005] The embodiments of the present application provide an image generation method, apparatus, device and storage medium. The technical solutions provided by the embodiments of the present application are as follows:

[0006] According to one aspect of an embodiment of the present application, a method for generating an image is provided, the method comprising:

[0007] Acquiring multimodal input information, the multimodal input information including input information of at least two modalities, the multimodal input information being used to represent image generation requirements with input information of different modalities;

[0008] Acquire a description text of the multimodal input information, where the description text is a text describing the information content of the multimodal input information;

[0009] Acquire at least one prompt word corresponding to the description text, where the prompt word is used to indicate the image content of the image to be generated;

[0010] Acquire images corresponding to the at least one prompt word from an image text library, wherein the images corresponding to the prompt word are used to graphically display the text content represented by the prompt word, wherein the image text library includes at least one set of sample images having a matching relationship and image texts corresponding to the sample images, wherein the image texts are description texts for characters or description texts for scenes;

[0011] An output image corresponding to the multimodal input information is generated according to the at least one prompt word and the image corresponding to the at least one prompt word through a text graph model.

[0012] According to one aspect of an embodiment of the present application, there is provided an image generating device, the device comprising:

[0013] An input acquisition module, used to acquire multimodal input information, wherein the multimodal input information includes input information of at least two modes, and the multimodal input information is used to represent image generation requirements with input information of different modes;

[0014] A text acquisition module, used to acquire a description text of the multimodal input information, wherein the description text is a text describing the information content of the multimodal input information;

[0015] A prompt word acquisition module, used to acquire at least one prompt word corresponding to the description text, wherein the prompt word is used to indicate the image content of the image to be generated;

[0016] An image acquisition module, used to acquire images corresponding to the at least one prompt word from an image text library, wherein the images corresponding to the prompt word are used to graphically display the text content represented by the prompt word, wherein the image text library includes at least one set of sample images with matching relationships and image texts corresponding to the sample images, wherein the image texts are description texts for characters or description texts for scenes;

[0017] The output generation module is used to generate an output image corresponding to the multimodal input information according to the at least one prompt word and the image corresponding to the at least one prompt word through a text graph model.

[0018] According to one aspect of an embodiment of the present application, a computer device is provided, the computer device comprising a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-mentioned image generation method.

[0019] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the above-mentioned image generation method.

[0020] According to one aspect of an embodiment of the present application, a computer program product is provided. The computer program product includes a computer program. The computer program is loaded and executed by a processor to implement the above-mentioned image generation method.

[0021] The technical solution provided in the embodiments of the present application can bring the following beneficial effects:

[0022] By adopting multi-modal input information, it is possible to express diversified image generation requirements through at least two modal input information, enrich the input information of the Wensheng graph model, effectively enhance the Wensheng graph model's control over image generation, and make the generated image meet user needs. In addition, at least one image corresponding to a prompt word is added to the Wensheng graph process, and the characters and scenes in the output image are controlled by the image corresponding to the prompt word, so that the image content in the output image can accurately express the textual meaning of at least one prompt word, improve the matching degree between the output image and the multi-modal input information, and improve the quality and accuracy of image generation, so that the generated image can better meet user needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of an implementation environment of a solution provided by an embodiment of the present application;

[0024] Figure 2 is a flow chart of an image generation method provided by an embodiment of the present application;

[0025] Figure 3 is a schematic diagram of a complete process of image generation provided by an embodiment of the present application;

[0026] Figure 4 is a block diagram of an image generating device provided by an embodiment of the present application;

[0027] Figure 5 It is a structural block diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0029] Please refer to Figure 1 , which shows a schematic diagram of a solution implementation environment provided by an embodiment of the present application. The solution implementation environment can be implemented as an image generation system. The solution implementation environment may include: a terminal device 10 and a server 20.

[0030] The number of terminal devices 10 can be one or more. The terminal device 10 can be an electronic device such as a mobile phone, a tablet computer, a laptop computer, a desktop computer, a game console, an e-book reader, a multimedia player, a wearable device, an intelligent voice interaction device, a smart home appliance, a vehicle terminal, an aircraft, etc.

[0031] The terminal device 10 may be installed with a client of a target application, and the target application has the function of generating an image based on input information. The user may input at least one modal input information in the target application, and the client of the target application generates a corresponding image based on the information input by the user, and displays it in the display interface of the target application. This application does not limit the type of target application, including but not limited to question-and-answer applications, music applications, video applications, game applications, web applications, etc. Optionally, the target application may be an application that needs to be downloaded and installed, or it may be a click-to-use application, and this application does not limit this.

[0032] The server 20 is used to provide background services for the client of the target application installed and running in the terminal device 10. For example, the server 20 can be the background server of the above-mentioned target application. The server 20 can be an independent physical server, or a server cluster composed of multiple servers, or a cloud computing service center. Optionally, the server 20 provides background services for the target applications in multiple terminal devices 10 at the same time. The terminal device 10 and the server 20 can communicate with each other through the network.

[0033] In an embodiment of the present application, a user inputs input information of at least two modes in a target application, and a terminal device sends the multimodal input information input by the user to a server, and the server generates a corresponding output image according to the multimodal input information. After receiving the multimodal input information, the server first obtains the description text of the multimodal input information, then obtains at least one prompt word corresponding to the description text, obtains an image corresponding to at least one prompt word from the image text library, and finally inputs at least one prompt word and at least one image corresponding to the prompt word into the text image model to generate an output image corresponding to the multimodal output information. When the output image passes the yellow anti-detection, the server returns the output image to the terminal device and displays the output image in the display interface of the target application.

[0034] Please refer to Figure 2 , which shows a flow chart of an image generation method provided by an embodiment of the present application. The execution subject of each step of the method may be a server. The method may include at least one of the following steps 210 to 250:

[0035] Step 210: Acquire multimodal input information, where the multimodal input information includes input information of at least two modalities, and the multimodal input information is used to represent image generation requirements with input information of different modalities.

[0036] The modalities of the input information include video modality, image modality, audio modality and text modality. The multimodal input information includes at least two of the following: input information in video modality, input information in image modality, input information in audio modality and input information in text modality.

[0037] The input information of the video modality can be understood as an input video, the input information of the image modality can be understood as an input image, the input information of the audio modality can be understood as an input audio, and the input information of the text modality can be understood as an input text. The multimodal input information includes at least two of the input video, the input image, the input audio, and the input text.

[0038] The input information is used to control the effect of image generation. Input information of different modalities can represent different degrees of image generation requirements, thereby achieving different degrees of image generation effects. Multimodal input information includes input information of at least two modalities, so that the information content contained in the multimodal input information is richer, thereby enhancing the control over the image generation effect.

[0039] Step 220, obtaining a description text of the multimodal input information, where the description text is a text that describes the information content of the multimodal input information.

[0040] Generate a description text of the multimodal input information according to the multimodal input information through the multimodal understanding model. Input information of at least two modes is respectively input into the multimodal understanding model, and the description texts corresponding to the input information of at least two modes are respectively output.

[0041] The multimodal understanding model here can be any publicly available large language model, such as a natural language model based on a transformer structure trained with a large amount of data. The large amount of data can reach a sample level of more than 100 million, and this application does not limit this.

[0042] The multimodal understanding model is used to extract the information content of the input information in the form of text, and the modalities of the input information include but are not limited to video modality, image modality and audio modality. For multimodal input information, input information of at least two modalities in the multimodal input information is respectively input into the multimodal understanding model to obtain description texts corresponding to the input information of at least two modalities.

[0043] In some embodiments, when the multimodal input information includes input information in a video modality, a video description text of the input video is obtained based on the input video through the multimodal understanding model. Then, when the multimodal input information includes input information in a video modality, the description text includes a video description text.

[0044] The video description text is a text used to describe the information content of the input information of the video modality, that is, the video description text is a text used to describe the video content of the input video.

[0045] In some embodiments, when the multimodal input information includes input information of image modality, an image description text of the input image is obtained according to the input image through the multimodal understanding model. Then, when the multimodal input information includes input information of image modality, the description text includes image description text.

[0046] The image description text is a text used to describe the information content of the input information of the image modality, that is, the image description text is a text used to describe the image content of the input image.

[0047] In some embodiments, when the multimodal input information includes input information in an audio mode, an audio description text of the input audio is obtained according to the input audio through the multimodal understanding model. Then, when the multimodal input information includes input information in an audio mode, the description text includes an audio description text.

[0048] The audio description text is a text used to describe the information content of the input information of the audio modality, that is, the audio description text is a text used to describe the audio content of the input audio.

[0049] The description text includes description texts corresponding to at least two modal input information. For example, if the multimodal input information includes input information of a video modality, input information of an image modality, input information of an audio modality, and input information of a text modality, the description text includes video description text, image description text, audio description text, and input information of a text modality.

[0050] By adopting multimodal input information and obtaining description text of the multimodal input information, the text meaning of the generated description text is enriched, thereby enhancing the control over the effect of image generation and making the generated image more in line with user needs.

[0051] Step 230: Obtain at least one prompt word corresponding to the description text, where the prompt word is used to indicate the image content of the image to be generated.

[0052] At least one prompt word is a prompt word obtained based on the description text, but at least one prompt word is not a word directly obtained from the description text. Exemplarily, the prompt word can be a formatted word, including words and word types obtained from the description text.

[0053] The prompt word is used to indicate the image content of the image to be generated, that is, the prompt word is used to indicate the image content of the output image. Specifically, the prompt word is used to indicate the person or scene in the image to be generated.

[0054] Step 240, obtaining at least one image corresponding to a prompt word from the image text library, the image corresponding to the prompt word is used to graphically display the text content represented by the prompt word, the image text library includes at least one group of sample images with a matching relationship and image texts corresponding to the sample images, and the image texts are descriptive texts for characters or descriptive texts for scenes.

[0055] The image-text library contains at least one set of image-text pairs, each set of image-text pairs includes a sample image and image text with a matching relationship, the sample image is used to graphically display the text content represented by the image text, and the image text is used to textually describe the image content in the sample image.

[0056] Optionally, the image text is a descriptive text for a person, and the image text includes but is not limited to a descriptive text for a person's posture, a descriptive text for a person's facial expression, a descriptive text for a person's facial features, a descriptive text for a person's body features, a descriptive text for a person's clothing, and the like. Exemplarily, the image text is a text describing a person's posture, and the image text includes but is not limited to a text describing a person's standing posture, a text describing a person's running posture, a text describing a person's sitting posture with legs crossed, and the like. Exemplarily, the image text is a text describing a person's facial expression, and the image text includes but is not limited to a text describing a person's smiling expression, a text describing a person's sad expression, a text describing a person's sad expression, and the like. Exemplarily, the image text is a text describing a person's facial features, and the image text includes but is not limited to a text describing a person's eye features, a text describing a person's lip features, a text describing a person's face shape features, and the like. Exemplarily, the image text is a descriptive text describing a person's body features, and the image text includes but is not limited to a text describing a person's body proportions, a text describing a person's fatness or thinness, a text describing a person's height, and the like.

[0057] Optionally, the image text is a description text for the scene, and the image text includes but is not limited to a description text for the scene layout, a description text for the scene elements, a description text for the scene location, etc. Exemplarily, the image text is a text describing the scene layout, and the image text may be a text describing the positional relationship of each element in the scene. Exemplarily, the image text is a text describing the scene elements, and the image text may be a text describing each element in the scene. Exemplarily, the image text is a text describing the scene location, and the image text may be a text describing the environment in which the scene is located, or a text describing the weather in which the scene is located.

[0058] Exemplarily, the image text corresponding to at least one prompt word can be obtained from the image text library, and sample images matching at least one image text can be obtained according to the image text corresponding to at least one prompt word, so as to obtain the image corresponding to at least one prompt word. The prompt word and the image corresponding to the prompt word, and the image text matching the prompt word and the image corresponding to the prompt word correspond to each other.

[0059] The image corresponding to the prompt word is used to graphically display the text content represented by the prompt word, and the image corresponding to at least one prompt word is used to graphically display the text content represented by the description text, thereby controlling the image generation content and image generation effect through the image corresponding to at least one prompt word.

[0060] Step 250: Generate an output image corresponding to the multimodal input information according to at least one prompt word and at least one image corresponding to the prompt word through a text-based graph model.

[0061] The text graph model here can be any publicly available diffusion model, such as a diffusion structure-based model trained with a large amount of data. The large amount of data can reach a sample level of more than 100 million, and this application does not limit this.

[0062] At least one prompt word and at least one image corresponding to the prompt word are input into the text graph model, and an output image corresponding to the multimodal input information is output.

[0063] Optionally, the image resolution of the output image may be a default image resolution preset by the text graph model, for example, the default image resolution is 720×1280. Optionally, the image resolution of the output image may also be the maximum value of the image resolution in the image corresponding to at least one prompt word. Optionally, the image resolution of the output image may also be the minimum value of the image resolution in the image corresponding to at least one prompt word. Optionally, the image resolution of the output image may also be the middle value of the image resolution in the image corresponding to at least one prompt word.

[0064] In some embodiments, an output image corresponding to the multimodal input information is generated by a text graph model according to at least one prompt word, an image corresponding to at least one prompt word, and a quality vocabulary, and the quality vocabulary is used to indicate the image quality of the output image.

[0065] The quality vocabulary is a vocabulary describing the image quality of the output image, and optionally, the quality vocabulary may be a vocabulary describing the image resolution of the output image, for example, the quality vocabulary is "720×1280". Alternatively, the quality vocabulary may also be a vocabulary describing the image definition of the output image, for example, the quality vocabulary is "HD".

[0066] At least one prompt word, an image corresponding to at least one prompt word, and a quality word are input into a text graph model, and an output image corresponding to the multimodal input information is output. The image quality of the output image is the same as the image quality indicated by the quality word.

[0067] By adding quality vocabulary when generating output images, the image quality of the output images can be adjusted through the quality vocabulary, thereby generating images that better meet user requirements, improving the diversity of image generation, and improving the image quality of the generated images.

[0068] The technical solution provided by the embodiment of the present application adopts multi-modal input information, so that the diversified image generation requirements can be represented by at least two modal input information, which enriches the input information of the Wensheng graph model, effectively enhances the Wensheng graph model's control over the effect of image generation, and enables the generated image to meet the user's needs. In addition, at least one image corresponding to a prompt word is added to the Wensheng graph process, and the characters and scenes in the output image are controlled by the image corresponding to the prompt word, so that the image content in the output image can accurately represent the textual meaning of at least one prompt word, improves the matching degree between the output image and the multi-modal input information, and improves the quality and accuracy of image generation, so that the generated image can better meet the user's needs.

[0069] In some embodiments, step 230 includes at least one of sub-steps 231 - 233 .

[0070] Sub-step 231, obtaining at least one entity word in the description text.

[0071] The description text is split into words to obtain at least one word in the description text. At least one entity word is obtained from the at least one word according to the word meaning of the at least one word.

[0072] Entity words refer to words that have a clear referent or have actual meaning. For example, entity words can be nouns, adjectives, pronouns, etc.

[0073] Sub-step 232, based on at least one illegal word in the illegal word library, remove words matching the illegal word from at least one entity word to obtain removed words, where the illegal word is a word used to represent illegal content.

[0074] The illegal vocabulary library contains at least one illegal word, and the illegal word is a word used to represent illegal content. Illegal content includes but is not limited to pornographic content, vulgar content, political content, bloody content, violent content, terrorist content, content with bad values, etc. There can be at least one illegal word in the illegal vocabulary library to represent the same illegal content, and the number of illegal words used to represent different illegal contents can be the same or different. For example, there can be one illegal word in the illegal vocabulary library to represent pornographic content, and there can also be multiple illegal words to represent pornographic content. If there is one illegal word in the illegal vocabulary library to represent pornographic content, the number of illegal words in the illegal vocabulary library to represent other illegal contents other than pornographic content can be one or more.

[0075] In some embodiments, for each entity word, the similarity between the entity word and at least one illegal vocabulary in the illegal vocabulary library is calculated respectively. When the maximum similarity in the similarity between the entity word and at least one illegal vocabulary is greater than or equal to the third threshold value, it can be determined that the illegal vocabulary corresponding to the entity word and the maximum similarity match, and then the entity word is removed from at least one entity word. When the maximum similarity in the similarity between the entity word and at least one illegal vocabulary is less than the third threshold value, it can be determined that the entity word and at least one illegal vocabulary in the illegal vocabulary library do not match, and then it is not necessary to remove the entity word from at least one entity word. With reference to the above steps, it is judged whether there are illegal vocabulary that matches each entity word in at least one entity word in the illegal vocabulary library, and when there are illegal vocabulary that matches the entity word in the illegal vocabulary library, the entity word is removed from at least one entity word, and when there are no illegal vocabulary that matches the entity word in the illegal vocabulary library, the entity word is retained in at least one entity word. After processing at least one entity word, the vocabulary after removal is obtained.

[0076] Optionally, the similarity between the word vector of the entity word and the word vector of at least one illegal word can be calculated by cosine similarity or Euclidean distance. Optionally, the similarity between the entity word and the illegal word can be obtained based on the entity word and the illegal word by a text matching model.

[0077] The specific value of the third threshold is set by the technicians according to the vocabulary elimination requirements, and this application does not limit it.

[0078] Each word in the eliminated vocabulary is a word that does not match at least one illegal word in the illegal vocabulary library, but it should be noted that since the maximum value of the similarity between each word in the eliminated vocabulary and at least one illegal word is less than the third threshold, when the third threshold is not 0, there may be words in the eliminated vocabulary that represent a part of illegal content. That is, each word in the eliminated vocabulary does not completely contain illegal content.

[0079] Sub-step 233, extracting at least one target word from the eliminated words through a formatted extraction model to obtain at least one prompt word, the target word is a word describing a person or a word describing a scene, and the prompt word includes the target word and the word type of the target word.

[0080] The formatted extraction model is used to extract at least one target word from the eliminated words, and generate the target word into a formatted target word, that is, generate a prompt word.

[0081] The target words are the words used to describe characters in the removed words, or the words used to describe scenes in the removed words. Words used to describe characters include, but are not limited to, words used to describe character postures, words used to describe character facial expressions, words used to describe character facial features, words used to describe character morphological features, words used to describe character clothing, etc. Words used to describe scenes include, but are not limited to, words used to describe scene layouts, words used to describe scene elements, words used to describe scene locations, etc.

[0082] The formatted target vocabulary (prompt word) is a vocabulary of a specific format generated on the basis of the target vocabulary. In the embodiment of the present application, the prompt word includes the target vocabulary and the type of the target vocabulary. The type of the target vocabulary is a person or a scene. Therefore, when the target vocabulary is a word describing a person, the prompt word includes the target vocabulary and the vocabulary type "person" of the target vocabulary. When the target vocabulary is a word describing a scene, the prompt word includes the target vocabulary and the vocabulary type "scene" of the target vocabulary.

[0083] Exemplarily, the prompt word may be expressed in the format of {"character": character 1}, {"character": character 2}, {"scene": seaside}, {"scene": beach}.

[0084] By removing words that match the illegal words from at least one entity word, the words after the removal do not contain words that match the illegal words, thereby reducing the probability that the generated output image contains illegal content, improving the quality of image generation, and reducing the yellow anti-interception rate.

[0085] In some embodiments, step 240 includes at least one of sub-steps 241 - 243 .

[0086] Sub-step 241 , acquiring image texts corresponding to at least one target word from an image text library, to obtain at least one image text.

[0087] In some embodiments, for each target word, the similarity between the target word and each image text in the image text library is calculated respectively. The image text corresponding to the maximum similarity among the similarities between the target word and each image text in the image text library is determined as the image text corresponding to the target word. Referring to the above steps, the image text corresponding to at least one target word is obtained, that is, at least one image text is obtained.

[0088] Optionally, the similarity between the word vector of the target vocabulary and the text vector of each image text in the image text library can be calculated by cosine similarity or Euclidean distance. Optionally, the similarity between the target vocabulary and the image text can also be obtained based on the target vocabulary and the image text by a text matching model.

[0089] Sub-step 242, obtaining a sample image corresponding to at least one image text from the image text library.

[0090] According to the matching relationship between the sample image and the image text in the image text library, at least one sample image corresponding to the image text is obtained from the image text library.

[0091] Sub-step 243, determining the sample images corresponding to the at least one image text as the images corresponding to the at least one prompt word.

[0092] By obtaining images corresponding to at least one prompt word from the image text library, the embodiment of each prompt word in the output image can be controlled individually, thereby achieving precise control of the characters and scenes in the output image, improving the quality of image generation, and effectively reducing the deformity rate of the output image.

[0093] In some embodiments, step 250 includes at least one of sub-steps 251 - 253 .

[0094] Sub-step 251, when the multimodal input information includes target input information, a segmented image of the target input information is obtained according to the target input information through a person segmentation model, where the target input information is input information of a video modality or input information of an image modality.

[0095] The person segmentation model is used to segment the image containing the person, and obtain the person region in the image to obtain a segmented image. The segmented image is an image containing only the person region. The person segmentation model here can be any publicly available person segmentation model, and this application does not limit this.

[0096] In some embodiments, when the target input information is input information of a video modality, if the multimodal input information includes the target input information, a segmented image of the input information of the video modality is obtained according to the input information of the video modality through a character segmentation model.

[0097] The input information of the video modality (input video) includes at least two video frames. The at least two video frames are segmented by the character segmentation model, and the character regions in the at least two video frames are obtained, so as to obtain the segmented images corresponding to the at least two video frames. The segmented images of the target input information (the segmented images of the input information of the video modality) mentioned above include the segmented images corresponding to the at least two video frames.

[0098] In some embodiments, when the target input information is input information of an image modality, if the multimodal input information includes the target input information, a segmented image of the input information of the image modality is obtained according to the input information of the image modality through a person segmentation model.

[0099] The input information of the image modality (input image) is an image, and the segmented image of the target input information (segmented image of the input information of the image modality) is the segmented image corresponding to the image.

[0100] Sub-step 252, obtaining the character features corresponding to the segmented image based on the segmented image through the visual language model.

[0101] The visual language model is used to extract relevant information from the input image to achieve visual question answering. In the embodiment of the present application, the visual language model is used to extract image features in the segmented image. Since the segmented image is an image that only contains the person area, the visual language model can extract the person features corresponding to the segmented image.

[0102] The visual language model here can be any publicly available visual language model, which is not limited in this application. Exemplarily, the visual language model can be a CLIP model.

[0103] When the input information of the text graph model includes quality words, the following sub-step 253 is executed. When the input information of the text graph model does not include quality words, the text graph model is used to generate an output image corresponding to the multimodal input information according to at least one prompt word, an image corresponding to at least one prompt word, and a character feature respectively corresponding to the at least one prompt word.

[0104] Sub-step 253, generating an output image corresponding to the multimodal input information according to the at least one prompt word, the image corresponding to the at least one prompt word, the quality vocabulary and the character feature through the text graph model.

[0105] At least one prompt word, an image corresponding to at least one prompt word, a quality vocabulary, and a character feature are input into a raw image model, and an output image corresponding to the multimodal input information is output.

[0106] By adding character features in multimodal input information to the process of text generation, the control over character generation during image generation can be enhanced, making the character image in the generated output image more accurate and prominent, thereby improving the quality of image generation.

[0107] In some embodiments, step 250 is followed by step 260 , and step 260 includes at least one of sub-steps 261 - 262 .

[0108] Sub-step 261 : when the output image passes the yellowing detection, the output image is determined as the final output image.

[0109] Sub-step 262 , when the output image fails the yellowing detection, re-execute step 250 .

[0110] The yellow back detection is used to detect whether the image contains illegal content. If the output image passes the yellow back detection, it can be considered that the output image does not contain illegal content, and the output image can be determined as the final output image. If the output image does not pass the yellow back detection, it can be considered that the output image contains illegal content, and the output image does not meet the requirements of the final output, and the output image needs to be regenerated, that is, step 250 is re-executed.

[0111] It should be noted that the fact that the output image passes the yellowing detection does not mean that the output image does not contain any illegal content at all, but rather that the illegal content contained in the output image is within the allowable range of the yellowing detection.

[0112] By performing yellowing detection on the output image and regenerating the output image that fails the yellowing detection, the security of the output image obtained by the user can be guaranteed while ensuring the image quality, and avoiding the output image containing illegal content and causing adverse effects on the user.

[0113] There are two ways to perform yellowing detection on the output image: one is to perform yellowing detection through a yellowing detection model, and the other is to perform yellowing detection through a defect detection model.

[0114] In some embodiments, a yellow back detection result of the output image is obtained according to the output image through a yellow back detection model, and the yellow back detection model is a pre-trained binary classification model; when the yellow back detection result is a first value, it is determined that the output image passes the yellow back detection; when the yellow back detection result is a second value, it is determined that the output image does not pass the yellow back detection.

[0115] The yellowing detection model is a pre-trained binary classification model. The output image is output to the yellowing detection model, and the yellowing detection result of the output image is output. The yellowing detection result is a first value or a second value. The first value is used to indicate that the output image passes the yellowing detection, and the second value is used to indicate that the output image fails the yellowing detection.

[0116] The present application does not limit the specific values ​​of the first value and the second value. For example, if the first value is 0 and the second value is 1, then when the yellow backlash detection result is 0, it is determined that the output image passes the yellow backlash detection, and when the yellow backlash detection result is 1, it is determined that the output image does not pass the yellow backlash detection.

[0117] By performing yellowing detection through a binary classification model, it is easier to determine whether the output image passes the yellowing detection through the numerical output of the model, thereby improving the convenience of yellowing detection.

[0118] In the case where the output image fails the yellowing detection, the defect rate of the output image can be further obtained according to the output image through the defect detection model, and then step 250 is re-executed. In the case where the number of times the output image is regenerated reaches the second threshold, and each regenerated image fails the yellowing detection, the output image corresponding to the minimum defect rate among the defect rates of the regenerated images is determined as the final output image.

[0119] In some embodiments, a defect rate of the output image is obtained based on the output image through a defect detection model, the defect detection model is a pre-trained machine learning model, and the defect rate is used to indicate the content of illegal content in the output image; when the defect rate of the output image is less than a first threshold, it is determined that the output image passes the yellow back detection; when the defect rate of the output image is greater than or equal to the first threshold, it is determined that the output image fails the yellow back detection.

[0120] The defect detection model is a pre-trained machine learning model. The output image is input into the defect detection model, and the defect rate of the output image is output. The defect rate ranges from 0 to 1. The defect rate is used to indicate the content of illegal content in the output image. The lower the defect rate, the lower the content of illegal content in the output image. The higher the defect rate, the higher the content of illegal content in the output image. If the defect rate is 0, it means that the output image does not contain illegal content.

[0121] The specific value of the first threshold is set by the technician according to the yellowing detection requirement, and this application does not limit it. For example, the first threshold may be 0.3, and when the defect rate of the output image is less than 0.3, it is determined that the output image passes the yellowing detection, and when the defect rate of the output image is greater than or equal to 0.3, it is determined that the output image fails the yellowing detection.

[0122] By performing yellowing detection through the defect detection model, the content of illegal content contained in the output image can be more intuitively seen according to the defect rate, so as to determine whether the image passes the yellowing detection, thereby improving the accuracy of the yellowing detection.

[0123] In some embodiments, when the number of times the output image is regenerated reaches a second threshold and each regenerated image fails the yellowing detection, the output image corresponding to the minimum value of the defect rate of the regenerated images is determined as the final output image.

[0124] The specific value of the second threshold is set by the technician according to the yellowing detection requirement, and is not limited in this application. For example, if the second threshold is 5, then when the number of times the output image is regenerated reaches 5 times, and none of the 5 regenerated images pass the yellowing detection, the output image corresponding to the minimum defect rate among the 5 regenerated images is determined as the final output image.

[0125] By setting an upper limit on the number of times the regenerated image is generated, it is avoided that the regenerated image always fails to pass the yellowing detection and step 250 is repeatedly executed, so that while ensuring the quality of image generation, computing resources are saved and the efficiency of image generation is guaranteed.

[0126] Figure 3The schematic diagram of the complete process of image generation is shown. The multimodal input information includes input video, input image, input audio and input text. The input video is decoded to obtain the decoded input video. On the one hand, the multimodal understanding model is used to obtain the video description text of the input video based on the decoded input video. On the other hand, the character segmentation model is used to obtain the segmented image of the input video based on the decoded input video. Then, the visual language model is used to obtain the character features corresponding to the input video. The input image is image-inversely inferred. On the one hand, the multimodal understanding model is used to obtain the image description text of the input image based on the input image. On the other hand, the character segmentation model is used to obtain the segmented image of the input image based on the input image. Then, the visual language model is used to obtain the character features corresponding to the input image. The input audio is audio-inversely inferred. The multimodal understanding model is used to obtain the audio description text of the input audio based on the input audio. The video description text, the image description text, the audio description text and the input text are subjected to text yellowing judgment. If the text fails the yellowing judgment, at least one entity word in the text is obtained, and the words matching the illegal words in the at least one entity word are eliminated to obtain the eliminated words. At least one target word in the eliminated words is extracted by the formatting extraction model to obtain at least one prompt word, and the prompt word includes the target word and the word type of the target word. The image corresponding to at least one prompt word is obtained from the image text library. The image text library contains sample images with matching relationships and image texts corresponding to the sample images. The image texts include but are not limited to texts for describing character postures, texts for describing character faces, texts for describing scene layouts, and texts for describing scene elements. The output image is generated by the text-born graph model according to at least one prompt word, the image corresponding to at least one prompt word, the quality words and the character features. The output image is subjected to image yellowing judgment. If the output image passes the yellowing judgment, the output image is determined as the final output image, and the image generation process is terminated. If the output image fails the yellowing judgment, the output image is regenerated cyclically until the number of regeneration times reaches an upper limit, and the output image corresponding to the maximum defect rate among the regenerated images is determined as the final output image.

[0127] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0128] Please refer to Figure 4 , which shows a block diagram of an image generation device provided by an embodiment of the present application. The device has the function of implementing the above-mentioned image generation method, and the function can be implemented by hardware, or by hardware executing corresponding software. The device can be the server introduced above, or it can be set in the server. Figure 4As shown, the device 400 may include: an input acquisition module 410 , a text acquisition module 420 , a prompt word acquisition module 430 , an image acquisition module 440 and an output generation module 450 .

[0129] The input acquisition module 410 is used to acquire multimodal input information, where the multimodal input information includes input information of at least two modes, and the multimodal input information is used to represent image generation requirements with input information of different modes.

[0130] The text acquisition module 420 is used to acquire the description text of the multimodal input information, where the description text is a text describing the information content of the multimodal input information.

[0131] The prompt word acquisition module 430 is used to acquire at least one prompt word corresponding to the description text, where the prompt word is used to indicate the image content of the image to be generated.

[0132] The image acquisition module 440 is used to acquire images corresponding to at least one prompt word from the image text library, wherein the images corresponding to the prompt words are used to graphically display the text content represented by the prompt words, and the image text library includes at least one group of sample images with matching relationships and image texts corresponding to the sample images, wherein the image texts are descriptive texts for characters or descriptive texts for scenes.

[0133] The output generation module 450 is used to generate an output image corresponding to the multimodal input information according to the at least one prompt word and the image corresponding to the at least one prompt word through a text graph model.

[0134] In some embodiments, the text acquisition module 420 is used to:

[0135] In a case where the multimodal input information includes input information of the video modality, obtaining a video description text of the input video according to the input video through a multimodal understanding model, wherein the description text includes the video description text;

[0136] In a case where the multimodal input information includes input information of the image modality, obtaining an image description text of the input image according to the input image through the multimodal understanding model, the description text including the image description text;

[0137] In the case where the multimodal input information includes input information of the audio modality, an audio description text of the input audio is obtained according to the input audio through the multimodal understanding model, and the description text includes the audio description text.

[0138] In some embodiments, the prompt word acquisition module 430 is used to:

[0139] Obtain at least one entity word in the description text;

[0140] According to at least one illegal word in the illegal word library, remove words matching the illegal word from the at least one entity word to obtain removed words, wherein the illegal word is a word used to represent illegal content;

[0141] At least one target word from the eliminated words is extracted through a formatted extraction model to obtain at least one prompt word, wherein the target word is a word describing a person or a word describing a scene, and the prompt word includes the target word and the word type of the target word.

[0142] In some embodiments, the image acquisition module 440 is used to:

[0143] Acquire the image texts corresponding to the at least one target word from the image text library to obtain at least one image text;

[0144] Acquire sample images corresponding to the at least one image text respectively from the image text library;

[0145] The sample images corresponding to the at least one image text are determined as the images corresponding to the at least one prompt word.

[0146] In some embodiments, the output generation module 450 is used to:

[0147] An output image corresponding to the multimodal input information is generated by the text graph model according to the at least one prompt word, the image corresponding to the at least one prompt word, and a quality vocabulary, wherein the quality vocabulary is used to indicate an image quality of the output image.

[0148] In some embodiments, the output generation module 450 is used to:

[0149] In a case where the multimodal input information includes target input information, a segmented image of the target input information is obtained according to the target input information by using a person segmentation model, wherein the target input information is input information of a video modality or input information of an image modality;

[0150] Obtaining character features corresponding to the segmented image according to the segmented image through a visual language model;

[0151] The text graph model generates an output image corresponding to the multimodal input information according to the at least one prompt word, the image corresponding to the at least one prompt word, the quality vocabulary and the character feature.

[0152] In some embodiments, the apparatus 400 further includes an image detection module, wherein the image detection module is configured to:

[0153] In a case where the output image passes the yellowing detection, determining the output image as a final output image;

[0154] In the case where the output image fails the yellow back detection, the step of generating the output image corresponding to the multimodal input information by using the text graph model according to the at least one prompt word and the image respectively corresponding to the at least one prompt word is re-executed.

[0155] In some embodiments, the image detection module is used to:

[0156] Obtaining a yellowing detection result of the output image according to the output image through a yellowing detection model, wherein the yellowing detection model is a pre-trained binary classification model;

[0157] When the yellowing detection result is a first value, determining that the output image passes the yellowing detection;

[0158] When the yellowing detection result is a second value, it is determined that the output image fails the yellowing detection.

[0159] In some embodiments, the image detection module is used to:

[0160] Obtaining a defect rate of the output image according to the output image through a defect detection model, wherein the defect detection model is a pre-trained machine learning model, and the defect rate is used to indicate the content of illegal content in the output image;

[0161] When the defect rate of the output image is less than the first threshold, determining that the output image passes the yellowing detection;

[0162] When the defect rate of the output image is greater than or equal to the first threshold, it is determined that the output image fails the yellowing detection.

[0163] In some embodiments, the image detection module is used to:

[0164] When the number of times the output image is regenerated reaches a second threshold and each regenerated image fails the yellowing detection, the output image corresponding to the minimum value of the defect rate of the regenerated images is determined as the final output image.

[0165] It should be noted that the device provided in the above embodiment, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the content structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0166] Please refer to Figure 5 , which shows a block diagram of a computer device 500 provided in one embodiment of the present application. The computer device 500 may be any electronic device with data calculation, processing and storage functions. The computer device 500 may be used to implement the image generation method provided in the above embodiment.

[0167] Typically, the computer device 500 includes a processor 501 and a memory 502 .

[0168] The processor 501 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 501 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). The processor 501 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 501 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 501 may also include an AI processor, which is used to process computing operations related to machine learning.

[0169] The memory 502 may include one or more computer-readable storage media, which may be non-transitory. The memory 502 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 502 is used to store a computer program, which is configured to be executed by one or more processors to implement the above-mentioned image generation method.

[0170] Those skilled in the art will understand that Figure 5 The structure shown in the figure does not constitute a limitation on the computer device 500, and the computer device 500 may include more or less components than those shown in the figure, or combine some components, or adopt a different arrangement of components.

[0171] In an illustrative embodiment, a computer-readable storage medium is also provided, wherein a computer program is stored in the storage medium, and when the computer program is executed by a processor of a computer device, the image generation method is implemented. Optionally, the computer-readable storage medium may be a ROM (Read-Only Memory), a RAM (Random Access Memory), a CD-ROM (Compact Disc Read-Only Memory), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0172] In an exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned image generation method.

[0173] It should be understood that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application are not limited to this.

[0174] The above description is only an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An image generation method, characterized in that: The method comprises: Acquiring multimodal input information, the multimodal input information including input information of at least two modalities, the multimodal input information being used to represent image generation requirements with input information of different modalities; Acquire a description text of the multimodal input information, where the description text is a text describing the information content of the multimodal input information; Acquire at least one prompt word corresponding to the description text, where the prompt word is used to indicate the image content of the image to be generated; Acquire images corresponding to the at least one prompt word from an image text library, wherein the images corresponding to the prompt word are used to graphically display the text content represented by the prompt word, wherein the image text library includes at least one set of sample images having a matching relationship and image texts corresponding to the sample images, wherein the image texts are description texts for characters or description texts for scenes; An output image corresponding to the multimodal input information is generated according to the at least one prompt word and the image corresponding to the at least one prompt word through a text graph model.

2. The method according to claim 1, characterized in that The obtaining of the description text of the multimodal input information includes: In a case where the multimodal input information includes input information of the video modality, obtaining a video description text of the input video according to the input video through a multimodal understanding model, wherein the description text includes the video description text; In a case where the multimodal input information includes input information of the image modality, obtaining an image description text of the input image according to the input image through the multimodal understanding model, the description text including the image description text; In the case where the multimodal input information includes input information of the audio modality, an audio description text of the input audio is obtained according to the input audio through the multimodal understanding model, and the description text includes the audio description text.

3. The method according to claim 1, characterized in that The obtaining, according to the description text, at least one prompt word corresponding to the description text includes: Obtain at least one entity word in the description text; According to at least one illegal word in the illegal word library, remove words matching the illegal word from the at least one entity word to obtain removed words, wherein the illegal word is a word used to represent illegal content; At least one target word from the eliminated words is extracted through a formatted extraction model to obtain at least one prompt word, wherein the target word is a word describing a person or a word describing a scene, and the prompt word includes the target word and the word type of the target word.

4. The method according to claim 3, characterized in that The acquiring, from the image text library, images corresponding to the at least one prompt word respectively comprises: Acquire the image texts corresponding to the at least one target word from the image text library to obtain at least one image text; Acquire sample images corresponding to the at least one image text respectively from the image text library; The sample images corresponding to the at least one image text are determined as the images corresponding to the at least one prompt word.

5. The method according to claim 1, characterized in that The step of generating an output image corresponding to the multimodal input information according to the at least one prompt word and the image corresponding to the at least one prompt word by using the text graph model includes: An output image corresponding to the multimodal input information is generated by the text graph model according to the at least one prompt word, the image corresponding to the at least one prompt word, and a quality vocabulary, wherein the quality vocabulary is used to indicate an image quality of the output image.

6. The method according to claim 5, characterized in that The step of generating an output image corresponding to the multimodal input information by using the text graph model according to the at least one prompt word, the image corresponding to the at least one prompt word, and the quality vocabulary includes: In a case where the multimodal input information includes target input information, obtaining a segmented image of the target input information according to the target input information by using a person segmentation model, wherein the target input information is input information of a video modality or input information of an image modality; Obtaining character features corresponding to the segmented image according to the segmented image through a visual language model; The text graph model generates an output image corresponding to the multimodal input information according to the at least one prompt word, the image corresponding to the at least one prompt word, the quality vocabulary and the character feature.

7. The method according to claim 1, characterized in that After generating the output image corresponding to the multimodal input information according to the at least one prompt word and the image corresponding to the at least one prompt word by using the text graph model, the method further includes: In a case where the output image passes the yellowing detection, determining the output image as a final output image; In the case where the output image fails the yellow back detection, the step of generating the output image corresponding to the multimodal input information by using the text graph model according to the at least one prompt word and the image respectively corresponding to the at least one prompt word is re-executed.

8. The method according to claim 7, characterized in that The method further comprises: Obtaining a yellowing detection result of the output image according to the output image through a yellowing detection model, wherein the yellowing detection model is a pre-trained binary classification model; When the yellowing detection result is a first value, determining that the output image passes the yellowing detection; When the yellowing detection result is a second value, it is determined that the output image fails the yellowing detection.

9. The method according to claim 7, characterized in that: The method further comprises: Obtaining a defect rate of the output image according to the output image through a defect detection model, wherein the defect detection model is a pre-trained machine learning model, and the defect rate is used to indicate the content of illegal content in the output image; When the defect rate of the output image is less than the first threshold, determining that the output image passes the yellowing detection; When the defect rate of the output image is greater than or equal to the first threshold, it is determined that the output image fails the yellowing detection.

10. The method according to claim 9, characterized in that After the step of re-executing the step of generating the output image corresponding to the multimodal input information according to the at least one prompt word and the image respectively corresponding to the at least one prompt word through the text graph model, the method further includes: When the number of times the output image is regenerated reaches a second threshold and each regenerated image fails the yellowing detection, the output image corresponding to the minimum value of the defect rate of the regenerated images is determined as the final output image.

11. An image generating device, characterized in that: The device comprises: An input acquisition module, used to acquire multimodal input information, wherein the multimodal input information includes input information of at least two modes, and the multimodal input information is used to represent image generation requirements with input information of different modes; A text acquisition module, used to acquire a description text of the multimodal input information, wherein the description text is a text describing the information content of the multimodal input information; A prompt word acquisition module, used to acquire at least one prompt word corresponding to the description text, wherein the prompt word is used to indicate the image content of the image to be generated; An image acquisition module, used to acquire images corresponding to the at least one prompt word from an image text library, wherein the images corresponding to the prompt word are used to graphically display the text content represented by the prompt word, wherein the image text library includes at least one set of sample images with matching relationships and image texts corresponding to the sample images, wherein the image texts are description texts for characters or description texts for scenes; The output generation module is used to generate an output image corresponding to the multimodal input information according to the at least one prompt word and the image corresponding to the at least one prompt word through a text graph model.

12. A computer device, characterized in that: The computer device comprises a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the image generating method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is loaded and executed by a processor to implement the image generating method according to any one of claims 1 to 10.

14. A computer program product, characterized in that The computer program product comprises a computer program, which is loaded and executed by a processor to implement the image generation method according to any one of claims 1 to 10.

Images