Image generation method, apparatus, device, and storage medium

Abstract

The application discloses an image generation method and device, equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: acquiring multi-modal input information, wherein the multi-modal input information comprises input information of at least two modes; acquiring a description text of the multi-modal input information, wherein the description text is a text for describing the information content of the multi-modal input information; acquiring at least one prompt word corresponding to the description text, wherein the prompt word is used for indicating the image content of an image to be generated; acquiring images corresponding to the at least one prompt word from an image text library, wherein the images corresponding to the prompt word are used for graphically displaying the text content represented by the prompt word; and generating an output image corresponding to the multi-modal input information by using a text-to-image model according to the at least one prompt word and the images corresponding to the at least one prompt word. According to the application, the multi-modal input information is used, and the images corresponding to the prompt words are used to control the generation of the output image, so that the quality of image generation is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and in particular to an image generation method and device, equipment and a storage medium. BACKGROUND

[0002] The text-to-image model is a deep learning-based generation model that can generate original and real images according to text descriptions.

[0003] In related technologies, a user inputs a prompt word into a text-to-image model, the text-to-image model translates the input prompt word into English, generates an image corresponding to the prompt word, and finally performs yellow and reverse detection on the generated image to detect whether the generated image contains illegal content and intercepts the image containing illegal content.

[0004] However, the above method has a high yellow and reverse interception rate for generated images and poor image generation quality. SUMMARY

[0005] The embodiments of the present application provide an image generation method, device, equipment and storage medium. The technical solutions provided by the embodiments of the present application are as follows.

[0006] According to an aspect of the embodiments of the present application, an image generation method is provided, which comprises:

[0007] Obtaining multi-modal input information, the multi-modal input information comprising input information of at least two modalities, the multi-modal input information being used to express image generation requirements in different modalities of input information;

[0008] Obtaining a description text of the multi-modal input information, the description text being a text describing the information content of the multi-modal input information;

[0009] Obtaining at least one prompt word corresponding to the description text, the prompt word being used to indicate the image content of the image to be generated;

[0010] Obtaining images corresponding to the at least one prompt word from an image text library, the images corresponding to the prompt word being used to graphically display the text content represented by the prompt word, the image text library comprising at least one set of sample images and image texts corresponding to the sample images having a matching relationship, the image texts being description texts of characters or description texts of scenes;

[0011] Generating, by a text-to-image model, an output image corresponding to the multi-modal input information according to the at least one prompt word and the images corresponding to the at least one prompt word.

[0012] According to an aspect of some embodiments of the present application, an image generation apparatus is provided, the apparatus comprising:

[0013] An input obtaining module is configured to obtain multi-modal input information, the multi-modal input information comprising input information of at least two modalities, the multi-modal input information being used to represent an image generation requirement with input information of different modalities;

[0014] A text obtaining module is configured to obtain a description text of the multi-modal input information, the description text being a text describing information content of the multi-modal input information;

[0015] A prompt word obtaining module is configured to obtain at least one prompt word corresponding to the description text, the prompt word being used to indicate image content of an image to be generated;

[0016] An image obtaining module is configured to obtain images corresponding to the at least one prompt word respectively from an image text library, the images corresponding to the prompt word being used to graphically display text content represented by the prompt word, the image text library comprising at least one set of sample images and image texts corresponding to the sample images having a matching relationship, the image texts being description texts of a person or a scene;

[0017] An output generation module is configured to generate an output image corresponding to the multi-modal input information by a text-to-image model according to the at least one prompt word and the images corresponding to the at least one prompt word respectively.

[0018] According to an aspect of some embodiments of the present application, a computer device is provided, the computer device comprising a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above image generation method.

[0019] According to an aspect of some embodiments of the present application, a computer readable storage medium is provided, the computer readable storage medium storing a computer program, the computer program being loaded and executed by a processor to implement the above image generation method.

[0020] According to an aspect of some embodiments of the present application, a computer program product is provided, the computer program product comprising a computer program, the computer program being loaded and executed by a processor to implement the above image generation method.

[0021] The technical solutions provided by the embodiments of the present application can bring the following beneficial effects:

[0022] By adopting the multi-modal input information, diversified image generation requirements can be represented by input information of at least two modalities, the input information of the text-to-image model is enriched, the effect control of the text-to-image model on image generation is effectively enhanced, and the generated image can meet the user requirements. At least one image corresponding to the prompt word is added in the text-to-image process, the image corresponding to the prompt word is used to control the characters and scenes in the output image, so that the image content in the output image can accurately represent the text meaning of the at least one prompt word, the matching degree of the output image and the multi-modal input information is improved, and the quality and accuracy of image generation are improved, so that the generated image can meet the user's requirements more. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is a schematic diagram of a scheme implementation environment provided by an embodiment of the present application;

[0024] Figure 2 is a flowchart of an image generation method provided by an embodiment of the present application;

[0025] Figure 3 is a schematic diagram of a complete image generation process provided by an embodiment of the present application;

[0026] Figure 4 is a block diagram of an image generation apparatus provided by an embodiment of the present application;

[0027] Figure 5 is a structural block diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0028] To make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application will be described in further detail below with reference to the drawings.

[0029] Reference should be made to Figure 1 which shows a schematic diagram of a scheme implementation environment provided by an embodiment of the present application. The scheme implementation environment can be implemented as an image generation system. The scheme implementation environment can include a terminal device 10 and a server 20.

[0030] The number of terminal devices 10 can be one or more. The terminal device 10 can be an electronic device such as a mobile phone, a tablet computer, a notebook computer, a desktop computer, a game console, an e-book reader, a multimedia playback device, a wearable device, a smart voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, etc.

[0031] The terminal device 10 can be installed with a client of a target application, the target application has a function of generating an image based on input information, a user can input input information of at least one modality in the target application, and the client of the target application generates a corresponding image according to the information input by the user and displays the image in a display interface of the target application. The type of the target application is not limited in the present application, including but not limited to a question and answer application, a music application, a video application, a game application, a web page application, etc. Optionally, the target application can be an application that needs to be downloaded and installed, or an application that can be used immediately, which is not limited in the present application.

[0032] The server 20 is configured to provide background services for the client of the target application installed and running in the terminal device 10. For example, the server 20 can be a background server of the target application. The server 20 can be a physical server, a server cluster composed of multiple servers, or a cloud computing service center. Optionally, the server 20 provides background services for multiple target applications in multiple terminal devices 10. The terminal device 10 and the server 20 can communicate with each other through a network.

[0033] In the embodiment of the present application, the user inputs input information of at least two modalities in the target application, and the terminal device sends the multi-modal input information input by the user to the server, and the server generates a corresponding output image according to the multi-modal input information. After receiving the multi-modal input information, the server first obtains a description text of the multi-modal input information, then obtains at least one prompt word corresponding to the description text, obtains images corresponding to the at least one prompt word from an image text library, and finally inputs the at least one prompt word and the images corresponding to the at least one prompt word into a text-to-image model to generate an output image corresponding to the multi-modal input information. In the case that the output image passes the yellow box detection, the server returns the output image to the terminal device, and displays the output image in the display interface of the target application.

[0034] Please refer to Figure 2 which shows a flowchart of an image generation method provided by an embodiment of the present application. The execution subject of each step of the method can be a server. The method can include at least one of the following steps 210-250:

[0035] Step 210, obtaining multi-modal input information, the multi-modal input information including input information of at least two modalities, the multi-modal input information being used to express an image generation requirement with input information of different modalities.

[0036] The modalities of the input information include a video modality, an image modality, an audio modality, and a text modality. The multi-modal input information includes at least two of the following: input information of the video modality, input information of the image modality, input information of the audio modality, and input information of the text modality.

[0037] The input information of the video modality can be understood as an input video, the input information of the image modality can be understood as an input image, the input information of the audio modality can be understood as an input audio, and the input information of the text modality can be understood as an input text. The multi-modal input information includes at least two of the input video, the input image, the input audio, and the input text.

[0038] The input information is used to control the effect of image generation. Different modalities of input information can represent different degrees of image generation requirements, thereby achieving different degrees of image generation effects. The multi-modal input information includes input information of at least two modalities, so that the information content contained in the multi-modal input information is more abundant, thereby enhancing the effect control of image generation.

[0039] In step 220, a description text of the multi-modal input information is obtained. The description text is a text describing the information content of the multi-modal input information.

[0040] The multi-modal understanding model generates a description text of the multi-modal input information according to the multi-modal input information. The input information of at least two modalities is respectively input into the multi-modal understanding model, and the description text corresponding to the input information of at least two modalities is output.

[0041] The multi-modal understanding model herein can be any publicly available large language model, such as a natural language model based on a transformer structure trained with a large amount of data. The amount of data can reach a sample level of hundreds of millions or more, and the present application does not limit this.

[0042] The multi-modal understanding model is used to extract the information content of the input information in the form of text, and the modalities of the input information include but are not limited to the video modality, the image modality, and the audio modality. For multi-modal input information, at least two modalities of input information in the multi-modal input information are respectively input into the multi-modal understanding model to obtain description texts corresponding to the input information of at least two modalities.

[0043] In some embodiments, when the multi-modal input information includes input information of the video modality, a video description text of the input video is obtained by the multi-modal understanding model according to the input video. Therefore, when the multi-modal input information includes input information of the video modality, the description text includes the video description text.

[0044] The video description text is text used to describe the information content of the input information of the video modality, i.e., the video description text is text used to describe the video content of the input video.

[0045] In some embodiments, in the case where the multi-modal input information includes input information of the image modality, the image description text is obtained by the multi-modal understanding model according to the input image. Then, in the case where the multi-modal input information includes input information of the image modality, the description text includes the image description text.

[0046] The image description text is text used to describe the information content of the input information of the image modality, i.e., the image description text is text used to describe the image content of the input image.

[0047] In some embodiments, in the case where the multi-modal input information includes input information of the audio modality, the audio description text is obtained by the multi-modal understanding model according to the input audio. Then, in the case where the multi-modal input information includes input information of the audio modality, the description text includes the audio description text.

[0048] The audio description text is text used to describe the information content of the input information of the audio modality, i.e., the audio description text is text used to describe the audio content of the input audio.

[0049] The description text includes at least two modalities of input information respectively corresponding to the description text. For example, if the multi-modal input information includes input information of the video modality, input information of the image modality, input information of the audio modality, and input information of the text modality, the description text includes the video description text, the image description text, the audio description text, and the input information of the text modality.

[0050] By using the multi-modal input information and obtaining the description text of the multi-modal input information, the text meaning of the generated description text is more rich, thereby the effect control on the image generation can be enhanced, and the generated image is more in line with the user demand.

[0051] In step 230, at least one prompt word corresponding to the description text is obtained, and the prompt word is used to indicate the image content of the image to be generated.

[0052] The at least one prompt word is a prompt word obtained based on the description text, but the at least one prompt word is not a word directly obtained from the description text. For example, the prompt word can be a formatted word including a word and a word type obtained from the description text.

[0053] The prompt word is used to indicate the image content of the image to be generated, i.e., the prompt word is used to indicate the image content of the output image. Specifically, the prompt word is used to indicate a person or a scene in the image to be generated.

[0054] Step 240, obtaining images corresponding to the at least one prompt word respectively from an image text library, the image corresponding to the prompt word being used to graphically display the text content represented by the prompt word, the image text library including at least one set of sample images and image texts corresponding to the sample images having a matching relationship, the image text being a description text for a person or a description text for a scene.

[0055] The image text library contains at least one set of image text pairs, each set of image text pairs including sample images and image texts having a matching relationship, the sample images being used to graphically display the text content represented by the image texts, and the image texts being used to textually describe the image content in the sample images.

[0056] Optionally, the image text is a description text for a person, and the image text includes but is not limited to a description text for a person's posture, a description text for a person's facial expression, a description text for a person's facial features, a description text for a person's body features, a description text for a person's clothing, and the like. Illustratively, the image text is a text describing a person's posture, and the image text includes but is not limited to a text describing a person's standing posture, a text describing a person's running posture, a text describing a person's cross-legged sitting posture, and the like. Illustratively, the image text is a text describing a person's facial expression, and the image text includes but is not limited to a text describing a person's smiling expression, a text describing a person's sad expression, a text describing a person's sorrowful expression, and the like. Illustratively, the image text is a text describing a person's facial features, and the image text includes but is not limited to a text describing a person's eye features, a text describing a person's lip features, a text describing a person's face shape features, and the like. Illustratively, the image text is a description text describing a person's body features, and the image text includes but is not limited to a text describing a person's body proportions, a text describing a person's fatness, a text describing a person's height, and the like.

[0057] Optionally, the image text is a description text for a scene, and the image text includes but is not limited to a description text for a scene layout, a description text for a scene element, a description text for a scene location, and the like. Illustratively, the image text is a text describing a scene layout, and the image text can be a text describing the positional relationship of various elements in the scene. Illustratively, the image text is a text describing a scene element, and the image text can be a text describing various elements in the scene. Illustratively, the image text is a text describing a scene location, and the image text can be a text describing the environment in which the scene is located, or a text describing the weather in which the scene is located.

[0058] Exemplarily, the image texts corresponding to the at least one prompt word respectively can be obtained from an image text library, and sample images respectively matched with the at least one image text can be obtained according to the image texts corresponding to the at least one prompt word respectively, so that the images corresponding to the at least one prompt word respectively can be obtained. The prompt words, the images corresponding to the prompt words, and the image texts matched with the images corresponding to the prompt words are mutually corresponding.

[0059] The images corresponding to the prompt words are used to graphically display the text content represented by the prompt words, and the images corresponding to the at least one prompt word respectively are used to graphically display the text content represented by the description text, so that the image generation content and the image generation effect are controlled through the images corresponding to the at least one prompt word respectively.

[0060] In step 250, the output image corresponding to the multi-modal input information is generated by the text-to-image model according to the at least one prompt word and the images corresponding to the at least one prompt word respectively.

[0061] The text-to-image model herein can be any publicly available diffusion model, such as a diffusion structure-based model trained with a large amount of data, such as a sample level of up to hundreds of millions, which is not limited in the present application.

[0062] The at least one prompt word and the images corresponding to the at least one prompt word respectively are input into the text-to-image model, and the output image corresponding to the multi-modal input information is output.

[0063] Optionally, the image resolution of the output image can be a default image resolution pre-set by the text-to-image model, for example, the default image resolution is 720x1280. Optionally, the image resolution of the output image can also be the maximum value of the image resolutions in the images corresponding to the at least one prompt word respectively. Optionally, the image resolution of the output image can also be the minimum value of the image resolutions in the images corresponding to the at least one prompt word respectively. Optionally, the image resolution of the output image can also be the intermediate value of the image resolutions in the images corresponding to the at least one prompt word respectively.

[0064] In some embodiments, the output image corresponding to the multi-modal input information is generated by the text-to-image model according to the at least one prompt word, the images corresponding to the at least one prompt word respectively, and a quality vocabulary, the quality vocabulary being used to indicate the image quality of the output image.

[0065] The quality vocabulary is a vocabulary describing the image quality of the output image. Optionally, the quality vocabulary can be a vocabulary describing the image resolution of the output image, for example, the quality vocabulary is "720x1280". Optionally, the quality vocabulary can also be a vocabulary describing the image definition of the output image, for example, the quality vocabulary is "high definition".

[0066] The at least one prompt word, the image corresponding to the at least one prompt word respectively, and the quality vocabulary are input into the text-to-image model, and an output image corresponding to the multi-modal input information is output. The image quality of the output image is the same as the image quality indicated by the quality vocabulary.

[0067] By adding the quality vocabulary when generating the output image, the image quality of the output image can be adjusted through the quality vocabulary, so that an image more in line with the user's requirements can be generated, the diversity of image generation is improved, and the image quality of the generated image is improved.

[0068] The technical scheme provided in the embodiments of the present application enriches the input information of the text-to-image model by using multi-modal input information, effectively enhances the effect control of the text-to-image model on image generation, so that the generated image can meet the user's requirements. At least one image corresponding to the prompt word is added in the text-to-image process, and the image corresponding to the prompt word is used to control the characters and scenes in the output image, so that the image content in the output image can accurately represent the text meaning of the at least one prompt word, improve the matching degree of the output image and the multi-modal input information, and improve the quality and accuracy of image generation, so that the generated image can meet the user's requirements better.

[0069] In some embodiments, step 230 includes at least one of sub-steps 231-233.

[0070] Sub-step 231: Obtain at least one entity word in the description text.

[0071] The description text is subjected to vocabulary splitting to obtain at least one word in the description text. According to the vocabulary meaning of the at least one word, at least one entity word is obtained from the at least one word.

[0072] The entity word refers to a word with a clear reference object or a word with actual meaning, for example, the entity word can be a noun, an adjective, a pronoun, etc.

[0073] Sub-step 232: According to at least one rule-breaking word in the rule-breaking word library, the word in the at least one entity word that matches the rule-breaking word is removed to obtain the removed word, and the rule-breaking word is a word used to represent rule-breaking content.

[0074] The violation vocabulary library includes at least one violation vocabulary, and the violation vocabulary is a vocabulary used to represent violation content. The violation content includes, but is not limited to, pornographic content, vulgar content, political content, bloody content, violent content, horror content, content of bad values, etc. There can be at least one violation vocabulary in the violation vocabulary library to represent the same violation content, and the number of violation vocabularies used to represent different violation contents can be the same or different. For example, there can be one violation vocabulary in the violation vocabulary library to represent pornographic content, or there can be multiple violation vocabularies to represent pornographic content. If there is one violation vocabulary in the violation vocabulary library to represent pornographic content, the number of violation vocabularies in the violation vocabulary library used to represent other violation contents other than pornographic content can be one or multiple.

[0075] In some embodiments, for each entity word, the similarity between the entity word and at least one violation vocabulary in the violation vocabulary library is calculated respectively. In the case that the maximum similarity in the similarity between the entity word and the at least one violation vocabulary is greater than or equal to a third threshold value, it can be determined that the entity word matches the violation vocabulary corresponding to the maximum similarity, and the entity word is then excluded from the at least one entity word. In the case that the maximum similarity in the similarity between the entity word and the at least one violation vocabulary is less than the third threshold value, it can be determined that the entity word does not match any of the at least one violation vocabulary in the violation vocabulary library, and the entity word does not need to be excluded from the at least one entity word. Referring to the above steps, it is respectively determined whether there is a violation vocabulary in the violation vocabulary library that matches each entity word in the at least one entity word, and the entity word is excluded from the at least one entity word in the case that there is a violation vocabulary in the violation vocabulary library that matches the entity word, and the entity word is retained in the at least one entity word in the case that there is no violation vocabulary in the violation vocabulary library that matches the entity word. After the at least one entity word is processed, the excluded vocabulary is obtained.

[0076] Optionally, the similarity between the word vector of the entity word and the word vector of the at least one violation vocabulary can be calculated by cosine similarity or Euclidean distance. Optionally, the similarity between the entity word and the violation vocabulary can also be obtained by a text matching model according to the entity word and the violation vocabulary.

[0077] The specific value of the third threshold value is set by the technician according to the vocabulary exclusion requirement, and the present application is not limited.

[0078] Each of the filtered words is a word that does not match any of the at least one rule-breaking word in the rule-breaking word library, but it needs to be noted that, since the maximum value in the similarity between each of the filtered words and the at least one rule-breaking word is less than the third threshold value, in the case where the third threshold value is not 0, there can be a word in the filtered words that represents a part of the rule-breaking content. That is, each of the filtered words does not completely contain no rule-breaking content.

[0079] In sub-step 233, at least one target word in the filtered words is extracted by the format extraction model to obtain at least one prompt word, the target word being a word describing a character or a word describing a scene, and the prompt word including the target word and a word type of the target word.

[0080] The format extraction model is used to extract at least one target word in the filtered words, and the target word is generated as a formatted target word, i.e., a prompt word.

[0081] The target word is a word in the filtered words for describing a character, or a word in the filtered words for describing a scene. The word for describing a character includes, but is not limited to, a word for describing a character posture, a word for describing a character facial expression, a word for describing a character facial feature, a word for describing a character morphological feature, a word for describing a character costume, and the like. The word for describing a scene includes, but is not limited to, a word for describing a scene layout, a word for describing a scene element, a word for describing a scene position, and the like.

[0082] The formatted target word (prompt word) is a specific format word generated on the basis of the target word. In the embodiments of the present application, the prompt word includes the target word and the type of the target word. The type of the target word is a character or a scene, so in the case where the target word is a word for describing a character, the prompt word includes the target word and the word type of the target word "character", and in the case where the target word is a word for describing a scene, the prompt word includes the target word and the word type of the target word "scene".

[0083] Exemplarily, the prompt word can be expressed in the format of {“character”: role 1}, {“character”: role 2}, {“scene”: seaside}, {“scene”: beach}.

[0084] By removing the words in the at least one entity word that match the rule-breaking words, the filtered words do not contain words that match the rule-breaking words, so that the probability of the generated output image containing rule-breaking content can be reduced, the quality of image generation can be improved, and the yellow anti-interception rate can be reduced.

[0085] In some embodiments, step 240 includes at least one of sub-steps 241-243.

[0086] Sub-step 241, obtaining at least one image text corresponding to each target vocabulary from the image text library, to obtain at least one image text.

[0087] In some embodiments, for each target vocabulary, the similarity between the target vocabulary and each image text in the image text library is calculated respectively. The image text corresponding to the maximum similarity between the target vocabulary and each image text in the image text library is determined as the image text corresponding to the target vocabulary. Referring to the above steps, at least one image text corresponding to each target vocabulary is obtained, that is, at least one image text is obtained.

[0088] Optionally, the similarity between the word vector of the target vocabulary and the text vector of each image text in the image text library can be calculated by cosine similarity or Euclidean distance. Optionally, the similarity between the target vocabulary and the image text can also be obtained by the text matching model according to the target vocabulary and the image text.

[0089] Sub-step 242, obtaining at least one sample image corresponding to each image text from the image text library.

[0090] According to the matching relationship between the sample image and the image text in the image text library, at least one sample image corresponding to each image text is obtained from the image text library.

[0091] Sub-step 243, determining at least one sample image corresponding to each image text as at least one prompt word corresponding to the image.

[0092] By obtaining at least one image corresponding to each prompt word from the image text library, the implementation of the prompt word in the output image can be controlled individually for each prompt word, the precise control of the person and the scene in the output image is realized, the quality of the image generation is improved, and the distortion rate of the output image can be effectively reduced.

[0093] In some embodiments, step 250 includes at least one of sub-steps 251-253.

[0094] Sub-step 251, in the case where the target input information is included in the multi-modal input information, obtaining a segmentation image of the target input information by the person segmentation model according to the target input information, the target input information being input information of a video modality or input information of an image modality.

[0095] The person segmentation model is used for person segmentation of an image containing a person, and obtains a person region in the image, that is, a segmentation image. The segmentation image is an image containing only the person region. The person segmentation model here can be any publicly available person segmentation model, which is not limited in the present application.

[0096] In some embodiments, in the case that the target input information is input information of a video modality, if the target input information is included in the multi-modal input information, a segmentation image of the input information of the video modality is obtained by the person segmentation model according to the input information of the video modality.

[0097] The input information of the video modality (input video) contains at least two video frames. The person segmentation model is used to perform person segmentation on the at least two video frames respectively, and the person regions in the at least two video frames are obtained, that is, the segmentation images corresponding to the at least two video frames are obtained. The segmentation image of the target input information (segmentation image of the input information of the video modality) includes the segmentation images corresponding to the at least two video frames.

[0098] In some embodiments, in the case that the target input information is input information of an image modality, if the target input information is included in the multi-modal input information, a segmentation image of the input information of the image modality is obtained by the person segmentation model according to the input information of the image modality.

[0099] The input information of the image modality (input image) is an image. Therefore, the segmentation image of the target input information (segmentation image of the input information of the image modality) is the segmentation image corresponding to the image.

[0100] In substep 252, the person feature corresponding to the segmentation image is obtained by the visual language model according to the segmentation image.

[0101] The visual language model is used to extract relevant information from the input image to realize visual question answering. In the embodiments of the present application, the visual language model is used to extract the image feature in the segmentation image. Since the segmentation image is an image containing only the person region, the visual language model can extract the person feature corresponding to the segmentation image.

[0102] The visual language model herein can be any publicly available visual language model, which is not limited in the present application. For example, the visual language model can be a CLIP model.

[0103] In the case that the input information of the text-to-image model contains the quality vocabulary, substep 253 is performed. In the case that the input information of the text-to-image model does not contain the quality vocabulary, the output image corresponding to the multi-modal input information is generated by the text-to-image model according to the at least one prompt word, the image corresponding to the at least one prompt word, and the person feature.

[0104] In substep 253, the output image corresponding to the multi-modal input information is generated by the text-to-image model according to the at least one prompt word, the image corresponding to the at least one prompt word, the quality vocabulary, and the person feature.

[0105] Input the at least one prompt word, the image corresponding to the at least one prompt word respectively, the quality vocabulary and the character feature into the image generation model, and output an output image corresponding to the multi-modal input information.

[0106] By adding the character feature in the multi-modal input information in the process of generating the image, the control of the character generation in the image generation can be enhanced, the character image in the generated output image is more accurate and prominent, and the quality of the image generation is improved.

[0107] In some embodiments, step 250 is followed by step 260, and step 260 includes at least one of sub-steps 261-262.

[0108] Sub-step 261, in the case where the output image passes the yellow and reverse detection, the output image is determined as the final output image.

[0109] Sub-step 262, in the case where the output image does not pass the yellow and reverse detection, step 250 is re-executed.

[0110] The yellow and reverse detection is used to detect whether the image contains illegal content. If the output image passes the yellow and reverse detection, it can be considered that the output image does not contain illegal content, and the output image can be determined as the final output image. If the output image does not pass the yellow and reverse detection, it can be considered that the output image contains illegal content, and the output image does not meet the requirement of the final output, and the output image needs to be re-generated, that is, step 250 is re-executed.

[0111] It should be noted that the output image passing the yellow and reverse detection does not mean that the output image does not contain illegal content at all, but means that the illegal content contained in the output image is within the allowable range of the yellow and reverse detection.

[0112] By performing the yellow and reverse detection on the output image and re-generating the output image that does not pass the yellow and reverse detection, the safety of the output image obtained by the user can be ensured while ensuring the quality of the image, and the adverse effects of the illegal content contained in the output image on the user can be avoided.

[0113] The yellow and reverse detection on the output image can be performed in the following two ways: one is to perform the yellow and reverse detection through a yellow and reverse detection model, and the other is to perform the yellow and reverse detection through a defect detection model.

[0114] In some embodiments, the yellow and reverse detection model obtains a yellow and reverse detection result of the output image according to the output image, the yellow and reverse detection model is a pre-trained binary classification model; in the case where the yellow and reverse detection result is a first numerical value, it is determined that the output image passes the yellow and reverse detection; and in the case where the yellow and reverse detection result is a second numerical value, it is determined that the output image does not pass the yellow and reverse detection.

[0115] The yellow reverse detection model is a pre-trained binary classification model. The output image is output to the yellow reverse detection model, and a yellow reverse detection result of the output image is output. The yellow reverse detection result is a first numerical value or a second numerical value. The first numerical value is used to indicate that the output image passes the yellow reverse detection, and the second numerical value is used to indicate that the output image fails the yellow reverse detection.

[0116] The specific numerical values of the first numerical value and the second numerical value are not limited in the present application. For example, the first numerical value is 0, and the second numerical value is 1. In the case of the yellow reverse detection result being 0, it is determined that the output image passes the yellow reverse detection. In the case of the yellow reverse detection result being 1, it is determined that the output image fails the yellow reverse detection.

[0117] The yellow reverse detection is performed by the binary classification model. It is easier to determine whether the output image passes the yellow reverse detection through the numerical value output of the model, and the convenience of the yellow reverse detection is improved.

[0118] In the case where the output image fails the yellow reverse detection, the flaw detection model can be further used to obtain a flaw rate of the output image according to the output image, and then step 250 is re-executed. In the case where the number of regenerated images of the output image reaches a second threshold value and each regenerated image fails the yellow reverse detection, the output image corresponding to the minimum value of the flaw rate in the flaw rates of the regenerated images of the output image is determined as the final output image.

[0119] In some embodiments, the flaw detection model is used to obtain a flaw rate of the output image according to the output image. The flaw detection model is a pre-trained machine learning model, and the flaw rate is used to indicate the content of the illegal content in the output image. In the case where the flaw rate of the output image is less than a first threshold value, it is determined that the output image passes the yellow reverse detection. In the case where the flaw rate of the output image is greater than or equal to the first threshold value, it is determined that the output image fails the yellow reverse detection.

[0120] The flaw detection model is a pre-trained machine learning model. The output image is input to the flaw detection model, and a flaw rate of the output image is output. The numerical range of the flaw rate is 0 to 1, and the flaw rate is used to indicate the content of the illegal content in the output image. The lower the flaw rate, the lower the content of the illegal content in the output image. The higher the flaw rate, the higher the content of the illegal content in the output image. If the flaw rate is 0, it indicates that the output image does not contain illegal content.

[0121] The specific numerical value of the first threshold value is set by the technician according to the yellow reverse detection requirement, and is not limited in the present application. For example, the first threshold value can be 0.3. In the case where the flaw rate of the output image is less than 0.3, it is determined that the output image passes the yellow reverse detection. In the case where the flaw rate of the output image is greater than or equal to 0.3, it is determined that the output image fails the yellow reverse detection.

[0122] The yellow reverse detection is performed by the defect detection model, and the content of the violation contained in the output image can be directly observed according to the defect rate, so that it is determined whether the image passes the yellow reverse detection, and the accuracy of the yellow reverse detection is improved.

[0123] In some embodiments, when the number of times of regeneration of the output image reaches the second threshold, and each of the regenerated images does not pass the yellow reverse detection, the output image corresponding to the minimum value of the defect rate in the defect rates of the regenerated images is determined as the final output image.

[0124] The specific value of the second threshold is set by the technician according to the yellow reverse detection requirement, and the present application is not limited. Exemplarily, the second threshold is 5, and when the number of times of regeneration of the output image reaches 5, and each of the 5 regenerated images does not pass the yellow reverse detection, the output image corresponding to the minimum value of the defect rate in the defect rates of the 5 regenerated images is determined as the final output image.

[0125] By setting an upper limit value for the number of times of regeneration of the regenerated image, it is avoided that the regenerated image does not pass the yellow reverse detection and the step 250 is always performed again, so that the calculation resource is saved while the image generation quality is ensured, and the efficiency of image generation is ensured.

[0126] Figure 3A schematic diagram showing the complete process of image generation is shown, and the multi-modal input information includes input video, input image, input audio and input text. The input video is decoded to obtain the decoded input video, on the one hand, the video description text of the input video is obtained from the decoded input video by the multi-modal understanding model, on the other hand, the segmentation image of the input video is obtained from the decoded input video by the character segmentation model, and then the character features corresponding to the input video are obtained by the visual language model. The input image is image backstepping, on the one hand, the image description text of the input image is obtained from the input image by the multi-modal understanding model, on the other hand, the segmentation image of the input image is obtained from the input image by the character segmentation model, and then the character features corresponding to the input image are obtained by the visual language model. The input audio is audio backstepping, and the audio description text of the input audio is obtained from the input audio by the multi-modal understanding model. The video description text, the image description text, the audio description text and the input text are subjected to text yellow judgment, in the case that the text does not pass the yellow judgment, at least one entity word in the text is obtained, the words in the at least one entity word that match the rule violation words are removed to obtain the removed words, at least one target word in the removed words is extracted by the format extraction model to obtain at least one prompt word, and the prompt word includes the target word and the word type of the target word. At least one prompt word respectively corresponds to the image obtained from the image text library, the image text library contains sample images and image texts corresponding to the sample images with a matching relationship, and the image text includes but is not limited to text describing the posture of a character, text describing the face of a character, text describing the layout of a scene, text describing the elements of a scene, etc. The output image is generated from the at least one prompt word, the image corresponding to the at least one prompt word, the quality word and the character features by the text-to-image model. The output image is subjected to image yellow judgment, if the output image passes the yellow judgment, the output image is determined as the final output image, and the image generation process is ended. If the output image does not pass the yellow judgment, the output image is regenerated in a loop until the number of regeneration reaches an upper limit, and the output image corresponding to the maximum value of the defect rate in the defect rate of the regenerated image is determined as the final output image.

[0127] The following is an embodiment of the device of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0128] Please refer to Figure 4 , which shows a block diagram of an image generation device according to an embodiment of the present application. The device has the functions of implementing the above-mentioned image generation method, which can be realized by hardware or by corresponding software executed by hardware. The device can be the server introduced above, or can be arranged in the server. For example, Figure 4As shown, the apparatus 400 can include an input obtaining module 410, a text obtaining module 420, a prompt word obtaining module 430, an image obtaining module 440, and an output generating module 450.

[0129] The input obtaining module 410 is configured to obtain multi-modal input information, the multi-modal input information including input information of at least two modalities, the multi-modal input information being used to express an image generation requirement with input information of different modalities.

[0130] The text obtaining module 420 is configured to obtain description text of the multi-modal input information, the description text being text describing information content of the multi-modal input information.

[0131] The prompt word obtaining module 430 is configured to obtain at least one prompt word corresponding to the description text, the prompt word being used to indicate image content of an image to be generated.

[0132] The image obtaining module 440 is configured to obtain images corresponding to the at least one prompt word respectively from an image text library, the images corresponding to the prompt word being used to graphically display text content represented by the prompt word, the image text library including at least one set of sample images and image texts corresponding to the sample images having a matching relationship, the image texts being description texts for a person or a scene.

[0133] The output generating module 450 is configured to generate, by a text-to-image model, an output image corresponding to the multi-modal input information according to the at least one prompt word and the images corresponding to the at least one prompt word respectively.

[0134] In some embodiments, the text obtaining module 420 is configured to:

[0135] In a case where the multi-modal input information includes input information of the video modality, obtain, by a multi-modal understanding model, a video description text of the input video according to the input video, the description text including the video description text.

[0136] In a case where the multi-modal input information includes input information of the image modality, obtain, by the multi-modal understanding model, an image description text of the input image according to the input image, the description text including the image description text.

[0137] In a case where the multi-modal input information includes input information of the audio modality, obtain, by the multi-modal understanding model, an audio description text of the input audio according to the input audio, the description text including the audio description text.

[0138] In some embodiments, the prompt word obtaining module 430 is configured to:

[0139] obtain at least one entity word in the description text;

[0140] eliminate, according to at least one violation word in a violation word library, a word in the at least one entity word that matches the violation word, to obtain an eliminated word, the violation word being a word used to represent violation content;

[0141] extract, by a format extraction model, at least one target word in the eliminated word, to obtain at least one prompt word, the target word being a word describing a character or a word describing a scene, the prompt word including the target word and a word type of the target word.

[0142] In some embodiments, the image obtaining module 440 is configured to:

[0143] obtain, from the image text library, an image text corresponding to each of the at least one target word, to obtain at least one image text;

[0144] obtain, from the image text library, a sample image corresponding to each of the at least one image text;

[0145] determine, as an image corresponding to each of the at least one prompt word, the sample image corresponding to each of the at least one image text.

[0146] In some embodiments, the output generating module 450 is configured to:

[0147] generate, by the text-to-image model, an output image corresponding to the multi-modal input information, according to the at least one prompt word, the image corresponding to each of the at least one prompt word, and a quality word used to indicate an image quality of the output image.

[0148] In some embodiments, the output generating module 450 is configured to:

[0149] in a case where the multi-modal input information includes target input information, obtain, by a character segmentation model, a segmented image of the target input information, according to the target input information, the target input information being input information of a video modality or input information of an image modality;

[0150] obtain, by a visual language model, a character feature corresponding to the segmented image, according to the segmented image;

[0151] generate, by the text-to-image model, the output image corresponding to the multi-modal input information, according to the at least one prompt word, the image corresponding to each of the at least one prompt word, the quality word, and the character feature.

[0152] In some embodiments, the apparatus 400 further includes an image detection module, configured to:

[0153] In a case where the output image passes the yellow-back detection, determining the output image as the final output image;

[0154] In a case where the output image fails the yellow-back detection, re-executing the step of generating the output image corresponding to the multi-modal input information according to the at least one prompt word and the image corresponding to the at least one prompt word, respectively, by the text-to-image model.

[0155] In some embodiments, the image detection module is configured to:

[0156] obtaining, by a yellow-back detection model, a yellow-back detection result of the output image according to the output image, the yellow-back detection model being a pre-trained binary classification model;

[0157] In a case where the yellow-back detection result is a first numerical value, determining that the output image passes the yellow-back detection;

[0158] In a case where the yellow-back detection result is a second numerical value, determining that the output image fails the yellow-back detection.

[0159] In some embodiments, the image detection module is configured to:

[0160] obtaining, by a defect detection model, a defect rate of the output image according to the output image, the defect detection model being a pre-trained machine learning model, the defect rate being used to indicate the content of the violation in the output image;

[0161] In a case where the defect rate of the output image is less than the first threshold value, determining that the output image passes the yellow-back detection;

[0162] In a case where the defect rate of the output image is greater than or equal to the first threshold value, determining that the output image fails the yellow-back detection.

[0163] In some embodiments, the image detection module is configured to:

[0164] In a case where the number of times of re-generating the output image reaches a second threshold value, and each re-generated image fails the yellow-back detection, determining, as the final output image, the output image corresponding to the minimum value of the defect rates of the re-generated images.

[0165] It should be noted that the apparatus provided by the above embodiments, in realizing its functions, is only exemplified by the above division of functional modules, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the content structure of the device is divided into different functional modules to complete all or part of the above-described functions. In addition, the apparatus and method embodiments provided by the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0166] Reference is made to Figure 5 which shows a structural block diagram of a computer device 500 provided by an embodiment of the present application. The computer device 500 can be any electronic device with data computing, processing and storage functions. The computer device 500 can be used to implement the image generation method provided in the above embodiments.

[0167] Generally, the computer device 500 includes a processor 501 and a memory 502.

[0168] The processor 501 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 501 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), a FPGA (Field Programmable Gate Array), a PLA (Programmable Logic Array). The processor 501 can also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit). The coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 501 can be integrated with a GPU (Graphics Processing Unit) that is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 501 can also include an AI processor for processing machine learning-related computing operations.

[0169] The memory 502 can include one or more computer-readable storage media, which can be non-transitory. The memory 502 can also include a high-speed random access memory and a non-volatile memory such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 502 is configured to store a computer program configured to be executed by one or more processors to implement the above image generation method.

[0170] Those skilled in the art can understand that, Figure 5 The structure shown in the figure is not a limitation of the computer device 500, and can include more or fewer components than the figure, or combine certain components, or use different component arrangements.

[0171] In the illustrative embodiment, a computer readable storage medium is also provided, the storage medium storing a computer program, the computer program being executed by the processor of the computer device to implement the above image generation method. Optionally, the above computer readable storage medium can be a ROM (Read-Only Memory), a RAM (Random Access Memory), a CD-ROM (Compact Disc Read-Only Memory), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0172] In the exemplary embodiment, a computer program product is also provided, the computer program product including a computer program stored in a computer readable storage medium. The processor of the computer device reads the computer program from the computer readable storage medium, and the processor executes the computer program to make the computer device execute the above image generation method.

[0173] It should be understood that "multiple" referred to herein refers to two or more. The "and / or" describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. The character " / " generally represents that the associated objects before and after it are in an "or" relationship. In addition, the step numbers described herein only exemplarily show a possible execution order between steps, and in some other embodiments, the above steps can also be executed in a different order from the number, such as simultaneously executing two steps with different numbers, or executing two steps with different numbers in an order opposite to the figure. The embodiments of the present application do not limit this.

[0174] The above only describes exemplary embodiments of the present application and does not limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An image generation method characterized by, The method includes: Acquire multimodal input information, which includes input information from at least two modalities, and the multimodal input information is used to represent image generation requirements in different modalities. Obtain the descriptive text of the multimodal input information, wherein the descriptive text is text that describes the information content of the multimodal input information; Obtain at least one prompt word corresponding to the description text, the prompt word being used to indicate the image content of the image to be generated; Images corresponding to the at least one prompt word are obtained from the image text library. The images corresponding to the prompt words are used to graphically display the text content represented by the prompt words. The image text library includes at least one set of sample images with matching relationships and image text corresponding to the sample images. The image text is descriptive text for people or descriptive text for scenes. The output image corresponding to the multimodal input information is generated by the text-based image model based on the at least one prompt word and the image corresponding to the at least one prompt word.

2. The method of claim 1, wherein, The descriptive text for acquiring the multimodal input information includes: When the multimodal input information includes video modal input information, the multimodal understanding model obtains the video description text of the input video based on the input video, and the description text includes the video description text. When the multimodal input information includes image modality input information, the multimodal understanding model obtains image description text of the input image based on the input image, and the description text includes the image description text. When the multimodal input information includes audio modal input information, the multimodal understanding model obtains an audio description text of the input audio based on the input audio, and the description text includes the audio description text.

3. The method of claim 1, wherein, The step of obtaining at least one prompt word corresponding to the description text based on the description text includes: Obtain at least one entity word from the description text; Based on at least one illegal word in the illegal word database, words that match the illegal word in the at least one entity word are removed to obtain the removed words, where the illegal word is a word used to represent illegal content; At least one target word is extracted from the removed words using a formatted extraction model to obtain at least one prompt word. The target word is a word describing a person or a scene. The prompt word includes the target word and the word type of the target word.

4. The method of claim 3, wherein, The step of obtaining the images corresponding to each of the at least one prompt word from the image text library includes: Obtain the image text corresponding to each of the at least one target word from the image text library to obtain at least one image text; Obtain sample images corresponding to each of the at least one image text from the image text library; The sample images corresponding to the at least one image text are determined as the images corresponding to the at least one prompt word.

5. The method of claim 1, wherein, The step of generating an output image corresponding to the multimodal input information using a text-based image model based on the at least one prompt word and the image corresponding to the at least one prompt word includes: The text-based image model generates an output image corresponding to the multimodal input information based on the at least one prompt word, the image corresponding to the at least one prompt word, and quality words, respectively. The quality words are used to indicate the image quality of the output image.

6. The method of claim 5, wherein, The step of generating an output image corresponding to the multimodal input information using the text-based image model based on the at least one prompt word, the image corresponding to the at least one prompt word, and quality vocabulary includes: When the multimodal input information includes target input information, a segmented image of the target input information is obtained by a person segmentation model based on the target input information, wherein the target input information is video modal input information or image modal input information; Based on the segmented image, the visual language model is used to obtain the human features corresponding to the segmented image; The text-based image model generates an output image corresponding to the multimodal input information based on the at least one prompt word, the image corresponding to the at least one prompt word, the quality vocabulary, and the character features.

7. The method of claim 1, wherein, After generating the output image corresponding to the multimodal input information using the text-based graph model based on the at least one prompt word and the image corresponding to the at least one prompt word, the method further includes: If the output image passes the yellow reflection detection, the output image is determined as the final output image; If the output image fails the yellow-screen detection, the step of generating the output image corresponding to the multimodal input information by using the text-based image model based on the at least one prompt word and the image corresponding to the at least one prompt word is executed again.

8. The method of claim 7, wherein, The method further includes: The yellow back detection model is used to obtain the yellow back detection result of the output image based on the output image. The yellow back detection model is a pre-trained binary classification model. If the yellow reflection detection result is the first value, it is determined that the output image passes the yellow reflection detection. If the yellow reflection detection result is the second value, it is determined that the output image has failed the yellow reflection detection.

9. The method according to claim 7, characterized in that, The method further includes: The defect rate of the output image is obtained by using a defect detection model based on the output image. The defect detection model is a pre-trained machine learning model, and the defect rate is used to indicate the amount of non-compliant content in the output image. If the defect rate of the output image is less than a first threshold, the output image is determined to pass the yellow reflection detection. If the defect rate of the output image is greater than or equal to the first threshold, the output image is determined to have failed the yellow reflection detection.

10. The method according to claim 9, characterized in that, After re-executing the step of generating the output image corresponding to the multimodal input information using the text-based graph model based on the at least one prompt word and the image corresponding to the at least one prompt word, the method further includes: If the number of times the output image is regenerated reaches a second threshold, and each regenerated image fails the yellow back detection, the output image corresponding to the minimum defect rate among the regenerated images is determined as the final output image.

11. An image generation apparatus, characterized in that, The device includes: An input acquisition module is used to acquire multimodal input information, which includes input information of at least two modalities, and the multimodal input information is used to represent image generation requirements with input information of different modalities; The text acquisition module is used to acquire the descriptive text of the multimodal input information, wherein the descriptive text is text that describes the information content of the multimodal input information; The prompt word acquisition module is used to acquire at least one prompt word corresponding to the description text, wherein the prompt word is used to indicate the image content of the image to be generated; The image acquisition module is used to acquire images corresponding to the at least one prompt word from the image text library. The images corresponding to the prompt words are used to graphically display the text content represented by the prompt words. The image text library includes at least one set of sample images with matching relationships and image text corresponding to the sample images. The image text is descriptive text for people or descriptive text for scenes. The output generation module is used to generate an output image corresponding to the multimodal input information by using a text-based image model based on the at least one prompt word and the image corresponding to the at least one prompt word.

12. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program that is loaded and executed by the processor to implement the image generation method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the image generation method as described in any one of claims 1 to 10.

14. A computer program product, characterized in that, The computer program product includes a computer program that is loaded and executed by a processor to implement the image generation method as described in any one of claims 1 to 10.

Images