Image generation method and device, agent, agent system and storage medium

Abstract

The present disclosure provides an image generation method and device, an agent, an agent system and a storage medium, relates to the field of artificial intelligence, in particular to the field of computer vision, deep learning and large model, and can be applied to the scene of content generation based on artificial intelligence. The specific implementation scheme is as follows: acquiring image generation requirement information; determining a corresponding target image generation mode according to the image generation requirement information; querying a first reference image based on the image generation requirement information; and generating a target image by using the target image generation mode based on the image generation requirement information and the first reference image.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical field of computer vision, deep learning, large model, etc., can be applied to the scene of artificial intelligence generated content (AIGC), and particularly relates to an image generation method and device, an agent, an agent system and a storage medium. BACKGROUND

[0002] The core of the AI image generation technology based on AIGC is to realize the conversion from text to image or image to image by using an AI image generation model. However, the image generation effect of the AI image generation technology is highly dependent on the quality and timeliness of the model training data. Specifically, after the AI image generation model is trained, the image content generated by the AI image generation model is often limited by the timeliness of the model training data. Before the model training data is updated, the AI image generation model cannot obtain the latest content, and the image generated by the AI image generation model may have a certain lag. SUMMARY

[0003] The present disclosure provides an image generation method, device, agent, agent system and storage medium.

[0004] According to a first aspect of the present disclosure, an image generation method is provided, comprising: obtaining image generation requirement information; determining a corresponding target image generation manner according to the image generation requirement information; querying a first reference image based on the image generation requirement information; and generating a target image by using the target image generation manner based on the image generation requirement information and the first reference image.

[0005] According to a second aspect of the present disclosure, an image generation method is provided, comprising: obtaining image generation requirement information; determining a corresponding target image generation manner according to the image generation requirement information; determining whether the image generation requirement information needs to be queried for a reference image; in the case that the image generation requirement information needs to be queried for a reference image, querying a first reference image based on the image generation requirement information; and generating a target image by using the target image generation manner based on the image generation requirement information and the first reference image.

[0006] According to a third aspect of the present disclosure, an image generation device is provided, comprising: an obtaining module configured to obtain image generation requirement information; a determining module configured to determine a corresponding target image generation manner according to the image generation requirement information; a querying module configured to query a first reference image based on the image generation requirement information; and a generating module configured to generate a target image by using the target image generation manner based on the image generation requirement information and the first reference image.

[0007] According to a fourth aspect of the present disclosure, an image generation apparatus is provided, comprising: an acquisition module configured to acquire image generation requirement information; a determination module configured to determine a corresponding target image generation manner according to the image generation requirement information; a judgment module configured to judge whether the image generation requirement information needs to be queried for a reference image; a query module configured to, in a case where the image generation requirement information needs to be queried for a reference image, query a first reference image based on the image generation requirement information; and a generation module configured to generate a target image based on the image generation requirement information and the first reference image by using the target image generation manner.

[0008] According to a fifth aspect of the present disclosure, an agent is provided, comprising: an input module configured to acquire image generation requirement information; a processing module configured to determine a corresponding target image generation manner according to the image generation requirement information, query a first reference image based on the image generation requirement information, and generate a target image based on the image generation requirement information and the first reference image by using the target image generation manner; and an output module configured to output the target image.

[0009] According to a sixth aspect of the present disclosure, an agent is provided, comprising: an input module configured to acquire image generation requirement information; a processing module configured to determine a corresponding target image generation manner according to the image generation requirement information, judge whether the image generation requirement information needs to be queried for a reference image, query a first reference image based on the image generation requirement information in a case where the image generation requirement information needs to be queried for a reference image, and generate a target image based on the image generation requirement information and the first reference image by using the target image generation manner; and an output module configured to output the target image.

[0010] According to a seventh aspect of the present disclosure, an agent system is provided, comprising:

[0011] at least one processor; and

[0012] a memory connected with the at least one processor in communication; wherein

[0013] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the image generation method according to the first aspect or the image generation method according to the second aspect.

[0014] According to an eighth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the image generation method according to the first aspect or the image generation method according to the second aspect.

[0015] According to a ninth aspect of the present disclosure, there is provided a computer program product comprising computer instructions which, when executed by a processor, implement the steps of the image generation method according to the first aspect or the image generation method according to the second aspect.

[0016] The image generation method, device, agent, agent system and storage medium provided by the present disclosure have the following beneficial effects:

[0017] The image generation requirement information is obtained, the corresponding target image generation mode is determined according to the image generation requirement information, the first reference image is obtained based on the image generation requirement information, and the target image is generated based on the image generation requirement information and the first reference image by using the target image generation mode. The present disclosure generates the target image based on the first reference image obtained by querying, that is, the image Retrieval-Augmented Generation (iRAG) technology is used, the timeliness of image generation is aligned with the timeliness of image search, and the problem of possible lag of the target image is avoided. In addition, since the image search almost covers all public knowledge, there is no memory capacity bottleneck, and the problem of limited memory capacity of the AI image generation model is also solved.

[0018] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings are used to better understand the present scheme and do not limit the present disclosure. Among them:

[0020] Figure 1 is a flowchart of an image generation method according to an embodiment of the present disclosure;

[0021] Figure 2 is a flowchart of an image generation method according to another embodiment of the present disclosure;

[0022] Figure 3 is a flowchart of an image generation method according to another embodiment of the present disclosure;

[0023] Figure 4 is a schematic diagram of the image-text interleaving feature according to an embodiment of the present disclosure;

[0024] Figure 5 is a flowchart of an image generation method according to another embodiment of the present disclosure;

[0025] Figure 6 is a flowchart of an image generation method according to an embodiment of the present disclosure;

[0026] Figure 7 is a flowchart of an image generation method according to another embodiment of the present disclosure;

[0027] Figure 8 is a structural schematic diagram of an image generation apparatus according to an embodiment of the present disclosure;

[0028] Figure 9 is a structural schematic diagram of an image generation apparatus according to an embodiment of the present disclosure;

[0029] Figure 10 is a schematic diagram of an agent according to an embodiment of the present disclosure;

[0030] Figure 11 is a block diagram of an agent system for implementing an image generation method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to assist in understanding, and should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0032] In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solutions are carried out on the premise of obtaining the consent of the user, and comply with the relevant legal regulations and do not violate public order and good customs.

[0033] The image generation method, apparatus, agent, agent system and storage medium of the embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0034] It should be noted that the execution subject of the image generation method of the present embodiment is an image generation apparatus, which can be realized by software and / or hardware, and can be configured in an agent.

[0035] Figure 1 is a flowchart of an image generation method according to an embodiment of the present disclosure.

[0036] As Figure 1As shown, the image generation method comprises:

[0037] In step 101, image generation requirement information is obtained.

[0038] The image generation requirement information can be used to indicate image style, image subject, image size, and the like of the image to be generated.

[0039] As an example, the image generation requirement information can include a user input image generation requirement prompt.

[0040] As another example, the image generation requirement information can include a user input prompt and a reference image.

[0041] In step 102, a corresponding target image generation manner is determined according to the image generation requirement information.

[0042] The image generation manner corresponds to the image generation requirement, and different image generation requirements require different image generation manners. For example, different AI image generation models can be used for different image generation manners.

[0043] As an example, the target image generation manner corresponding to the image generation requirement information can be determined according to the correspondence between the image generation requirement and the image generation manner.

[0044] As an example, the target image generation manner corresponding to the image generation requirement information can be determined based on a large model.

[0045] In step 103, a first reference image is obtained based on the image generation requirement information.

[0046] The image generation requirement information can be used to query images in a set image library to obtain the first reference image. It should be noted that the first reference image can be one or more.

[0047] As an example, the vector feature corresponding to the image generation requirement information can be obtained, and the vector feature is matched with the description feature of each image in the image library to obtain the first reference image.

[0048] In step 104, a target image is generated based on the image generation requirement information and the first reference image using the target image generation manner.

[0049] The image generation requirement information and the first reference image are inputs of the target image generation manner. After obtaining the image generation requirement information and the first reference image, the image generation requirement information and the first reference image can be processed according to the target image generation manner, and then the target image is obtained.

[0050] It should be noted that in the case that the reference image is included in the image generation requirement information, the editing intention of the user for the reference image can be obtained based on the image generation requirement information; and based on the image generation requirement information, the first reference image and the editing intention, the target image is generated by using the target image generation manner.

[0051] In the embodiments of the present disclosure, image generation requirement information is obtained; a corresponding target image generation manner is determined according to the image generation requirement information; a first reference image is obtained by querying based on the image generation requirement information; and a target image is generated by using the target image generation manner based on the image generation requirement information and the first reference image. The present disclosure generates a target image based on the queried first reference image, that is, the image Retrieval-Augmented Generation (iRAG) technology is used, the timeliness of image generation is aligned with the timeliness of image search, and the problem of possible lag of the target image is avoided. In addition, since image search almost covers all public knowledge, there is no memory capacity bottleneck in the present disclosure, and the problem of limited memory capacity of the AI image generation model is also solved.

[0052] Figure 2 is a flowchart of an image generation method according to another embodiment of the present disclosure.

[0053] As shown in Figure 2 , the image generation method comprises:

[0054] Step 201, obtaining image generation requirement information.

[0055] Step 202, determining a corresponding target image generation manner according to the image generation requirement information.

[0056] Step 203, obtaining image subject information included in the image generation requirement information; and based on the image subject information, performing image query in a set image library to obtain a first reference image.

[0057] Among them, the image subject information included in the image generation requirement information can be subject type, subject name and the like.

[0058] For example, assuming that the image generation requirement information is "draw a red vehicle with a modern fashion design, with a futuristic atmosphere", the image subject information can be a red vehicle. It should be noted that if the image generation requirement information also includes the vehicle model, the image subject information can be the red vehicle and the vehicle model.

[0059] As an example, the image subject information in the image generation requirement information can be extracted based on a large model, and then the first reference image is obtained by image search based on the image subject information.

[0060] Based on the image subject information, the image query is performed, and reference images that match the image generation requirement can be accurately screened out from the image library, thereby improving the image generation effect.

[0061] The quality of the first reference image affects the image generation effect. If the quality of the first reference image is poor, the quality of the finally generated image can also be poor. Therefore, in order to ensure the image generation effect and further improve the quality of the generated image, as an example, based on the image subject information, the image query is performed in the set image library to obtain a candidate reference image; the image quality of the candidate reference image is obtained, and a set quality requirement corresponding to the image generation requirement information is obtained; and the candidate reference image is screened according to the image quality and the set quality requirement, and the first reference image is obtained.

[0062] The quality requirement corresponding to different image generation requirements is different. For example, if the image generation requirement is to generate a portrait image, the corresponding quality requirement can be that the face definition in the image is higher than a set definition threshold; and if the image generation requirement is to generate a poster image, the corresponding quality requirement can be that the image does not include a watermark.

[0063] As a possible implementation manner, the candidate reference image whose image quality meets the set quality requirement can be taken as the first reference image according to whether the image quality of the candidate reference image meets the set quality requirement.

[0064] As another possible implementation manner, the first reference image can be obtained by sorting the candidate reference images in descending order of image quality according to the image quality and the set quality requirement, and taking the first N candidate reference images as the first reference image; wherein N is a positive integer.

[0065] In step 204, the target image is generated by using a target image generation manner based on the image generation requirement information and the first reference image.

[0066] It should be noted that the explanations of steps 201, 202 and 204 can be referred to the related descriptions in any embodiment of the present disclosure, which will not be repeated here.

[0067] In the embodiments of the present disclosure, the image generation requirement information is obtained; a corresponding target image generation manner is determined according to the image generation requirement information; the image subject information included in the image generation requirement information is obtained; the first reference image is obtained by performing image query in a set image library based on the image subject information; and the target image is generated by using the target image generation manner based on the image generation requirement information and the first reference image. Based on the image subject information, the image query is performed, and reference images that match the image generation requirement can be accurately screened out from the image library, thereby improving the image generation effect.

[0068] Figure 3 is a flowchart of an image generation method provided according to another embodiment of the present disclosure.

[0069] As shown in Figure 3 , the image generation method comprises:

[0070] Step 301, obtaining image generation requirement information.

[0071] Step 302, obtaining a requirement text in the image generation requirement information.

[0072] The requirement text can refer to a prompt in the image generation requirement information.

[0073] Step 303, inputting the requirement text into a first large model to detect a target subject modification intention corresponding to the image generation requirement information, wherein the target subject modification intention is used to indicate whether the subject in the first reference image needs to be modified.

[0074] The first large model can determine whether the subject in the first reference image needs to be modified when generating an image based on the requirement text. If the target subject modification intention indicates that the subject in the first reference image needs to be modified, the image generation requirement information corresponds to a generalization requirement scenario under high recognition. If the target subject modification intention indicates that the subject in the first reference image does not need to be modified, the image generation requirement information corresponds to a pixel-level fidelity requirement scenario.

[0075] For example, assuming that the image generation requirement information is “generate an image of a vehicle of A type driving in a desert” or “generate an image of a vehicle of A type passing through a forest”, the target subject modification intention corresponding to the image generation requirement information is that the subject in the first reference image does not need to be modified. Assuming that the image generation requirement information is “generate an image of a person of B wearing a C style costume” or “generate an image of a person of B as a hand-made doll”, the target subject modification intention corresponding to the image generation requirement information is that the subject in the first reference image needs to be modified.

[0076] It should be noted that if the image generation requirement information includes a second reference image input by a user, the subject modification intention obtained by performing the subject modification intention detection can also be used to indicate whether the subject in the second reference image needs to be modified.

[0077] Step 304, obtaining a target image generation manner corresponding to the target subject modification intention based on a mapping relationship between the subject modification intention and the image generation manner.

[0078] Corresponding image generation manners are different for different subject modification intentions. As an example, a mapping relationship between the subject modification intention and the image generation manner can be preset. After the target subject modification intention is determined, the corresponding target image generation manner can be determined based on the preset mapping relationship.

[0079] Different subject modification intentions require different focuses when generating images. Therefore, selecting an image generation manner that matches the subject modification intention can significantly improve the effect and efficiency of image generation.

[0080] In step 305, the first reference image is obtained based on the image generation requirement information.

[0081] In step 306, the target image is generated based on the image generation requirement information and the first reference image using the target image generation manner.

[0082] To extract the key features in the requirement text, the requirement text can be text encoded to obtain text features.

[0083] The image generation process is described below in combination with the image generation manners corresponding to two subject modification intentions.

[0084] As an example, in response to the target subject modification intention indicating that the subject in the first reference image does not need to be modified, the first reference image is feature extracted to obtain first image features, and a subject segmentation image in the first reference image is obtained. The first image features and the text features are input into a first image generation model to obtain a background image and subject layout information. The subject segmentation image and the background image are fused according to the subject layout information to obtain the target image.

[0085] The image generation manner corresponding to this example can be referred to as an accurate image generation manner. The first image features can refer to vector features obtained by using a small-scale feature extractor. The subject segmentation image can refer to an image of the subject part obtained by performing semantic segmentation on the first reference image. The background image can refer to an image of the background part other than the subject part. The subject layout information is used to indicate the layout of the subject in the background image. For example, the subject layout information includes the position and size of the subject in the background image.

[0086] The disclosure obtains the content in the first reference image, i.e., the subject segmentation image, from the pixel level. The subject segmentation image not only preserves the key details and features of the subject in the first reference image, achieving strong fidelity of the subject, but also ensures the integrity of the subject in the target image. In addition, image fusion based on the subject layout information can make the subject naturally blend into the background image and maintain harmony with other parts in the background image.

[0087] As another example, in response to the target subject modifying the intention indicating that the subject in the first reference image needs to be modified, feature extraction is performed on the first reference image to obtain second image features; the text features and the second image features are interleaved and spliced based on the description objects corresponding to each sub-feature in the text features and the second image features to obtain interleaved text-image features; and the interleaved text-image features are input into the second image generation model to obtain the target image.

[0088] In this example, the corresponding image generation manner can be referred to as a high generalization image generation manner; the second image features are vector features, and the second image features and the first image features can be different; the text features include text sub-features, and the image features include image sub-features, and the text sub-features and the image sub-features correspond to description objects, respectively; the interleaved splicing refers to combining the image features and the text features in a certain specific manner (for example, the text sub-features and the image sub-features corresponding to the same description object are spliced together) to form a new encoding feature, that is, the interleaved text-image features, please refer to Figure 4 , Figure 4 the schematic diagram of the interleaved text-image features; this manner combines the image features and the text features, which are two different modalities of information, in a specific manner, realizing the correlation between the multi-modal features.

[0089] For example, assuming that the image generation requirement information is “generate a figurine image of the B character and the D character”, the corresponding description objects include the B character and the D character, the B character corresponds to text sub-features and image sub-features, and the D character also corresponds to text sub-features and image sub-features; when interleaved splicing is performed, the image sub-features corresponding to the B character can be inserted after the text sub-features corresponding to the B character in the text features, and the image sub-features corresponding to the D character can be inserted after the text sub-features corresponding to the D character in the text features.

[0090] The interleaved splicing of the text features and the second image features realizes the accurate alignment of the two in the description objects, and this interleaved splicing manner not only retains the key information in the respective features, but also promotes the deep fusion between the text features and the image features; when the target image is generated based on the interleaved text-image features, not only the characteristics of the subject in the first reference image can be fully considered, but also the fidelity of the subject can be considered while performing the generalization secondary creation, and the image generation effect is good.

[0091] The image generation requirement information can include a second reference image input by the user, in which case, the interleaved text-image features are obtained in the following manner: feature extraction is performed on the second reference image to obtain third image features; the text features, the second image features, and the third image features are interleaved and spliced based on the description objects corresponding to each sub-feature in the text features, the second image features, and the third image features to obtain the interleaved text-image features.

[0092] For example, assuming that the image generation requirement information is a second reference image including a B character and "generate a figurine photograph image of the B character and a D character", when generating the target image, the text sub-feature corresponding to the B character and the image sub-feature of the B character in the third image feature need to be interleaved and spliced, and the text sub-feature corresponding to the D character and the image sub-feature of the D character in the second image feature need to be interleaved and spliced.

[0093] In the case where the image generation requirement information includes the second reference image input by the user, the interleaved splicing of features is performed based on the requirement text, the first reference image, and the second reference image input by the user, and then the target image is generated, which not only meets the personalized image generation requirement of the user, realizes flexible personalized image generation, but also ensures the image generation effect.

[0094] In addition, for non-public knowledge, the disclosure can avoid the problem of lack of memory of the image generation model for these contents through the way of user self-provided reference image.

[0095] It should be noted that the explanations of steps 301, 302, and 305 can be referred to the related descriptions in any embodiment of the disclosure, which will not be repeated here.

[0096] In the embodiment of the disclosure, the image generation requirement information is obtained, the requirement text in the image generation requirement information is obtained, the requirement text is input into the first large model for subject modification intention detection, the target subject modification intention corresponding to the image generation requirement information is obtained, wherein the target subject modification intention is used to indicate whether the subject in the first reference image needs to be modified, the target image generation mode corresponding to the target subject modification intention is obtained based on the mapping relationship between the subject modification intention and the image generation mode, the first reference image is obtained based on the image generation requirement information, and the target image is generated based on the image generation requirement information and the first reference image by using the target image generation mode. The focus of attention is different when generating images according to different subject modification intentions, therefore, the image generation effect and efficiency can be significantly improved by selecting the image generation mode matched with the subject modification intention.

[0097] Figure 5 is a flowchart of an image generation method according to another embodiment of the disclosure.

[0098] As shown in Figure 5 , the image generation method includes:

[0099] Step 501, obtaining image generation requirement information.

[0100] Step 502, determining the corresponding target image generation mode according to the image generation requirement information.

[0101] At step 503, the demand information is generated based on the image, and a first reference image is queried.

[0102] At step 504, the demand text in the image generation demand information is obtained, the demand text is rewritten and expanded to obtain target demand text, and the target image is generated based on the target demand text and the first reference image by using a target image generation method.

[0103] Compared with the demand text, the target demand text has a more regular format and more comprehensive detail description. As an example, the demand text can be input into a large model, and the large model can rewrite and expand the demand text based on core picture content, picture style description, picture main body and limitation description, picture detail description, picture background modification description, special effect, composition, color tone, clarity description, quality description, and other dimensions.

[0104] The demand text in the image generation demand information may have problems such as abstraction and ambiguous expression, and direct image generation based on the demand text often cannot obtain the best image generation result. Therefore, the demand text is rewritten and expanded in the present disclosure to improve the image generation effect.

[0105] As another example, the context information of the image generation demand information is obtained, and in the case that the image generation demand information includes a second reference image input by the user, the description information of the second reference image is obtained, and at least one of the context information and the description information and the demand text are input into a second large model to rewrite and expand to obtain the target demand text.

[0106] The second large model and the first large model can be the same large model or different large models.

[0107] In the process of image generation, the image generation demand information provided by the user may not be complete, which may cause the generated image to deviate from the user's expectation. In order to more accurately understand the user's demand and improve the accuracy and effect of image generation, the present disclosure can further obtain context information related to the image generation demand information, and in the case that the user inputs a second reference image, obtain description information of the second reference image. Combined with these information, the present disclosure can more comprehensively understand the real image generation intention of the user, and then improve the image generation effect and improve the user experience.

[0108] It should be noted that the explanation and description of steps 501-503 can be referred to the related description in any embodiment of the present disclosure, which will not be repeated here.

[0109] In the embodiments of the present disclosure, the demand text in the image generation demand information is obtained; the demand text is rewritten and expanded to obtain a target demand text; and a target image is generated based on the target demand text and a first reference image by using a target image generation manner. The demand text in the image generation demand information may have problems such as abstraction and ambiguous expression, and direct image generation based on the demand text often cannot obtain the best image generation result. Therefore, the demand text is rewritten and expanded in the present disclosure, which can improve the image generation effect.

[0110] Figure 6 FIG. 1 is a flowchart of an image generation method according to an embodiment of the present disclosure.

[0111] As shown in FIG. 1, the image generation method comprises the following steps. Figure 6

[0112] In step 601, image generation demand information is obtained.

[0113] In step 602, a corresponding target image generation manner is determined according to the image generation demand information.

[0114] In step 603, it is determined whether the image generation demand information needs to be queried for a reference image.

[0115] Not all image generation demands need to be enhanced by retrieval to improve the effect when image generation is performed. For example, if the image generation demand information is “draw a small rabbit”, this is a general text-to-image model that can be processed by itself and does not need to trigger retrieval enhancement. Therefore, in order to avoid unnecessary complex calculation and resource waste, the present disclosure determines whether the reference image needs to be queried before image generation.

[0116] As an example, the image generation demand information can be input to a third large model to determine whether the image generation demand information needs to be queried for a reference image. The third large model, the second large model and the first large model can be the same large model or different large models.

[0117] In step 604, if the image generation demand information needs to be queried for a reference image, a first reference image is obtained based on the image generation demand information.

[0118] In step 605, a target image is generated based on the image generation demand information and the first reference image by using a target image generation manner.

[0119] It should be noted that the explanations of steps 601, 602, 604 and 605 can be referred to the related descriptions in any embodiment of the present disclosure, which will not be repeated here.

[0120] ​In the embodiments of the present disclosure, image generation requirement information is obtained, a corresponding target image generation manner is determined according to the image generation requirement information, it is judged whether the image generation requirement information needs to be queried for a reference image, in the case that the image generation requirement information needs to be queried for a reference image, a first reference image is obtained based on the image generation requirement information, and the target image generation manner is used to generate a target image based on the image generation requirement information and the first reference image. The present disclosure judges whether the image generation requirement information needs to be queried for a reference image, and in the case that the image generation requirement information needs to be queried for a reference image, the iRAG technology is used for image generation, which not only aligns the timeliness of image generation with the timeliness of image search, avoids the problem that the target image may exist lag, but also avoids unnecessary complex calculation and resource waste when image retrieval is not needed, thereby improving the overall image generation efficiency. In addition, since image search almost covers all public knowledge, the present disclosure has no memory capacity bottleneck, and also solves the problem of limited memory capacity of AI image generation model.

[0121] Figure 7 is a flowchart of an image generation method according to another embodiment of the present disclosure.

[0122] As shown in Figure 7 , the image generation method comprises:

[0123] Step 701, obtaining image generation requirement information.

[0124] Step 702, determining a corresponding target image generation manner according to the image generation requirement information.

[0125] In addition to the precise image generation manner and the high generalization image generation manner, the present disclosure also provides a general text image generation manner and an image image generation manner for bottom-up. For the image generation requirement information, a matching target image generation manner can be selected from the foregoing image generation manners.

[0126] For example, assuming that the image generation requirement information is "draw a small rabbit", the corresponding target image generation manner can be the general text image generation manner; assuming that the image generation requirement information is "draw a Flemish giant rabbit", the corresponding target image generation manner can be the precise image generation manner.

[0127] As an example, the requirement text in the image generation requirement information is obtained, the requirement text is input into a first large model for subject modification intention detection to obtain a target subject modification intention corresponding to the image generation requirement information, wherein the target subject modification intention is used to indicate whether the subject in the first reference image needs to be modified, and a target image generation manner corresponding to the target subject modification intention is obtained based on a mapping relationship between the subject modification intention and the image generation manner.

[0128] Step 703, obtaining the requirement text in the image generation requirement information.

[0129] Step 704, understanding the requirement text to obtain the requirement type corresponding to the image generation requirement information.

[0130] The requirement type can be used to indicate the knowledge type involved in the image generation requirement information. For example, the requirement type includes simple knowledge, professional knowledge, long-tail knowledge, and time-sensitive knowledge.

[0131] As an example, the requirement text can be input into the fourth large model for requirement understanding to obtain the requirement type corresponding to the image generation requirement information. The fourth large model, the third large model, the second large model, and the first large model can be the same large model or different large models.

[0132] Step 705, determining whether the image generation requirement information needs to be queried based on the requirement type.

[0133] The requirement type and whether the reference image needs to be queried have a corresponding relationship. For example, simple knowledge does not need to be queried, that is, no first reference image needs to be obtained. Professional knowledge, long-tail knowledge, and time-sensitive knowledge need to be queried, that is, a first reference image needs to be obtained.

[0134] In processing the image generation requirement, it is considered that not all requirements need to be enhanced by retrieval to improve the effect. Therefore, the present disclosure adopts an intelligent strategy, that is, whether to query the reference image is determined flexibly according to the requirement type. By accurately judging the requirement, the present disclosure can implement retrieval enhancement targetedly, not only avoiding excessive retrieval enhancement load, but also meeting the user's demand while optimizing the allocation of resources and avoiding unnecessary resource waste.

[0135] Step 706, in the case that the image generation requirement information needs to be queried, obtaining the first reference image based on the image generation requirement information.

[0136] As an example, the image subject information included in the image generation requirement information is obtained; based on the image subject information, image query is performed in the set image library to obtain the first reference image.

[0137] As another example, the image subject information included in the image generation requirement information is obtained; based on the image subject information, image query is performed in the set image library to obtain the candidate reference image; the image quality of the candidate reference image is obtained, and the set quality requirement corresponding to the image generation requirement information is obtained; the candidate reference image is filtered according to the image quality and the set quality requirement to obtain the first reference image.

[0138] At step 707, the target image is generated based on the image generation requirement information and the first reference image, by using a target image generation manner.

[0139] As an example, the requirement text corresponds to a text feature, and in response to the target subject modification intention indicating that the subject in the first reference image does not need to be modified, feature extraction is performed on the first reference image to obtain a first image feature, and a subject segmentation image in the first reference image is obtained; the first image feature and the text feature are input into a first image generation model to obtain a background image and subject layout information; and the subject segmentation image and the background image are fused according to the subject layout information to obtain the target image.

[0140] As another example, the requirement text corresponds to a text feature, and in response to the target subject modification intention indicating that the subject in the first reference image needs to be modified, feature extraction is performed on the first reference image to obtain a second image feature; the text feature and the second image feature are interleaved and spliced based on the description objects corresponding to each sub-feature in the text feature and the second image feature to obtain an interleaved text-image feature; and the interleaved text-image feature is input into a second image generation model to obtain the target image.

[0141] As a possible implementation manner of the embodiments of the present disclosure, the text feature and the second image feature are interleaved and spliced based on the description objects corresponding to each sub-feature in the text feature and the second image feature to obtain an interleaved text-image feature, which includes that the image generation requirement information includes a second reference image input by a user, feature extraction is performed on the second reference image to obtain a third image feature; and the text feature, the second image feature and the third image feature are interleaved and spliced based on the description objects corresponding to each sub-feature in the text feature, the second image feature and the third image feature to obtain the interleaved text-image feature.

[0142] As an example, the requirement text in the image generation requirement information is obtained; the requirement text is rewritten and expanded to obtain a target requirement text; and the target image is generated based on the target requirement text and the first reference image, by using a target image generation manner.

[0143] As an example, the context information of the image generation requirement information is obtained; in the case that the image generation requirement information includes a second reference image input by a user, description information of the second reference image is obtained; and at least one of the context information and the description information, and the requirement text are input into a second large model for rewriting and expansion to obtain a target requirement text.

[0144] At step 708, in the case that the image generation requirement information does not need to be queried for a reference image, the target image is generated based on the image generation requirement information, by using a target image generation manner.

[0145] The disclosure can also generate demand information based on images without reference image query, efficiently and accurately complete image generation tasks, meet diversified image generation demands, and further expand the application scenarios of the disclosure.

[0146] As an example, if the detected subject modification intention indicates that the subject in the second reference image does not need to be modified, the target image generation manner can be an accurate image generation manner. Specifically, feature extraction is performed on the second reference image to obtain fourth image features, and a subject segmentation image in the second reference image is obtained. The fourth image features and the text features are input into a first image generation model to obtain a background image and subject layout information. The subject segmentation image and the background image are fused according to the subject layout information to obtain the target image.

[0147] As another example, if the detected subject modification intention indicates that the subject in the second reference image needs to be modified, the target image generation manner can be a high generalization image generation manner. Specifically, feature extraction is performed on the second reference image to obtain fifth image features. The text features and the fifth image features are staggered and spliced based on the description objects corresponding to each sub-feature in the text features and the fifth image features to obtain staggered text-image features. The staggered text-image features are input into a second image generation model to obtain the target image.

[0148] It should be noted that the explanations of steps 701, 702, 703, 706, and 707 can be referred to the related descriptions in any embodiment of the disclosure, which will not be repeated here.

[0149] In the embodiments of the disclosure, the demand text in the image generation demand information is obtained. The demand understanding is performed on the demand text to obtain the demand type corresponding to the image generation demand information. Based on the demand type, it is determined whether the image generation demand information needs to be queried for a reference image. In the case that the image generation demand information does not need to be queried for a reference image, the target image generation manner is adopted based on the image generation demand information to generate the target image. In the case that the image generation demand information needs to be queried for a reference image, the first reference image is obtained based on the image generation demand information. The target image generation manner is adopted based on the image generation demand information and the first reference image to generate the target image. Considering that not all demands need to be enhanced by retrieval, the disclosure flexibly determines whether to obtain the first reference image according to the demand type. By accurately judging the demand, the disclosure can implement retrieval enhancement pertinently, not only avoiding excessive load of the retrieval enhancement system, but also optimizing the resource allocation while meeting the user demand, thereby avoiding unnecessary resource waste.

[0150] Figure 8 is a structural schematic diagram of an image generation device according to an embodiment of the disclosure.

[0151] As shown in Figure 8 The image generation apparatus includes an acquisition module 801, a determination module 802, a query module 803, and a generation module 804, where:

[0152] The acquisition module 801 is configured to acquire image generation requirement information. The determination module 802 is configured to determine a corresponding target image generation manner according to the image generation requirement information. The query module 803 is configured to query a first reference image based on the image generation requirement information. The generation module 804 is configured to generate a target image based on the image generation requirement information and the first reference image, and using the target image generation manner.

[0153] In a possible implementation manner of the present disclosure, the query module 803 is configured to: acquire image subject information included in the image generation requirement information; and perform image query in a set image library based on the image subject information to obtain the first reference image.

[0154] In a possible implementation manner of the present disclosure, the query module 803 is configured to: perform image query in a set image library based on the image subject information to obtain a candidate reference image; acquire image quality of the candidate reference image, and acquire a set quality requirement corresponding to the image generation requirement information; and perform screening on the candidate reference image according to the image quality and the set quality requirement to obtain the first reference image.

[0155] In a possible implementation manner of the present disclosure, the determination module 802 is configured to: acquire a requirement text in the image generation requirement information; input the requirement text into a first large model to perform subject modification intention detection, and acquire a target subject modification intention corresponding to the image generation requirement information, where the target subject modification intention is used to indicate whether a subject in the first reference image needs to be modified; and acquire a target image generation manner corresponding to the target subject modification intention based on a mapping relationship between the subject modification intention and the image generation manner.

[0156] In a possible implementation manner of the present disclosure, the requirement text corresponds to a text feature, and the generation module 804 is configured to: in response to the target subject modification intention indicating that the subject in the first reference image does not need to be modified, perform feature extraction on the first reference image to obtain a first image feature, and acquire a subject segmentation image in the first reference image; input the first image feature and the text feature into a first image generation model to acquire a background image and subject layout information; and perform fusion on the subject segmentation image and the background image according to the subject layout information to obtain the target image.

[0157] As a possible implementation manner of the embodiment of the present disclosure, the demand text corresponds to a text feature, and the generation module 804 is configured to: in response to the target subject modification intention indicating that the subject in the first reference image needs to be modified, performing feature extraction on the first reference image to obtain second image features; based on the description object corresponding to each sub-feature in the text features and the second image features, interleaving and splicing the text features and the second image features to obtain image-text interleaving features; and inputting the image-text interleaving features into the second image generation model to obtain the target image.

[0158] As a possible implementation manner of the embodiment of the present disclosure, the image generation demand information includes a second reference image input by the user, and the generation module 804 is configured to: performing feature extraction on the second reference image to obtain third image features; based on the description object corresponding to each sub-feature in the text features, the second image features and the third image features, interleaving and splicing the text features, the second image features and the third image features to obtain image-text interleaving features.

[0159] As a possible implementation manner of the embodiment of the present disclosure, the generation module 804 is configured to: obtaining the demand text in the image generation demand information; rewriting and expanding the demand text to obtain a target demand text; and based on the target demand text and the first reference image, generating the target image by using the target image generation manner.

[0160] As a possible implementation manner of the embodiment of the present disclosure, the generation module 804 is configured to: obtaining the context information of the image generation demand information; in the case that the image generation demand information includes a second reference image input by the user, obtaining description information of the second reference image; and inputting at least one of the context information and the description information and the demand text into the second large model to rewrite and expand the demand text to obtain a target demand text.

[0161] Figure 9 FIG. 1 is a structural schematic diagram of an image generation apparatus according to an embodiment of the present disclosure.

[0162] As shown in Figure 9 FIG. 1, the image generation apparatus includes an obtaining module 901, a determining module 902, a judging module 903, a querying module 904 and a generation module 905, wherein:

[0163] The obtaining module 901 is configured to obtain image generation demand information; the determining module 902 is configured to determine a corresponding target image generation manner according to the image generation demand information; the judging module 903 is configured to judge whether the image generation demand information needs to be queried for a reference image; the querying module 904 is configured to, in the case that the image generation demand information needs to be queried for a reference image, query a first reference image based on the image generation demand information; and the generation module 905 is configured to generate a target image by using the target image generation manner based on the image generation demand information and the first reference image.

[0164] As a possible implementation manner of the embodiment of the present disclosure, the determining module 903 is configured to: obtain the requirement text in the image generation requirement information; perform requirement understanding on the requirement text to obtain the requirement type corresponding to the image generation requirement information; and determine whether the image generation requirement information needs to be queried for a reference image based on the requirement type.

[0165] As a possible implementation manner of the embodiment of the present disclosure, the generating module 905 is further configured to: in the case that the image generation requirement information does not need to be queried for a reference image, generate the target image based on the image generation requirement information by using a target image generation manner.

[0166] As a possible implementation manner of the embodiment of the present disclosure, the querying module 904 is configured to: obtain the image subject information included in the image generation requirement information; and perform image query in a set image library based on the image subject information to obtain the first reference image.

[0167] As a possible implementation manner of the embodiment of the present disclosure, the querying module 904 is configured to: perform image query in a set image library based on the image subject information to obtain a candidate reference image; obtain the image quality of the candidate reference image, and obtain a set quality requirement corresponding to the image generation requirement information; and filter the candidate reference image according to the image quality and the set quality requirement to obtain the first reference image.

[0168] As a possible implementation manner of the embodiment of the present disclosure, the determining module 902 is configured to: obtain the requirement text in the image generation requirement information; input the requirement text into a first large model to detect a subject modification intention, and obtain a target subject modification intention corresponding to the image generation requirement information, wherein the target subject modification intention is used to indicate whether the subject in the first reference image needs to be modified; and obtain a target image generation manner corresponding to the target subject modification intention based on a mapping relationship between the subject modification intention and the image generation manner.

[0169] As a possible implementation manner of the embodiment of the present disclosure, the requirement text corresponds to a text feature, and the generating module 905 is configured to: in response to the target subject modification intention indicating that the subject in the first reference image does not need to be modified, perform feature extraction on the first reference image to obtain a first image feature, and obtain a subject segmentation image in the first reference image; input the first image feature and the text feature into a first image generation model to obtain a background image and subject layout information; and fuse the subject segmentation image and the background image according to the subject layout information to obtain the target image.

[0170] As a possible implementation manner of the embodiment of the present disclosure, the demand text corresponds to a text feature, and the generation module 905 is configured to: in response to the target subject modifying the intention indicating that the subject in the first reference image needs to be modified, performing feature extraction on the first reference image to obtain second image features; based on the description object corresponding to each sub-feature in the text features and the second image features, interleaving and splicing the text features and the second image features to obtain image-text interleaving features; and inputting the image-text interleaving features into the second image generation model to obtain the target image.

[0171] As a possible implementation manner of the embodiment of the present disclosure, the image generation demand information includes a second reference image input by the user, and the generation module 905 is configured to: performing feature extraction on the second reference image to obtain third image features; based on the description object corresponding to each sub-feature in the text features, the second image features and the third image features, interleaving and splicing the text features, the second image features and the third image features to obtain image-text interleaving features.

[0172] As a possible implementation manner of the embodiment of the present disclosure, the generation module 905 is configured to: obtain the demand text in the image generation demand information; rewrite and expand the demand text to obtain a target demand text; and based on the target demand text and the first reference image, generate the target image by using a target image generation manner.

[0173] As a possible implementation manner of the embodiment of the present disclosure, the generation module 905 is configured to: obtain the context information of the image generation demand information; in the case that the image generation demand information includes a second reference image input by the user, obtain description information of the second reference image; and input at least one of the context information and the description information and the demand text into the second large model to rewrite and expand to obtain a target demand text.

[0174] It should be noted that the foregoing explanation and description of the image generation method also apply to the image generation apparatus of the present embodiment, which will not be described here again.

[0175] Figure 10 is a schematic diagram of an intelligent agent according to an embodiment of the present disclosure.

[0176] As Figure 10 shown, the intelligent agent 1000 can include an input module 1001, a processing module 1002 and an output module 1003.

[0177] As an example, the input module 1001 is configured to acquire image generation requirement information; the processing module 1002 is configured to determine a target image generation manner according to the image generation requirement information; based on the image generation requirement information, a first reference image is queried; based on the image generation requirement information and the first reference image, the target image generation manner is used to generate a target image; and the output module 1003 is configured to output the target image.

[0178] As another example, the input module 1001 is configured to acquire image generation requirement information; the processing module 1002 is configured to determine a target image generation manner according to the image generation requirement information; it is determined whether the image generation requirement information needs to be queried for a reference image; in the case that the image generation requirement information needs to be queried for a reference image, based on the image generation requirement information, a first reference image is queried; based on the image generation requirement information and the first reference image, the target image generation manner is used to generate a target image; and the output module 1003 is configured to output the target image.

[0179] According to an embodiment of the present disclosure, the present disclosure further provides an agent system, a readable storage medium and a computer program product.

[0180] Figure 11 A schematic block diagram of an example agent system 1100 that can be used to implement embodiments of the present disclosure is shown. The agent system 1100 is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The agent system can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0181] As shown in Figure 11 The agent system 1100 includes a computing unit 1101 that can perform various suitable actions and processes in accordance with computer programs stored in a ROM (Read-Only Memory) 1102 or loaded into a RAM (Random Access Memory) 1103 from a storage unit 1108. Various programs and data required for the operation of the agent system 1100 can also be stored in the RAM 1103. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An I / O (Input / Output) interface 1105 is also connected to the bus 1104.

[0182] A plurality of components in the agent system 1100 are connected to the I / O interface 1105, including: an input unit 1106, such as a keyboard, a mouse, etc.; an output unit 1107, such as various types of displays, speakers, etc.; a storage unit 1108, such as a magnetic disk, an optical disk, etc.; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1109 allows the agent system 1100 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0183] The computing unit 1101 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, a DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 1101 performs various methods and processes described above, such as the image generation method. For example, in some embodiments, the image generation method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on the agent system 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of the image generation method described above can be performed. Alternatively, in other embodiments, the computing unit 1101 can be configured to perform the image generation method by any other appropriate means, such as by means of firmware.

[0184] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a Field Programmable Gate Array (FPGA), an Application-Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a System on a Chip (SOC), a Complex Programmable Logic Device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0185] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general or special purpose computer, such that the program code, when executed by the processor or controller, causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can execute entirely on a machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

[0186] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include a linearly-programmed electronic storage, a portable computer diskette, a hard disk, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory), or flash memory, an optical fiber, a CD-ROM (Compact Disc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0187] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0188] The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, and a blockchain network.

[0189] The computer system can include clients and servers. This relationship can be between a client and a server that are typically remote from each other and typically interact through a communication network. The relationship between client and server exists by virtue of computer programs running on the respective computer systems and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS (Virtual Private Server, or VPS for short) services. The server can also be a server of a distributed system, or a server combined with a blockchain.

[0190] It should be noted that artificial intelligence is a discipline that studies enabling computers to simulate some thinking processes and intelligent behaviors of people (such as learning, reasoning, thinking, planning, etc.), both hardware and software technologies. Artificial intelligence hardware technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing, etc.; artificial intelligence software technology mainly includes computer vision technology, speech recognition technology, natural language processing technology, and machine learning / deep learning, big data processing technology, knowledge graph technology, etc. several major directions.

[0191] It should be understood that the various forms of the flow shown above can be used to reorder, add or delete steps. For example, the steps described in the present disclosure can be executed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions of the present disclosure can be achieved, which is not limited herein.

[0192] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image generation method, comprising: Obtain image generation requirement information; Based on the image generation requirement information, a first reference image is obtained by querying; Obtain the requirement text from the image generation requirement information; The requirement text is input into the first model to detect the subject modification intent, and the target subject modification intent corresponding to the image generation requirement information is obtained. The target subject modification intent is used to indicate whether the subject in the first reference image needs to be modified. Based on the mapping relationship between the subject modification intention and the image generation method, the target image generation method corresponding to the target subject modification intention is obtained; wherein, if the target subject modification intention indicates that the subject in the first reference image does not need to be modified, the target image generation method is an accurate image generation method; if the target subject modification intention indicates that the subject in the first reference image needs to be modified, the target image generation method is a high-generalization image generation method. Based on the image generation requirement information and the first reference image, the target image is generated using the target image generation method. The requirement text corresponds to text features. Based on the image, requirement information and the first reference image are generated. The target image is generated using the high-generalization image generation method, including: Feature extraction is performed on the first reference image to obtain the second image features; Based on the description objects corresponding to each sub-feature in the text features and the second image features, the text features and the second image features are interleaved to obtain image-text interleaved features. The image-text interleaving features are input into the second image model to obtain the target image.

2. The method according to claim 1, wherein, The step of generating the requirement information based on the image and querying to obtain the first reference image includes: Obtain the image subject information included in the image generation requirement information; Based on the image subject information, an image query is performed in the set image library to obtain the first reference image.

3. The method according to claim 2, wherein, The step of querying an image in a designated image library based on the image subject information to obtain the first reference image includes: Based on the image subject information, an image query is performed in the set image library to obtain candidate reference images; Obtain the image quality of the candidate reference image, and obtain the set quality requirements corresponding to the image generation requirement information; Based on the image quality and the set quality requirements, the candidate reference images are filtered to obtain the first reference image.

4. The method according to claim 1, wherein, The requirement text corresponds to text features. The step of generating requirement information based on the image and the first reference image, and generating the target image using the precise image generation method, includes: Feature extraction is performed on the first reference image to obtain the first image features, and the main body segmentation image in the first reference image is obtained; Input the first image features and the text features into the first raw image model to obtain the background image and subject layout information; Based on the main body layout information, the main body segmentation image and the background image are fused to obtain the target image.

5. The method according to claim 1, wherein, The image generation requirement information includes a second reference image input by the user. The text features and the second image features are interleaved and spliced ​​together based on the description objects corresponding to each sub-feature in the text features and the second image features to obtain interleaved image features, including: Feature extraction is performed on the second reference image to obtain the features of the third image; Based on the description objects corresponding to each sub-feature in the text feature, the second image feature, and the third image feature, the text feature, the second image feature, and the third image feature are interleaved to obtain the image-text interleaved feature.

6. The method according to claim 1, wherein, The step of generating a target image based on the image generation requirement information and the first reference image, using the target image generation method, includes: Obtain the requirement text from the image generation requirement information; The requirement text is rewritten and expanded to obtain the target requirement text; Based on the target requirement text and the first reference image, the target image is generated using the target image generation method.

7. The method according to claim 6, wherein, The process of rewriting and expanding the requirement text to obtain the target requirement text includes: Obtain the context information of the image generation requirement information; If the image generation requirement information includes a second reference image input by the user, the description information of the second reference image is obtained; At least one of the context information and the description information, along with the requirement text, are input into the second model for rewriting and expansion to obtain the target requirement text.

8. An image generation method, comprising: Obtain image generation requirement information; Determine whether the image generation requirement information needs to be queried for a reference image; If the image generation requirement information necessitates a reference image query, a first reference image is obtained based on the image generation requirement information. Obtain the requirement text from the image generation requirement information; The requirement text is input into the first model to detect the subject modification intent, and the target subject modification intent corresponding to the image generation requirement information is obtained. The target subject modification intent is used to indicate whether the subject in the first reference image needs to be modified. Based on the mapping relationship between the subject modification intention and the image generation method, the target image generation method corresponding to the target subject modification intention is obtained; wherein, if the target subject modification intention indicates that the subject in the first reference image does not need to be modified, the target image generation method is an accurate image generation method; if the target subject modification intention indicates that the subject in the first reference image needs to be modified, the target image generation method is a high-generalization image generation method. Based on the image generation requirement information and the first reference image, the target image is generated using the target image generation method. The requirement text corresponds to text features. Based on the image, requirement information and the first reference image are generated. The target image is generated using the high-generalization image generation method, including: Feature extraction is performed on the first reference image to obtain the second image features; Based on the description objects corresponding to each sub-feature in the text features and the second image features, the text features and the second image features are interleaved to obtain image-text interleaved features. The image-text interleaving features are input into the second image model to obtain the target image.

9. The method according to claim 8, wherein, The step of determining whether the image generation requirement information needs to be queried for a reference image includes: Obtain the requirement text from the image generation requirement information; The requirement text is analyzed to understand the requirement and obtain the requirement type corresponding to the image generation requirement information; Based on the aforementioned requirement type, determine whether the image generation requirement information needs to be queried using a reference image.

10. The method according to claim 8, wherein, The method further includes: If the image generation requirement information does not require a reference image query, the target image is generated based on the image generation requirement information and using the target image generation method.

11. The method according to claim 8, wherein, The step of generating the requirement information based on the image and querying to obtain the first reference image includes: Obtain the image subject information included in the image generation requirement information; Based on the image subject information, an image query is performed in the set image library to obtain the first reference image.

12. The method according to claim 11, wherein, The step of querying an image in a designated image library based on the image subject information to obtain the first reference image includes: Based on the image subject information, an image query is performed in the set image library to obtain candidate reference images; Obtain the image quality of the candidate reference image, and obtain the set quality requirements corresponding to the image generation requirement information; Based on the image quality and the set quality requirements, the candidate reference images are filtered to obtain the first reference image.

13. An image generation apparatus, comprising: The acquisition module is used to acquire image generation requirement information; The query module is used to generate demand information based on the image and query to obtain the first reference image; The determination module is used to obtain the requirement text from the image generation requirement information; The requirement text is input into the first model for subject modification intent detection to obtain the target subject modification intent corresponding to the image generation requirement information. The target subject modification intent indicates whether the subject in the first reference image needs to be modified. Based on the mapping relationship between the subject modification intent and the image generation method, the target image generation method corresponding to the target subject modification intent is obtained. If the target subject modification intent indicates that the subject in the first reference image does not need to be modified, the target image generation method is an accurate image generation method; if the target subject modification intent indicates that the subject in the first reference image needs to be modified, the target image generation method is a high-generalization image generation method. The generation module is used to generate a target image based on the image generation requirement information and the first reference image, using the target image generation method. The required text corresponds to text features, the target image is generated using a high-generalization image generation method, and the generation module is used for: Feature extraction is performed on the first reference image to obtain the second image features; Based on the description objects corresponding to each sub-feature in the text features and the second image features, the text features and the second image features are interleaved to obtain image-text interleaved features. The image-text interleaving features are input into the second image model to obtain the target image.

14. The apparatus according to claim 13, wherein, The query module is used for: Obtain the image subject information included in the image generation requirement information; Based on the image subject information, an image query is performed in the set image library to obtain the first reference image.

15. The apparatus according to claim 14, wherein, The query module is used for: Based on the image subject information, an image query is performed in the set image library to obtain candidate reference images; Obtain the image quality of the candidate reference image, and obtain the set quality requirements corresponding to the image generation requirement information; Based on the image quality and the set quality requirements, the candidate reference images are filtered to obtain the first reference image.

16. The apparatus according to claim 13, wherein, The required text corresponds to text features, the target image is generated using a precise image generation method, and the generation module is used for: Feature extraction is performed on the first reference image to obtain the first image features, and the main body segmentation image in the first reference image is obtained; Input the first image features and the text features into the first raw image model to obtain the background image and subject layout information; Based on the main body layout information, the main body segmentation image and the background image are fused to obtain the target image.

17. The apparatus according to claim 13, wherein, The image generation requirement information includes a second reference image input by the user, and the generation module is used for: Feature extraction is performed on the second reference image to obtain the features of the third image; Based on the description objects corresponding to each sub-feature in the text feature, the second image feature, and the third image feature, the text feature, the second image feature, and the third image feature are interleaved to obtain the image-text interleaved feature.

18. The apparatus according to claim 13, wherein, The generation module is used for: Obtain the requirement text from the image generation requirement information; The requirement text is rewritten and expanded to obtain the target requirement text; Based on the target requirement text and the first reference image, the target image is generated using the target image generation method.

19. The apparatus according to claim 18, wherein, The generation module is used for: Obtain the context information of the image generation requirement information; If the image generation requirement information includes a second reference image input by the user, the description information of the second reference image is obtained; At least one of the context information and the description information, along with the requirement text, are input into the second model for rewriting and expansion to obtain the target requirement text.

20. An image generation apparatus, comprising: The acquisition module is used to acquire image generation requirement information; The judgment module is used to determine whether the image generation requirement information needs to be queried for a reference image; The query module is used to retrieve a first reference image based on the image generation requirement information when the image generation requirement information necessitates a reference image query. The determination module is used to obtain the requirement text from the image generation requirement information; The requirement text is input into the first model for subject modification intent detection to obtain the target subject modification intent corresponding to the image generation requirement information. The target subject modification intent indicates whether the subject in the first reference image needs to be modified. Based on the mapping relationship between the subject modification intent and the image generation method, the target image generation method corresponding to the target subject modification intent is obtained. If the target subject modification intent indicates that the subject in the first reference image does not need to be modified, the target image generation method is an accurate image generation method; if the target subject modification intent indicates that the subject in the first reference image needs to be modified, the target image generation method is a high-generalization image generation method. The generation module is used to generate a target image based on the image generation requirement information and the first reference image, using the target image generation method. The required text corresponds to text features, the target image is generated using a high-generalization image generation method, and the generation module is used for: Feature extraction is performed on the first reference image to obtain the second image features; Based on the description objects corresponding to each sub-feature in the text features and the second image features, the text features and the second image features are interleaved to obtain image-text interleaved features. The image-text interleaving features are input into the second image model to obtain the target image.

21. The apparatus according to claim 20, wherein, The judgment module is used for: Obtain the requirement text from the image generation requirement information; The requirement text is analyzed to understand the requirement and obtain the requirement type corresponding to the image generation requirement information; Based on the aforementioned requirement type, determine whether the image generation requirement information needs to be queried using a reference image.

22. The apparatus according to claim 20, wherein, The generation module is also used for: If the image generation requirement information does not require a reference image query, the target image is generated based on the image generation requirement information and using the target image generation method.

23. The apparatus according to claim 20, wherein, The query module is used for: Obtain the image subject information included in the image generation requirement information; Based on the image subject information, an image query is performed in the set image library to obtain the first reference image.

24. The apparatus according to claim 23, wherein, The query module is used for: Based on the image subject information, an image query is performed in the set image library to obtain candidate reference images; Obtain the image quality of the candidate reference image, and obtain the set quality requirements corresponding to the image generation requirement information; Based on the image quality and the set quality requirements, the candidate reference images are filtered to obtain the first reference image.

25. An intelligent agent, comprising: The input module is used to obtain image generation requirement information; The processing module is used to generate requirement information based on the image and query to obtain the first reference image; Obtain the requirement text from the image generation requirement information; The requirement text is input into the first model for subject modification intent detection to obtain the target subject modification intent corresponding to the image generation requirement information. The target subject modification intent indicates whether the subject in the first reference image needs to be modified. Based on the mapping relationship between the subject modification intent and the image generation method, the target image generation method corresponding to the target subject modification intent is obtained. If the target subject modification intent indicates that the subject in the first reference image does not need to be modified, the target image generation method is an accurate image generation method; if the target subject modification intent indicates that the subject in the first reference image needs to be modified, the target image generation method is a high-generalization image generation method. Based on the image generation requirement information and the first reference image, the target image is generated using the target image generation method. Output module, used to output the target image; The required text corresponds to text features, the target image is generated using a high-generalization image generation method, and the processing module is used for: Feature extraction is performed on the first reference image to obtain the second image features; Based on the description objects corresponding to each sub-feature in the text features and the second image features, the text features and the second image features are interleaved to obtain image-text interleaved features. The image-text interleaving features are input into the second image model to obtain the target image.

26. An intelligent agent, comprising: The input module is used to obtain image generation requirement information; The processing module is used to determine whether the image generation requirement information needs to be queried for a reference image; If the image generation requirement information necessitates a reference image query, a first reference image is obtained based on the image generation requirement information; and the requirement text in the image generation requirement information is retrieved. The requirement text is input into the first model for subject modification intent detection to obtain the target subject modification intent corresponding to the image generation requirement information. The target subject modification intent indicates whether the subject in the first reference image needs to be modified. Based on the mapping relationship between the subject modification intent and the image generation method, the target image generation method corresponding to the target subject modification intent is obtained. If the target subject modification intent indicates that the subject in the first reference image does not need to be modified, the target image generation method is an accurate image generation method; if the target subject modification intent indicates that the subject in the first reference image needs to be modified, the target image generation method is a high-generalization image generation method. Based on the image generation requirement information and the first reference image, the target image is generated using the target image generation method. Output module, used to output the target image; The required text corresponds to text features, the target image is generated using a high-generalization image generation method, and the processing module is used for: Feature extraction is performed on the first reference image to obtain the second image features; Based on the description objects corresponding to each sub-feature in the text features and the second image features, the text features and the second image features are interleaved to obtain image-text interleaved features. The image-text interleaving features are input into the second image model to obtain the target image.

27. An intelligent agent system, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7 or 8-12.

28. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method of any one of claims 1-7 or 8-12.

29. A computer program product comprising a computer program that, when executed by a processor, implements the method of any one of claims 1-7 or 8-12.

Images