Information processing method, device and system for image generation, and related equipment

Abstract

The invention provides an information processing method, device and system for image generation and related equipment, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like, and can be applied to scenes such as AIGC content generation based on artificial intelligence. According to the specific implementation scheme, the method comprises the steps of obtaining image generation demand information, used for generating an image, of a target object; and inputting the image generation demand information into a prompt engineering system so as to convert the image generation demand information into image generation parameters adopted by an image generation system based on the prompt engineering system.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to technical fields such as computer vision, deep learning, and large models, and can be applied to scenarios such as AIGC (Artificial Intelligence Generative Content) content generation based on artificial intelligence. Background Art

[0002] Image generation technology is an important research direction in the field of artificial intelligence. With the development of artificial intelligence technology, image generation technology is constantly being updated. Image generation technology is diverse and practical in different industries. For example, image generation technology can assist in artistic creation and design, and can help understand abstract concepts in the field of education.

[0003] With the continuous advancement of technology, the application areas of image generation technology will be further expanded. Summary of the invention

[0004] The present disclosure provides an information processing method, apparatus, system, and related equipment for image generation.

[0005] According to one aspect of the present disclosure, there is provided an information processing method for image generation, comprising:

[0006] Obtaining the raw image requirement information of the target object for generating an image;

[0007] The raw image requirement information is input into a prompt engineering system, so as to convert the raw image requirement information into raw image parameters for use by an image generation system based on the prompt engineering system.

[0008] According to another aspect of the present disclosure, there is provided an information processing device for image generation, comprising:

[0009] An acquisition unit, used for acquiring image generation requirement information of a target object for generating an image;

[0010] The processing unit is used to input the raw image requirement information into the prompt engineering system, so as to convert the raw image requirement information into raw image parameters for use by the image generation system based on the prompt engineering system.

[0011] According to another aspect of the present disclosure, there is provided an information processing system for image generation, comprising:

[0012] A prompt engineering system, used for processing raw image requirement information of a target object, so as to convert the raw image requirement information into raw image parameters adopted by an image generation system based on the prompt engineering system;

[0013] The image generation system is used to generate a target image based on the image generation parameters.

[0014] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0018] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0019] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0022] Figure 1 is a flowchart of an information processing method for image generation according to an embodiment of the present disclosure;

[0023] Figure 2 is a schematic diagram of a scenario of an information processing method for image generation according to an embodiment of the present disclosure;

[0024] Figure 3 is a training schematic diagram of a first intent recognition sub-model according to an embodiment of the present disclosure;

[0025] Figure 4 is another flowchart of an information processing method for image generation according to an embodiment of the present disclosure;

[0026] Figure 5 is a schematic diagram of the intention of identifying image generation requirement information according to an embodiment of the present disclosure;

[0027] Figure 6is an architectural diagram of a prompt engineering system according to an embodiment of the present disclosure;

[0028] Figure 7 is a model type of a target raw image model according to an embodiment of the present disclosure;

[0029] Figure 8 is a schematic diagram of a prompt engineering system according to an embodiment of the present disclosure;

[0030] Fig. 9 is a schematic diagram of a prompt engineering system under multi-round dialogue according to an embodiment of the present disclosure;

[0031] Fig.10 is a schematic diagram of a security management strategy according to an embodiment of the present disclosure;

[0032] Fig.11 is an architectural diagram of an information processing system for image generation according to an embodiment of the present disclosure;

[0033] Fig.12 is a structural schematic diagram of an information processing device for image generation according to an embodiment of the present disclosure;

[0034] Fig.13 It is a block diagram of an electronic device used to implement the information processing method for image generation according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0036] The terms "first", "second", etc. in this disclosure are used to distinguish similar objects, and are not necessarily used to describe a specific order or precedence. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, including a series of steps or units. The method, system, product or device is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] In order to better improve the implementation effect of the image generation system in actual application products, a Prompt engineering system is proposed in the embodiment of the present disclosure to provide key parameters for the image generation system to facilitate the implementation of the image generation system and improve the quality of images generated by the image generation system.

[0038] like Figure 1 FIG. 1 is a flow chart of an information processing method provided in an embodiment of the present disclosure, which includes the following contents:

[0039] S101, obtaining image generation requirement information of a target object for generating an image.

[0040] The image generation requirement information is used to represent the description information of at least one dimension of the image to be generated by the target object, such as subject, style, content description, etc. Text description or graphic description can be used.

[0041] When the raw image requirement information is in the form of voice, the voice can also be converted into text, and the text is used to construct the raw image requirement information.

[0042] S102, inputting the raw image requirement information into the prompt engineering system, so as to convert the raw image requirement information into raw image parameters for use by the image generation system based on the prompt engineering system.

[0043] In the disclosed embodiment, a plurality of neural network models may be set in the prompt engineering system, and different neural network models process all or part of the information in the raw image requirement information based on different task requirements, so as to extract raw image parameters suitable for the image generation system from the raw image requirement information. These raw image parameters can be more adapted to the image generation system than the raw image requirement information. For example, they can use expressions that can be accurately understood by the image generation system to describe the requirements for generating images. It can also more accurately describe the services that the image generation system should provide. For example, the image generation system provides a variety of models for generating images, and different models meet different requirements. Raw image parameters can better use these models to meet the personalized needs of the target object.

[0044] In the disclosed embodiment, the content in the raw image requirement information is processed based on the prompt engineering system. By utilizing the prompt engineering system, the raw image requirement information can be systematically understood based on the characteristics of the image generation system, so that the raw image requirements of the target object are clearer and the image generation system can be better used, thereby effectively improving the quality of the generated image and enhancing the user experience.

[0045] In some embodiments, the image generation parameters include at least one of the following:

[0046] 1) Risk information corresponding to the raw image demand information. The risk information is used to indicate whether the raw image demand information contains malicious demands. Malicious raw image demands need to be filtered or rewritten into normal demands.

[0047] 2) Image generation intention information corresponding to the image generation requirement information. In the disclosed embodiment, the image generation intention information may describe the image content expected to be generated and / or the method of generating the image from multiple dimensions or angles.

[0048] Among them, the raw image intention information includes but is not limited to the raw image type (such as the first round of text-generated images, the first round of image-generated images, multiple rounds of text-generated images, etc.), image size, image style, reference image information, the type of target subject to be generated, etc.

[0049] 3) Model type of the target image generation model that adapts to the image generation requirement information. That is, the image generation system provides a variety of models for generating images. Different types of models have their own characteristics. By predicting the model type, a suitable model can be selected as the target image generation model to generate images, which can improve the implementation effect of the image generation system.

[0050] 4) Rewrite the target prompt words obtained by the raw image demand information. By prompting the engineering system to rewrite the raw image demand information, the characteristics of the image generation system can be automatically adapted to improve the quality of the image generated by the image generation system. Moreover, it can also support the target object to flexibly use different expressions to express the raw image demand, thereby improving the user experience.

[0051] 5) Reply information for raw image demand information. The reply information is used to make a personalized reply to the raw image demand information of the target object, so as to facilitate interaction with the target object and improve the user experience.

[0052] 6) Typesetting layout information of the target subject type in the image required to be generated by the raw image requirement information.

[0053] Among them, the target subject types include but are not limited to modern names, ancient names, fictitious names, animal names, plant names, product models, brand names, location landmarks, film and television drama culture, popular Internet buzzwords, etc.

[0054] The typesetting layout information of the target subject type in the image to be generated by the image generation requirement information is used to indicate the position of the subject corresponding to the target subject type in the generated image. For example, the image can be divided into multiple regions, and the typesetting layout information can express the image region where the subject corresponding to the target subject type is located.

[0055] In implementation, when the raw image parameters include risk information, raw image intent information, model type, rewritten target prompt words, reply information, and typesetting layout information of the target subject type, such as Figure 2As shown, the raw image requirement information 201 is input into the prompt engineering system 202, and the aforementioned six raw image parameters can be obtained, namely, the risk information 2021 corresponding to the raw image requirement information, the raw image intention information 2022 corresponding to the raw image requirement information, the model type 2023 of the target raw image model adapted to the raw image requirement information, the prompt word 2024 obtained by rewriting the raw image requirement information, the reply information 2025 for the raw image requirement information, and the typesetting layout information 2026 of the target subject type in the image to be generated by the raw image requirement information, which are transmitted to the downstream network application service 203 (i.e., the image generation system, exemplarily which can be an iRAG (image based Retrieval-Augmented Generation, retrieval enhanced raw image) system) to obtain the target image 204 generated by the downstream network application service.

[0056] In the disclosed embodiment, multi-dimensional image generation parameters are used for use by the image generation system, so that the image generation system can be better utilized and the quality of the generated images and the user experience can be improved.

[0057] Below, the aforementioned raw image parameters are described in detail:

[0058] 1) Risk information corresponding to raw image demand information

[0059] In some embodiments, risk information is determined in the following manner, which can be implemented as follows: inputting the requirement text in the raw image requirement information into a security risk identification model to identify the risk information of the raw image requirement information; the risk information includes risk type and / or risk level.

[0060] Among them, the security risk identification model can be obtained by training a 3B (3 billion) scale pre-trained language model.

[0061] Among them, the pre-trained language model can be a large language model (LLMs). Large language models refer to a specific type of large model that is specifically used to process text data. This type of model is a natural language processing model based on a neural network, which can be used to generate, understand and process text data. Large language models can have tens of billions of parameters, can generate high-quality text, and can be used for various natural language processing tasks, such as question answering, text generation, dialogue systems, etc.

[0062] The large language model has good reasoning ability and small sample learning ability. The large language model can understand the model on a large scale or can be trained on a large number of samples. Based on the large model, accurate semantic understanding can be achieved.

[0063] Among them, the risk type is a type in the risk type set. The set can define different risk types in a fine-grained manner for classification and prediction by the security risk identification model. During implementation, the finer the granularity of the risk type classification in the risk type set, the more accurate the identification of specific risk types can be, and subsequent processing can be made accordingly, such as deciding whether to filter or rewrite according to the situation.

[0064] The risk level is used to indicate the level of risk involved. Exemplarily, the risk level can be divided into three levels, including high risk, medium risk, and low risk. During implementation, the risk level can be determined according to the risk type involved and / or the number of risk types.

[0065] When the risk level is determined to be high risk, the generation requirement of the target object may not be supported, and the target object may be prompted to modify the generation requirement information.

[0066] When the risk level is determined to be low risk and the risk type is a preset type, the risky content in the image generation requirement information can be deleted. For example, if the image content of the preset type is not the core content of the image (such as background, decoration), the corresponding image content of the preset type may not be generated.

[0067] In addition, when there is replaceable safety text, the safety text can be used to replace the corresponding risk text. After the replacement is completed, the safety risk identification model is input to identify the risk information of the replaced raw image requirement information to ensure that there is no risk before continuing the subsequent processing flow.

[0068] In the disclosed embodiment, risk detection is performed on the raw image requirement information to reduce risk factors in the raw image requirement information and improve the quality of the generated image.

[0069] In some embodiments, a supervised training method can be used to obtain a security risk identification model. Specifically, it can be implemented as follows: obtain a first sample set, wherein the first sample set includes multiple first samples, the annotated risk corresponding to each first sample (such as including risk type and / or risk level), and the first task instruction of each first sample; input the first sample set into the initial security risk identification model to obtain the predicted risk output by the model; compare the predicted risk with the annotated risk to obtain a loss value, adjust the model parameters of the initial security risk identification model based on the loss value, and train the security risk identification model when the convergence condition is met; wherein the first task instruction is used to indicate that the task that the initial security risk identification model needs to perform is to perform a risk detection task.

[0070] 2) Raw image intention information corresponding to raw image demand information

[0071] In some embodiments, the raw image intent information is determined in the following manner, which can be implemented as follows: raw image requirement information is identified based on a multimodal intent understanding model to obtain the raw image intent information.

[0072] Among them, the multimodal intent understanding model is a model that can understand information of multiple modalities to predict the intent of the raw image. Among them, the multimodal intent understanding model can be a multimodal large language model (MLLM), or it can be constructed by multiple sub-models. Among them, different sub-models complete different intent understanding tasks. For example, the multimodal intent understanding model may include a first intent recognition sub-model and a second intent recognition sub-model, as well as a multimodal large model.

[0073] In the disclosed embodiment, the image generation requirement information is identified based on the multimodal intention understanding model. On the one hand, through multimodal intention understanding, it can support the target object to use multiple modal information to express the image generation intention, thereby improving the flexibility and convenience of expressing the image generation requirement. On the other hand, through the multimodal intention understanding model, the image generation requirement of the target object can be accurately understood, thereby improving the quality of the generated image.

[0074] In some embodiments, raw image requirement information is identified based on a multimodal intent understanding model to obtain raw image intent information, which can be implemented as follows: text understanding prompt words are constructed based on the requirement text and target prompt template in the raw image requirement information; the text understanding prompt words are input into the first intent recognition sub-model in the multimodal intent understanding model to obtain at least one of the following information in the raw image intent information: raw image type, image size, image style; wherein the raw image type is a type in a preset type set.

[0075] Among them, in some embodiments, the preset type set includes at least one of the following types: first-round text-generated image type, first-round picture-generated image type, text-generated image multiple-round subsequent type, picture-generated image multiple-round subsequent type, and image editing type.

[0076] The first-round text-generated image type refers to the image-generated image requirement information obtained in the first round of dialogue in a image-generated image task. The image-generated image requirement information is the text content provided by the target object.

[0077] The first-round image generation type indicates the image generation requirement information obtained in the first round of dialogue in a image generation task, and the image generation requirement information is expressed in images;

[0078] The text-generated image multiple-round subsequent type means that after the first round of dialogue uses text to express the image generation requirements, the target object continues to use multiple rounds of dialogue to improve the image generation requirements in order to continue generating images;

[0079] The multiple rounds of subsequent type of image generation indicates that after the first round of dialogue uses images to express the demand for image generation, the target object continues to use multiple rounds of dialogue to improve the demand for image generation and continue to generate images;

[0080] The image editing type indicates that there are local contents in the known image that need to be added, modified, deleted, etc. The known image can be an image generated by a text-generated image or a picture-generated image, or it can be an image provided by the target object.

[0081] In the disclosed embodiment, the raw image types are divided into at least one of the aforementioned types, so that the raw image intention can be refined, and the accuracy of understanding the raw image requirements is improved, thereby further improving the quality of the generated image.

[0082] The image size is used to indicate the size requirement for the generated image. For example, if the image required to be generated is a poster, the image needs to be adjusted to the size corresponding to the poster. In addition, in multiple rounds of raw image tasks, for example, the size of the first round of raw images can be required to be a poster, and the size of the second round of raw images can be required to be a book cover. The image size is predicted based on different usage requirements.

[0083] The image style is used to represent the style of the image to be generated, and examples thereof include abstract style, realistic style, cartoon style, etc. In addition, in multiple rounds of image generation tasks, the style of each round may also be different.

[0084] In the disclosed embodiment, the required text is input into the first intent recognition sub-model, and the first intent recognition sub-model focuses on recognizing the raw image type, image size, and image style. Based on this, the accuracy of the raw image intent information can be improved, and the quality of the generated image can be improved.

[0085] In some embodiments, the first intent recognition sub-model is obtained by the following training method, which can be specifically implemented as follows:

[0086] Step A1: Distill training sample sets with different raw image intent labels based on the large language model.

[0087] In some embodiments, an initial sample set may be obtained by random sampling. Then, the samples are distilled from the large language model to train the first intent recognition sub-model. The distilled samples may be implemented as follows:

[0088] Step A11, obtaining an initial sample set.

[0089] The initial sample set may be sample data obtained by random sampling, and the sample data includes text content provided online when implementing the text-generating graph task.

[0090] Step A12, based on the sequence labeling task, multiple intermediate samples are screened out from the initial sample set to obtain an intermediate sample set.

[0091] The sequence labeling task is a basic task in natural language processing (NLP), which involves assigning a label or category to each word or subword in the text.

[0092] The sequence labeling problem can be seen as a generalization of the classification problem, or a simple form of the more complex structure prediction problem. In sequence labeling, the input is an observation sequence, the output is a label sequence or state sequence, and the goal is to give a label sequence as a prediction for the observation sequence.

[0093] Since the quality of the initial sample set obtained by random sampling is uneven and even mixed with noisy data, the sequence labeling task can complete the sample screening and obtain intermediate samples with better quality.

[0094] The intermediate sample set contains multiple user requirements (queries). In order to train the first intent recognition sub-model, a large language model is required to distill training samples.

[0095] Step A13, using a large language model to generate a raw image intent label for each sample in the intermediate sample set to obtain a training sample set.

[0096] That is, the large language model processes each user query in the intermediate sample set and labels it according to the task requirements of the first intent recognition sub-model.

[0097] Among them, the raw image intent label includes but is not limited to the raw image type (including the first-round text raw image intent, the first-round image raw image intent, the text raw image multiple-round subsequent intent, the image raw image multiple-round subsequent intent, and the image editing intent), image size, image style, etc.

[0098] In the disclosed embodiment, the initial sample set is preliminarily screened based on the sequence labeling task to optimize the sample quality, and the samples are distilled out through a large language model, which can realize automatic labeling, so as to facilitate the transfer of the knowledge of the large model to the first intent recognition sub-model through training samples, thereby improving the intent recognition accuracy of the first intent recognition sub-model.

[0099] Step A2: classify the training sample set based on the raw image intent labels to obtain sample subsets corresponding to each raw image intent label.

[0100] Exemplarily, when training sample A is {sample A-raw image intent label 1}, training sample B is {sample B-raw image intent label 2}, training sample C is {sample C-raw image intent label 1}, and training sample D is {sample D-raw image intent label 1}, the generated sample subsets include two types, one is raw image intent label 1 {training sample A; training sample C; training sample D}, and the other is raw image intent label 2 {training sample B}.

[0101] Step A3: Based on the sample subsets corresponding to the raw image intent labels, supervised training is performed on the pre-trained initial model to obtain a first intent recognition sub-model.

[0102] During implementation, the initial model is trained with samples of general tasks, and when the convergence conditions are met, a pre-trained initial model is obtained, and then steps A1-A3 are executed to obtain the first intent recognition sub-model.

[0103] During implementation, a subset of samples of each raw image intent label is sampled according to a preset ratio to obtain a batch of multiple samples. These multiple samples are input into the pre-trained initial model for supervised training, so that the pre-trained initial model learns the recognition pattern of each intent, and when the convergence conditions are met, the first intent recognition sub-model is obtained.

[0104] In the disclosed embodiment, a training sample set is distilled based on a large language model, and a pre-trained initial model is trained in a supervised manner based on the training sample set, so that the initial model can learn how to identify the raw image intent information through supervised training based on certain knowledge, and the first intent recognition sub-model obtained can more accurately identify the raw image intent.

[0105] In some embodiments, in order to improve the recognition accuracy of the first intent recognition sub-model, the reflective optimization mechanism of the large language model can also be relied upon to optimize the target prompt template used by the first intent recognition sub-model, which can be implemented as follows:

[0106] Step B1, predicting the initial intent label of the target sample set based on the first intent recognition sub-model while adopting the prompt template to be adjusted.

[0107] During implementation, the powerful semantic understanding ability of the large model is used to predict the initial intent labels of the target sample set.

[0108] The target sample set and the training sample set may be the same sample set or different sample sets.

[0109] Step B2, using a large language model to determine whether the initial intent tag meets the requirements of multiple preset rules in the rule set.

[0110] Among them, the preset rule requirements are used to restrict the initial intent label. An example is to judge whether the initial intent label meets the preset rule requirements. The preset rule requirements may be that the initial intent label cannot contain metaphors or unclear content, etc. These preset rules can set the description method or content that should not be included in the initial intent label.

[0111] In addition, the aforementioned raw image types can be further subdivided into subtypes with different operation intentions. The preset rule requirements can also be used to distinguish similar raw image subtypes. The specific subtypes can be referred to later. Figure 5 For ease of understanding, for example, the raw image requirement information is "delete glasses on the face", that is, the raw image subtype is local deletion processing, and the raw image requirement information is "delete spots on the face", that is, the raw image subtype is local modification processing. The two are similar in language, but different in fine-grained intent. Therefore, the constraints required by the preset rules can be used to train the first intent recognition submodel to distinguish similar raw image subtypes.

[0112] Step B3: When any preset rule requirement is not met, the prompt template to be adjusted is optimized based on the large language model and any preset rule requirement is used as a benchmark to obtain a target prompt template.

[0113] In the disclosed embodiment, the prompt template to be adjusted is optimized based on preset rules to avoid the model outputting irrelevant content or hallucinations. At the same time, a standardized target prompt template is used to further improve the accuracy of intent recognition by the first intent recognition sub-model, thereby improving the quality of the raw image parameters and the quality of the generated image.

[0114] In some embodiments, the first intent recognition sub-model is trained using the 10B pre-trained language model as a base model. Figure 3The schematic diagram of its training framework is shown as follows: samples can be randomly collected online to obtain an initial sample set 301, and multiple intermediate samples can be screened out from the initial sample set 301 based on the sequence labeling task to obtain an intermediate sample set 302; the intermediate sample set 302 is input into a large-sized LLM / MLLM303, and after distillation of the large model 303, the large model is a large-sized LLM / MLLM, and the raw image intent label of each sample is obtained, thereby obtaining a synthetic sample 304 constructed by the user query and its corresponding raw image intent label, and the synthetic sample 304 is a sample pair including the user query and its corresponding raw image intent label. In order to improve the sample quality, the synthetic sample 304 can be modified and optimized through manual review 305. And new sample pairs can be automatically synthesized to finally obtain a training sample set 306. The training sample set can be divided into different sample subsets 307 based on each raw image intent label. The training samples of the same batch are selected from different sample subsets 307 according to the ratio to train the first intent recognition sub-model 308, wherein the first intent recognition sub-model 308 is a small-sized LLM / MLLM. Then the large model 303 uses multiple preset rules in the rule set to discriminate the intent label predicted by the first intent recognition sub-model 308, that is, the evaluation sample 309. The large model 303 is used for multi-dimensional evaluation. When it is detected that any preset rule requirement is not met, the large model 303 reflects on and corrects the prompt template used by the first intent recognition sub-model 308.

[0115] In addition, the first intent recognition sub-model may include multiple model modules, each of which is used to identify the type of raw image, image size, and image style. Figure 3 way to train.

[0116] In some embodiments, the target object may also provide a first reference image for the image generation system to generate an image. In this case, the raw image intent expressed by the target object based on the first reference image can be identified, which can be implemented as follows: when the raw image requirement information includes the first reference image provided by the target object, the reference image information of the first reference image is obtained based on the multimodal large model in the multimodal intent understanding model; the raw image intent information includes the reference image information. The reference image information includes at least one of the following information: the reference subject type, the number of reference subjects, and the text description information of the first reference image.

[0117] During implementation, when a first reference image is included, a multimodal large model can be used to understand the first reference image. When the first reference image includes multiple reference subjects, the number of each reference subject can be identified, and the operations corresponding to each reference subject may be different.

[0118] For example, the first reference image may include subject A and subject B, and its raw image requirement information is "adjust subject A to a realistic style and delete subject B". Therefore, the intention of "changing the color of subject A" is to modify the part, and the intention of "deleting subject B" is to delete the part of the image. The two intentions are different, so the corresponding operations are different.

[0119] The first reference subject type may exemplarily include modern names, ancient names, fictitious names, animal names, plant names, product models, brand names, location landmarks, film and television drama culture, popular Internet buzzwords, and the like.

[0120] The text description information of the first reference image may be the text description information of the first reference image obtained by understanding the content of the first reference image using the powerful reasoning ability and content understanding ability of the multimodal large model.

[0121] What information is included in the specific reference image information can be set based on actual conditions, and the embodiments of the present disclosure do not limit this.

[0122] In the embodiment of the present disclosure, when the first reference image is used, the multimodal intent understanding model can accurately understand the content expressed by the first reference image to assist the model's intent recognition task, and can help the image generation system accurately understand the user's purpose of using the first reference image to improve the quality of the generated image.

[0123] In some embodiments, not only can the reference subject in the reference image be identified, but also the subject expected to be generated in the text description can be understood. Therefore, the raw image requirement information is identified based on the multimodal intention understanding model to obtain the raw image intention information, which can be implemented as follows: the requirement text in the raw image requirement information is input into the second intention recognition sub-model in the multimodal intention understanding model to obtain the target subject type indicated by the requirement text; the raw image intention information includes the target subject type.

[0124] Among them, the type range covered by the target subject type is consistent with the first reference subject type, which has been described in the previous text and will not be repeated here in the embodiment of the present disclosure.

[0125] In the disclosed embodiment, the target subject type is identified using the requirement text in the image generation requirement information to help the image generation system generate an image around the subject, thereby improving the quality of the generated image.

[0126] 3) Model type of target raw image model that adapts to raw image demand information

[0127] In some embodiments, the model type of the target image generation model is determined in the following manner, which can be specifically implemented as follows: processing the image generation intent information based on the distribution strategy model, and determining the model type of the target image generation model for generating images for the image generation requirement information in the image generation system.

[0128] In the disclosed embodiment, a suitable image generation model type can be screened for the target object based on the distribution strategy model, and the model type can be provided to the image generation system as an image generation parameter. An appropriate model can be used to generate images for the target object, thereby improving the user experience.

[0129] In some embodiments, the model type of the target image generation model includes at least one of the following:

[0130] The base model supports general image generation tasks. The base model may be an SD model (stable defusion, a general text-generated image model), which is used for general image generation tasks. The base model can support various image generation tasks, with the emphasis on supporting more task types, but the accuracy may be limited.

[0131] The precise model is used to generate high-fidelity images. The precise model can be used to generate images that are required to be highly consistent with the actual scene, for example, specific models of vehicles, trademarks, etc.

[0132] A highly generalized model is used to generate images with fidelity that meets preset requirements and has artistic effects; a highly generalized model can be used to allow certain modifications or optimizations, and to allow the addition of ideas and creations, relative to a precise model. For example, a portrait image, if it is a portrait, may have a certain artistic style.

[0133] The editing model is used to perform editing operations on known images. The editing model is used to make local adjustments to known images, for example, deleting glasses on a person's face, modifying the position of a mole on the face, the shape and color of glasses, etc.

[0134] In the disclosed embodiment, based on the corresponding intentions of the aforementioned multiple models, most image generation tasks can be processed as much as possible to improve the ability of the image generation system to adapt to multiple tasks and improve the image generation quality.

[0135] In some embodiments, when the target subject type in the image to be generated according to the raw image requirement information is a preset type, the target model type for processing the raw image requirement information is a security model; wherein the security model is used to generate images that avoid public opinion risks.

[0136] For example, the preset type can be a landmark building in each region, a landmark building in a scenic spot, etc. During implementation, the subject text and image corresponding to each preset type can be associated and stored in the form of a key-value pair. When the target subject is detected as a subject in the key-value pair list, it is determined to be a preset type and generated using a security model.

[0137] In the disclosed embodiment, for key target subject types that require high consistency, a security model is used to process them to avoid public opinion risks.

[0138] 4) Rewrite the target prompt words obtained from the raw image demand information, as well as the response information for the raw image demand information

[0139] In some embodiments, the target prompt word and the reply information of the raw image requirement information are determined by: Figure 4 As shown, it can be implemented as:

[0140] S401, processing the target object's image generation demand information based on the distribution strategy model to obtain a processing result.

[0141] Among them, the processing results include those that require retrieval enhancement and those that do not require retrieval enhancement. When retrieval enhancement is not required, the raw image demand information is rewritten based on the prompt word optimization model to obtain the target prompt word and the reply information; when retrieval enhancement is required, S402 is executed.

[0142] S402: When the processing result indicates that search enhancement is required, a search statement generated by the distribution strategy model is obtained from the processing result.

[0143] The search statement may be descriptive information of a key subject obtained from the raw image requirement information of the target object for generating an image, and the key subject may exemplarily be the target subject in the raw image requirement information.

[0144] In some embodiments, the training method of the distribution strategy model can be implemented as follows: obtaining a second sample set; wherein the second sample set includes multiple second samples, annotated labels corresponding to each second sample, and a second task instruction, wherein the annotated label indicates the model type applicable to the second sample, whether to perform retrieval enhancement, and may even include the target subject type in the second sample. Wherein, the annotated model types include but are not limited to base models, precise models, high generalization models, and edited models, and the second sample set is input into the initial distribution strategy model to obtain a prediction result, which includes whether the binary classification result of enhancement, the prediction subject, and the prediction type, and the prediction type indicates the model type to be adopted for the predicted second sample. Based on the comparison between the prediction result and the annotated label, the distribution strategy model is obtained when the convergence condition is met. Wherein, the second task instruction is used to indicate the tasks that the initial distribution strategy model needs to perform, including whether the classification task is enhanced, the prediction task of the target subject type, and the distribution strategy task.

[0145] S403, based on the search statement, searching for a second reference chart of the birth chart required information.

[0146] Among them, the second reference image is used to strengthen the image generation intention of the image generation requirement information.

[0147] S404, rewriting the second reference image and the raw image requirement information based on the prompt word optimization model to obtain target prompt words of the target raw image model in the adaptive image generation system, and generating reply information.

[0148] In the disclosed embodiment, it is possible to automatically determine whether the raw image requirement information needs to be enhanced. If enhancement is required, a second reference image is generated to assist the model in understanding, thereby making the intent of the raw image requirement information clearer and further improving the image generation system's ability to understand the raw image requirements.

[0149] In some embodiments, a rewriting operation is performed on the second reference image and the raw image requirement information based on a prompt word optimization model to obtain target prompt words for the target raw image model in the adaptive image generation system. This can be implemented as follows: obtaining a target dialogue text for the current round of dialogue with the target object from the raw image requirement information; constructing information to be rewritten based on the target dialogue text, historical dialogue situations in the raw image requirement information, and the second reference image; and inputting the information to be rewritten into the prompt word optimization model to obtain target prompt words.

[0150] In the disclosed embodiment, the drawing requirements can be more clearly expressed based on the second reference image, thereby improving the drawing quality.

[0151] If the current round does not require retrieval enhancement, the information to be rewritten is constructed based on the target dialogue text of the current round of dialogue and the historical dialogue situation; the information to be rewritten is input into the prompt word optimization model to obtain the target prompt word. Specifically, when the current round of dialogue is the first round of dialogue, the target dialogue text and the target characters are spliced ​​into structured information to be rewritten according to a preset format.

[0152] When implemented, the preset format can be set to a format such as <target dialogue text, target character>. In the case of the first round of dialogue, the format meets the following requirements:

[0153] 1) Use the target field to express the target dialogue text, and the target dialogue text is the value of the target field. Use the dialogue start symbol and dialogue end symbol to express the start and end of the target dialogue text, so that the text graph model can recognize and verify the integrity of the target dialogue text.

[0154] 2) Use target characters to represent historical conversation records. If the target character is empty, it means that the current round of conversation is the first round of conversation and there is no historical conversation record.

[0155] In the disclosed embodiment, the target dialogue text and the target characters are spliced ​​in a preset format, which can ensure that the information to be rewritten is regular and logical, so that the large language model can understand and sort out the image generation requirements of the current round of dialogue and the specific content of the historical situation, which can reduce the understanding error of the large model and improve the accuracy of the target prompt words, thereby improving the quality of the generated image.

[0156] When the current round of dialogue is the latest round of dialogue among multiple rounds of dialogue, the information to be rewritten can be constructed based on the dialogue text and historical dialogue situations, which can be achieved by the following steps:

[0157] Step D1, obtaining the historical conversation text provided by the target object in the historical conversation record, and the historical prompt words generated by the prompt word optimization model according to the requirements of the historical conversation text;

[0158] Step D2, splicing the target conversation text and the historical conversation record according to a preset format to obtain the information to be rewritten; wherein, in the information to be rewritten, the historical conversation text and the historical prompt words of the historical conversation text are organized into information tuples according to the association relationship, and the information tuple is the value of the target character.

[0159] Based on the above, the preset format can be set to a format such as <target dialogue text, target characters>. When the current dialogue round is the latest dialogue round in multiple dialogue rounds, and the current dialogue round is the second dialogue round, the format meets the following requirements:

[0160] 1) Use the target field to express the target dialogue text of the current dialogue round, and the target dialogue text is the value of the target field. Use the dialogue start symbol and dialogue end symbol to express the start and end of the target dialogue text, so that the text graph model can recognize and verify the integrity of the target dialogue text.

[0161] 2) Using target characters to represent historical conversation records, the target characters contain two key fields, namely the first subfield and the second subfield.

[0162] 3) The first subfield is used to represent the historical conversation text of the first round of conversation, and the historical conversation text is the value of the first subfield. Moreover, the historical conversation text uses the first start symbol and the first end symbol to represent the beginning and the end of the historical conversation text, and is used to identify and verify the integrity of the historical conversation text. Similarly, the second subfield represents the historical prompt word of the first round of conversation, and the historical prompt word is the value of the second subfield. Moreover, the historical prompt word uses the second start symbol and the second end symbol to represent the beginning and the end of the historical prompt word, and is used to identify and verify the integrity of the historical prompt word.

[0163] 4) Use the target start symbol and target end symbol of the historical conversation record to indicate the start and end position of the target character value, so as to identify and verify the integrity of the target character value.

[0164] In the embodiment of the disclosure, in the scenario of multiple rounds of dialogue, the dialogue text and the historical dialogue record are spliced ​​in a preset format, and the requirements for image generation expressed in the historical dialogue record can be collected, and then spliced ​​and integrated with the latest round of dialogue text to obtain the information to be rewritten. In the information to be rewritten, the information tuple is expressed in a formatted way, which can facilitate the large model to understand the drawing requirements and preferences, so as to improve the quality of the generated target prompt words.

[0165] Accordingly, during the creation process, the target object may refer to the generated image and change the requirements for generating the image, which may require multiple rounds of dialogue to iterate the drawing requirements and finally obtain the desired image. When the historical dialogue record includes multiple rounds of historical dialogues, there will be multiple groups of information tuples in the historical dialogue record. Among them, the historical dialogue text and the corresponding historical prompt words of each round of historical dialogue are constructed as corresponding information tuples.

[0166] Taking multiple rounds of historical dialogue as an example, the format of the prompt information to be rewritten meets the following requirements:

[0167] 1) Use the target field to express the target dialogue text of the current dialogue round, and the target dialogue text is the value of the target field. Use the dialogue start symbol and dialogue end symbol to express the start and end of the target dialogue text, so that the text graph model can recognize and verify the integrity of the target dialogue text.

[0168] 2) Using target characters to represent historical conversation records, the target characters contain two key fields, namely the first subfield and the second subfield.

[0169] 3) In the case of multiple rounds of historical conversations, each round of historical conversations includes a corresponding pair of first subfields and second subfields. The first subfield and the second subfield of the same round of historical conversations are represented by the third start symbol and the third end symbol. This allows the text graph model to accurately understand the historical conversation text and historical prompt words of the same round of historical conversations.

[0170] 4) In each round of historical conversation, the corresponding first subfield is used to represent the historical conversation text of the round of historical conversation, and the historical conversation text is the value of the first subfield. Moreover, the historical conversation text uses the first start symbol and the first end symbol to represent the beginning and end of the historical conversation text, which is used to identify and verify the integrity of the historical conversation text. Similarly, the second subfield represents the historical prompt word of the round of historical conversation, and the historical prompt word is the value of the second subfield. Moreover, the historical prompt word uses the second start symbol and the second end symbol to represent the beginning and end of the historical prompt word, which is used to identify and verify the integrity of the historical prompt word.

[0171] 5) Use the target start symbol and target end symbol of the historical conversation record to indicate the start and end position of the target character value, so as to identify and verify the integrity of the target character value.

[0172] In the disclosed embodiment, the historical conversation text of each round of historical conversation and the corresponding historical prompt words are constructed as corresponding information tuples, which can provide a structured storage method for conversation information. By arranging these information tuples by round, the evolution process of the target object's raw image demand can be clearly expressed, which enables the large language model to accurately capture the target object's raw image demand and preference, clarify the rewriting direction and purpose of the generated historical prompt words, and improve the quality of the rewritten target prompt words.

[0173] In the disclosed embodiment, the target object may generate one or more images using a piece of text. Each image has its own characteristics and may also have its own corresponding core content. Therefore, the prompt words used to generate each image are different and independent. Based on this, when the first target round of dialogue in the historical dialogue record generates multiple images, the historical prompt words of the first target round of dialogue include sub-prompt words corresponding to the multiple images.

[0174] That is, the multiple sub-prompt words of the multiple images and the historical dialogue text of the first target round dialogue are constructed as the information tuple of the first target round dialogue.

[0175] Taking any round of historical dialogue as an example, assuming that multiple images are generated in any round of historical dialogue, in addition to meeting the above requirements, the expression of the historical prompt words in any round of historical dialogue meets the following requirements:

[0176] 1) The historical prompt word in the second subfield includes multiple sub-prompt words, and the multiple sub-prompt words are separated by separators to facilitate identification of the sub-prompt words of each image.

[0177] 2) It can express the correspondence between each sub-prompt word and the corresponding image; when implementing, the sub-prompt words can be sorted in the same sorting order as the generated multiple images, thereby establishing the correspondence between the sub-prompt words and the images through the sorting position; in addition, special descriptors can be added to the second subfield to express the correspondence between the sub-prompt words and the images. For example, the first special symbol and the second special symbol can be used to express that the two special symbols contain a sub-prompt word and a corresponding image identifier. It can be understood that it is sufficient to be able to identify the correspondence between the sub-prompt words and the images, and the embodiments of the present disclosure are not limited to this.

[0178] Thus, a group of information tuples is constructed from the multiple sub-prompt words of the multiple images and the historical dialogue text of the first target round dialogue.

[0179] In some embodiments, the target object uploads a first reference image and hopes to generate an image similar in style or content to the first reference image. In this case, in the current round of dialogue, if the target object provides the first reference image, the text description of the first reference image can be used as the content of the historical prompt words corresponding to the previous round of dialogue of the current round of dialogue, so as to construct the information to be rewritten.

[0180] Similarly, when retrieval enhancement is required, the second reference image can be automatically mined based on the aforementioned retrieval enhancement method, and the text description of the second reference image can be used as the content of the historical prompt words corresponding to the previous round of dialogue before the current round of dialogue, so as to construct the information to be rewritten.

[0181] In the disclosed embodiment, the second reference image obtained by retrieval enhancement is converted into a text description, and the text description of the second reference image is used as the content of the historical prompt word corresponding to the previous round of dialogue before the current round of dialogue, so as to construct the image generation requirement information, so that the information contained in the second reference image can be integrated with the previous dialogue information. The expression form of an information tuple can be used for the prompt word optimization model to sort out and understand the rewriting task, so as to ensure that the generated target prompt word can meet the expectations and improve the quality of the generated image.

[0182] That is, whether it is the first reference image or the second reference image, when there is a reference image, a corresponding text description can be generated for this reference image by combining image recognition technology and natural language generation technology. The generated text description can be determined based on the target dialogue text of the current round of dialogue. For example, if the target object specifies to generate a similar subject with reference to a subject in the reference image, the subject can be identified based on image recognition technology, and a corresponding description can be generated. If the target object specifies to generate an image with reference to the overall layout style of the reference image, the layout style of the reference image can be classified based on image recognition technology to obtain the corresponding layout style category. During implementation, the multimodal large model in the multimodal intent understanding model can be used to extract the reference information of the reference image for constructing the information to be rewritten.

[0183] In a multi-round conversation, as the number of conversation rounds increases, the time cost and resource consumption cost for processing and analyzing historical conversation data will increase accordingly. Therefore, in the embodiment of the present disclosure, when the number of conversation rounds in the historical conversation record is greater than m, the historical conversation text of m rounds of historical conversations is selected from the historical conversation record; wherein m is a positive integer greater than 1.

[0184] During implementation, the most recent m rounds of historical conversations can be selected for retention. This ensures that the generated image is more focused on the current drawing requirements of the target object, so that the generated image is more in line with the expectations of the target object. In addition, selecting m rounds of historical conversations from a large number of historical conversations can also effectively reduce the memory resources required to store and process a large number of historical conversations, thereby improving the efficiency of image generation.

[0185] In the disclosed embodiment, when the current round of dialogue is the first round of dialogue, the target prompt word is the prompt word obtained by rewriting the dialogue text by the prompt word optimization model. Based on the rewriting ability of the prompt word optimization model, the dialogue text of the target object can be converted into standardized and professional instructions that are more in line with the image generation technical requirements of the image generation system, which helps to better understand the target object's image generation requirements.

[0186] When the current round of dialogue is the latest round of dialogue among multiple rounds of dialogue, the target prompt word is the prompt word obtained by rewriting the designated prompt word by the prompt word optimization model; wherein the designated prompt word is the historical prompt word of the second target round of dialogue indicated by the dialogue text.

[0187] Take the second round of dialogue as the current round as an example, and if the current round of dialogue is the latest round of dialogue in multiple rounds of dialogue. If the dialogue text input by the target object of the current round of dialogue is: "The silver sword should be longer", the corresponding "prompts" is: ["The screen shows a fantasy anime style scene, a man wearing leather armor, in a fantasy forest full of monsters, with an iron sword on his back and a longer silver sword in his right hand. He is staring at a monster in front of him with his eyes fixed, giving people a sense of fantasy and mystery."]. From this, it can be seen that the historical prompt words in the second round of historical dialogue are rewritten based on the historical prompt words in the first round of historical dialogue combined with the second round of target dialogue text.

[0188] The prompt word obtained by rewriting the specified prompt word through the prompt word optimization model can be "The picture shows a fantasy anime style scene. A man wearing leather armor is in a fantasy forest full of monsters. He has an iron sword on his back and a longer silver sword in his right hand. He is staring at a monster in front of him with his eyes fixed, giving people a sense of fantasy and mystery."

[0189] That is, when generating an image, if the target dialogue text of the current round of dialogue does not clearly indicate which prompt word to rewrite, the historical prompt word of the previous round of dialogue is used as the designated prompt word by default. Similarly, when generating an image, if the target dialogue text of the current round of dialogue clearly indicates a historical prompt word, the historical prompt word is the designated prompt word.

[0190] When generating multiple images, if the target dialogue text of the current round of dialogue does not clearly indicate which prompt word to rewrite, and does not clearly indicate which image's prompt word to rewrite, the sub-prompt word of a randomly selected image in the previous round of dialogue is used as the designated prompt word by default. If the target dialogue text of the current round of dialogue clearly indicates which image's prompt word is for that round, the sub-prompt word of the image in the clearly specified round is used as the designated prompt word.

[0191] In the disclosed embodiment, in multiple rounds of conversations, the prompt words obtained by rewriting the specified prompt words through the prompt word optimization model can be integrated into the new image generation requirements based on the historical prompt words. The rewritten target prompt words can provide more comprehensive and accurate instructions for the image generation system.

[0192] Based on the prompt word optimization model, the raw image demand information is fully rewritten and optimized, so that various differentiated raw image demands can be aligned with the expression of the training data of the target raw image model, that is, it is mainly aligned with the expression of the training data, which can also be understood as alignment to the expression that the target raw image model can adapt to.

[0193] In addition, during the alignment optimization process, by constructing an art label system under the raw image scene, the richness and completeness of the content expressed by the prompt words are comprehensively improved from the dimensions of core picture content, picture style description, picture subject and limitation description, picture detail description, picture background modification description, special effects, composition, color tone, clarity description, and quality description to ensure the artistic quality of the raw image results.

[0194] in:

[0195] The image style description refers to the artistic style of the image, presenting the representative artistic characteristics of the image, such as the second dimension, illustration, Chinese style, ink painting, cartoon, animation, sketch, stick figure, landscape painting, etc. Realistic limited descriptions such as photo, realism, and reality are not artistic styles.

[0196] The subject of the image and the limited description refer to the subject presented in the image, which may include people, animals, plants, buildings, scenery, etc., as well as limited descriptions, such as an 18-year-old girl, blue sky, and white clouds.

[0197] The detailed description of the picture refers to the detailed description of the subject, which further refines and enriches the subject. For example, an 18-year-old girl has a delicate face and wears comfortable clothes with beautiful patterns on them.

[0198] The background modification description of the picture refers to adding background details to the subject to enhance the overall beauty of the picture and prevent the picture from appearing simple and empty. For example, the background is a green meadow with flowers and trees. Another example is that the background is blank, transparent, or pure color, which can make the picture more beautiful.

[0199] Special effects refer to beautifying and enriching the content of the picture, making it more exquisite, and can enhance the impact and appeal of the picture. There are words such as picture effects, light, focus, etc., such as: thick coating, perfect light and shadow, texture, ray tracing, Tyndall effect, blur, sharp focus, light and shadow, reflection, cg (Computer Graphics) rendering, 3d (Three Dimensions) rendering, multiple exposure, double exposure.

[0200] Composition refers to the position of the subject, that is, to organize the subject appropriately according to the subject's requirements to form a coordinated and complete picture. There are words such as lens, perspective, depth of field, such as: long shot, close-up, center, close-up, ultra-wide angle, front view, and bird's-eye view.

[0201] Tone refers to the relative lightness and darkness of the picture, depicting the color of the subject and the background color, such as: white, gold, color, rich color, colorful, rich color, colorful.

[0202] Clarity description refers to the clarity of the overall picture quality, such as: high definition, ultra high definition, 8k high definition.

[0203] Quality descriptions refer to the texture and subjective feeling of the overall picture, such as: masterpiece, highest quality, master work, award-winning, immersive, simple, and dilapidated.

[0204] When rewriting, the earlier the rewriting order is, the more important it is. The content included in the rewriting order depends on the dialogue text (user query) in the image generation requirement information. For content that is not very important, if the dialogue text does not mention it, it can be excluded from the rewritten prompt words. If the dialogue text requires image quality, the rewritten prompt words will include a description of the image quality. This applies to both the training stage and the inference stage.

[0205] When the target raw image model is an accurate model, the layout information of the main content can be automatically planned and generated according to the search statement, and the layout information, the second reference image and the raw image requirement information are rewritten based on the prompt word optimization model to obtain prompt words that match the accurate model and generate reply information. The layout information is the placement position of the main body, which includes upper left, lower left, upper right and lower right.

[0206] With the continuous development of artificial intelligence, in some scenarios, although relevant technologies can provide responses in the reply link while providing generated images, these responses are difficult to achieve personalized customization of the response content and cannot effectively improve the user experience and satisfaction of the target object.

[0207] In view of this, in the disclosed embodiment, the prompt word optimization model can have the ability to rewrite and generate prompt words through fine-tuning training based on the large language model, and the prompt word optimization model can generate personalized replies based on fine-tuning training. Thus, the reply text generated by the prompt word optimization model based on the raw image requirement information can be obtained; the reply text and the generated image are fed back to the target object.

[0208] That is, the prompt word optimization model is used to generate a reply text adapted to the raw image demand information according to the raw image demand information, and then the generated image is fed back to the target object to provide the target object with a more intuitive and rich feedback experience.

[0209] In the disclosed embodiment, for the prompt word optimization model, after constructing the training corpus, the SFT (Supervised Fine-Tuning) and DPO (Direct Preference Optimization) training methods can be used to fine-tune the base model to obtain the prompt word optimization model. The prompt word optimization model can provide an interface to the outside world for providing reasoning and prediction services. The main function of this service is to achieve text rewriting, obtain high-quality prompt words and personalized replies, and provide them to the downstream image generation system for use. In addition, during the online application process, prompt word examples and reply examples can be collected and added to the training corpus to iteratively optimize the prompt word optimization model. For example, for a new scenario, it is only necessary to construct the corresponding training corpus to optimize the prompt word optimization model.

[0210] In some embodiments, when the image generation requirement information includes historical conversation records, when the current round of conversation is a successor of multiple rounds of conversations, the text with potential risks in the historical conversation records can be deleted.

[0211] The text with potential risks may be the risk content identified by the aforementioned security risk identification model.

[0212] In the disclosed embodiment, texts with potential risks in historical conversation records are deleted to avoid public opinion risks and improve the content quality of the generated image.

[0213] In some embodiments, the raw image types described above can be fine-grainedly identified, and their fine-grained content is as follows: Figure 5 As shown, based on the first round of reference images, the target dialogue text of this round, and the historical dialogue situation, the intention is determined to be the first round of raw images 511 and the multiple rounds of successors 512. The first round of raw images 511 may include text-generated images 521 and image-generated images 522. The multiple rounds of successors 512 may include text-generated image successors 523 and image-generated image successors 524, and the text-generated image successors 523 include demand design classes 541, redrawing classes 542, and editing classes 543. The demand design class 541 includes a change demand type 551, and the change demand type 551 is to change the type of the generated image, which can be exemplified by changing the poster to a wide image. The editing class 543 includes modifying the aspect ratio 552, adding elements 553, deleting elements 554, replacing elements 555, and changing the style 556.

[0214] In addition, the image generation successor 524 is based on at least one of the first reference image, the second reference image, and the reference image generated in the previous round in the reference image 531 for reference, and the obtained intentions include the demand design class 544, the redrawing class 545, the editing class 546, and the creation similarity class 547. The demand design class, the redrawing class, and the editing class are similar to the above, and the creation similarity class 547 includes the reference image subject similarity 567, the reference image background similarity 568, and the reference image style similarity 569.

[0215] In summary, the architecture diagram of the prompt engineering system provided in the embodiment of the present disclosure is as follows: Figure 6 As shown, the entire system architecture is divided into multiple small models and three layers of cognitive capabilities. The three layers of cognitive capabilities may include cognitive correction, thinking planning, and basic knowledge. Among them, basic knowledge is used to obtain the result of intent recognition, and the basic knowledge includes determining the security intent 611 based on the security risk identification model 610, wherein the security intent is used to identify the risk information corresponding to the raw image demand information. Basic knowledge also includes multiple intentions obtained based on multimodal intention understanding 620, including reference image intention 621, multi-round intention 622 (including the first round of text-born image type, the first round of image-born image type, the text-born image multi-round subsequent type, and the image-born image multi-round subsequent type), size intention 623 (i.e., identifying the image size in the raw image intention information), type intention 624 (i.e., identifying the model type), style intention 625 (i.e., image style), subject intention 626 (i.e., target subject type), and editing intention 627 (i.e., image editing type in the raw image type). Among them, the reference image intention 621 (i.e., identifying the reference image information in the birth image intention information) can be used to understand the content of the first reference image provided by the target object, extract the reference subject type and the number of reference subjects included therein. The multi-round intention 622 is used to delete the risk content in the historical dialogue according to the risks identified by the security intention in the scenario of multi-round dialogue, and then identify the birth image type.

[0216] Based on the result of intention recognition, thinking planning is performed, wherein the thinking planning mainly includes retrieval enhancement planning 630 and prompt information alignment enhancement planning 640. Among them, the retrieval enhancement planning 630 includes whether to enhance retrieval 631, retrieval query planning 632, raw image model planning 633, and subject layout planning 634. Among them, whether to enhance retrieval 631 is used to determine whether the result of the intention recognition needs retrieval enhancement, the retrieval query planning 632 is used to generate a retrieval statement based on the raw image requirement information, the raw image model planning 633 is used to retrieve the second reference image based on the retrieval statement, and the subject layout planning 634 is used to determine the layout information of the target subject type. The prompt information alignment enhancement planning 640 includes prompt information structure alignment 641, prompt information subject semantic attribute enhancement 642, prompt information screen art enhancement 643, and personalized reply generation 644. Among them, the prompt information structure alignment 641 is used to align to the expression method that the target raw image model can adapt to, the prompt information subject semantic attribute enhancement 642 is used to semantically enhance the subject in the second reference image and / or the raw image requirement information, and the prompt information screen artistic enhancement 643 is used to enhance the content such as art tags. The personalized reply generation 644 is used to generate a personalized reply based on the raw image requirement information and the generated image.

[0217] Cognitive correction includes an online system intervention mechanism 650 and an offline evaluation reflection optimization mechanism 660. Among them, the online system intervention mechanism 650 includes a recall mechanism 651 and a sorting mechanism 652, wherein the recall mechanism 651 and the sorting mechanism 652 are used to optimize the recall and sorting mechanisms of the online system, such as quickly avoiding risks generated in real time. The offline evaluation reflection optimization mechanism 660 includes a multidimensional judgment-type evaluation task 661 and an evaluation feedback thinking chain optimization 662. Among them, the multidimensional judgment-type evaluation task 661 and the evaluation feedback thinking chain optimization 662 are used to reflect and optimize the model recognition results in order to output more accurate intention labels.

[0218] In some embodiments, the prompt word optimization model in the embodiment of the present disclosure may include a prompt information rewriting model based on the second reference graph and a general prompt information rewriting model. On this basis, the prompt engineering system is described from the perspective of the neural network model. Figure 7As shown: for the raw image requirement text 71 in the raw image requirement information, the security risk identification model 72 is called to perform security identification on it to obtain risk information. When the risk information indicates that there is no risk, the text requirement information is constructed, and the first intention identification sub-model 731 in the multimodal intention understanding model 73 is used to analyze the raw image type, image size and other information in the raw image intention information from the text requirement information. When the risk information indicates that there is a risk, it is adjusted. After adjustment, it is input into the first intention identification sub-model 731 in the multimodal intention understanding model 73 to obtain the raw image type, image size and other information in its raw image intention information. Among them, when the target object provides a first reference image 78, the image-based risk identification model 79 can be used to process the first reference image 78. When it is determined that there is no security risk in the first reference image 78, the multimodal large model 710 is used to identify the reference information of the first reference image as a supplement to the aforementioned raw image intention information. In addition, when the target object provides a first reference image 78 and a raw image requirement text 71, and when it is determined that neither of them presents a security risk, the multimodal large model 710 is used to identify the reference information of the first reference image, and the second intention recognition sub-model 732 is used to identify the intention information in the raw image requirement text, so as to obtain the raw image intention information based on the intention information in the raw image requirement text and the reference information of the first reference image.

[0219] Then, based on the raw image intention information obtained above, the target subject type in the raw image demand information is identified based on the distribution strategy model 74, and it is determined whether the raw image intention information needs to be enhanced. If it is determined that enhancement is required, the second reference image is retrieved. Then the second reference image and the text demand information are provided to the prompt information rewriting model 77 based on the second reference image for rewriting to obtain the target prompt word. If retrieval enhancement is not required, the general prompt information rewriting model 76 is used to rewrite the text demand information to obtain the target prompt word.

[0220] In addition, Figure 7 As shown, the distribution strategy model 74 can determine the model type of the target raw image model used to generate images in this round based on the raw image intent information and text requirement information, where the model types available for selection include a base model 741, a precise model 742, a highly generalized model 743, and an editing model 744.

[0221] The prompt engineering system can use the layout planning model to generate typesetting layout information 75 for the target subject type of the raw image requirement text 71. The typesetting layout information 75 can be used by the precise model 742 in the image generation system.

[0222] In the embodiment of the present disclosure, a prompt engineering system is also provided, such as Figure 8 As shown, at least one of the following services is included:

[0223] The first service 801 is used to process the raw image requirement information of the target object for generating an image, and determine the risk information corresponding to the raw image requirement information;

[0224] Among them, in addition to determining risk information, the first service can also determine the raw image intention information corresponding to the raw image demand information.

[0225] The second service 802 is used to determine the model type of the target raw image model adapted to the raw image requirement information;

[0226] The third service 803 is used to rewrite the target prompt words obtained by the raw image requirement information.

[0227] In the disclosed embodiment, multiple services are used to apply to different text-generated image scenarios, and the multimodal image-generated image demand information of the target object and multiple rounds of historical conversation information are provided as initialization input content to the prompt engineering system. The system outputs risk information, model type of the target image-generated image model, prompt words and other content. Based on the optimized information, the generated image is closer to the user's expectations, thereby improving the user experience.

[0228] In some embodiments, the second service is specifically configured to:

[0229] Determine the raw image intention information corresponding to the raw image requirement information;

[0230] Based on the raw image intent information, determine the model type of the target raw image model that is adapted to the raw image requirement information.

[0231] The model type of the target raw image model includes at least one of the following:

[0232] Base model: The base model supports common image generation tasks;

[0233] Accurate model, the accurate model is used to generate high-fidelity images;

[0234] Highly generalized models are used to generate images with fidelity that meets preset requirements and has artistic effects.

[0235] Editing model,The editing model is used to perform editing operations on known images.

[0236] In the disclosed embodiment, based on the second service, the raw image intent information corresponding to the raw image requirement information can be clearly understood, so as to adapt the model type of the target raw image model of the raw image requirement information so that the model can accurately generate images.

[0237] In some embodiments, the third service is also used to generate reply information in response to the image generation requirement information.

[0238] Among them, the reply information is personalized reply information, which can adapt to multiple differentiated scenarios.

[0239] In the disclosed embodiment, a third service is used to generate a reply text adapted to the raw image demand information according to the raw image demand information, and then the generated image is fed back to the target object to provide the target object with a more intuitive and rich feedback experience.

[0240] In some embodiments, a fourth service is also included for determining typesetting layout information of a target subject type in an image to be generated according to the image generation requirement information.

[0241] During implementation, since the raw image demand information may have requirements for the placement of the target subject, the target subject can be placed based on the requirements in the raw image demand information. When the raw image demand information does not contain typesetting layout information, and when it is determined that retrieval enhancement is required, the typesetting layout information of the target subject type can be determined based on the second reference image.

[0242] In the disclosed embodiment, using the typesetting layout information of the target subject type can make the generated image more in line with user expectations, thereby improving the user experience.

[0243] For example, take a multi-round conversation as an example. Fig. 9 As shown, Image901 (the first reference image in the raw image requirement information), Query902 (the target conversation text of the current round), and History903 (historical conversation situations) can be input into the prompt engineering system 904 to realize the input of the raw image requirement information into the unified multi-round raw image type understanding and safety understanding service 905 (also known as the first service).

[0244] The historical conversation information includes at least one of the following:

[0245] Prompt indicates prompt information, including rewritten Prompt generated by the conversation history;

[0246] sub_task_id indicates the ID of the raw image task;

[0247] refer_image_caption represents the description text of the second reference image;

[0248] risk_type indicates the risk level;

[0249] is_rag_trigger indicates whether to trigger retrieval enhancement;

[0250] is_user_refer_image indicates whether the target object has uploaded the first reference image;

[0251] search_query_list indicates the rewriting requirements after retrieval enhancement;

[0252] query_type_list indicates the subject type that is intended to be understood;

[0253] Styles refers to the style of the picture, such as realistic, artistic, etc.

[0254] Among them, the first service 905 calls the security risk identification model (that is, the security intention operator 909) to process the text in the raw image demand information, identify risks, and filter out risky content or rewrite risky content. The first service calls the multimodal intention understanding model (that is, the multi-round intention operator 910) to obtain the raw image type, which may include raw image type 1 and raw image type 2, where:

[0255] imggen_type1 means raw image type 1;

[0256] imggen_type2 indicates raw image type 2; raw image type 2 indicates the fine-grained raw image intent of raw image type 1, such as the specific operations required in the edit image type, such as deleting local elements and modifying elements;

[0257] refer_dialogue_id indicates the reference round ID. The first round of dialogue is used as the reference round ID for a raw image task.

[0258] edit_elements represents editing elements, that is, which subjects in the reference image are edited if a reference image exists;

[0259] risk_level indicates the risk level;

[0260] risk_cls_type indicates the risk type.

[0261] The above-obtained information is input into the unified multi-round iRAG trigger check and understanding service 906 (ie, the above-mentioned second service). The second service relies on the iRAG trigger operator 911, and the following content is obtained:

[0262] is_user_refer_imag: whether there is a first reference image;

[0263] search_query_list: retrieve the enhanced search statement list;

[0264] query_type_list intends to understand the target subject type;

[0265] styles indicates the screen style.

[0266] Then, the universal multi-round Prompt rewriting generation service 907 (the third service) is input, and the third service calls the universal multi-round Prompt rewriting operator 912 (i.e., the prompt word optimization model) to rewrite the raw image requirement information to obtain the target prompt word and the reply information for the raw image requirement information.

[0267] In addition, before the unified multi-round iRAG trigger inspection and understanding service 906 and the universal multi-round Prompt rewrite generation service 907 call the corresponding operator, History 903 can also be processed based on the History preprocessing strategy 908, exemplarily including deleting risky content in History 903.

[0268] In the embodiment of the present disclosure, a security management strategy is also provided, such as Fig.10 As shown, it can be divided into three stages: training stage, generation stage, and propagation stage. Among them, the training stage includes risk element cleaning of training data set 1001, which is used to collect training data set and clean the risk elements therein. The generation stage includes target object input text / image audit strategy 1002, image generation audit strategy 1003, prompt information security optimization strategy 1004, image retrieval enhancement strategy 1005, and image and text intervention strategy based on knowledge strategy and text semantics 1006. Among them, target object input text / image audit strategy 1002 is specifically used to audit the text / image input by the target object, and image generation audit strategy 1003 is used to audit the generated image. Prompt information security optimization strategy 1004 is used to generate prompt information using the rewriting model after identifying risks based on the security risk identification model. Image retrieval enhancement strategy 1005 is used to determine whether the original image intention information needs to be retrieved and enhanced. The image and text intervention strategy 1006 based on knowledge strategy and text semantics is a supplement to the security risk identification capability. Because the security risk identification model is trained based on training samples, as new vocabulary is generated, the recognition ability of the model may need to be strengthened. If the model is adjusted and optimized by accumulating training samples, it will take a long time, and using coding for risk identification will also increase development costs. Therefore, in the embodiment of the present disclosure, the ability of the large model to understand pictures and texts can be utilized, and the large model can be configured with the description of the information that needs to be filtered, so that risks can be identified instantly through the understanding ability of the large model. In the event of an emergency in the intervention system, no code is required, and intervention can be carried out directly to avoid large-scale public opinion problems. A manual review mechanism 1007 can be introduced in the dissemination stage to ensure that the image content meets relevant requirements. In summary, based on multiple protection mechanisms, the security of the entire prompt engineering system is improved.

[0269] Based on the same technical concept, the present disclosure also provides an information processing system 1100 for image generation. Fig.11 As shown, including:

[0270] The prompt engineering system 1101 is used to process the raw image requirement information of the target object, so as to convert the raw image requirement information into raw image parameters for use by the image generation system based on the prompt engineering system;

[0271] The raw image parameters include at least one of the following:

[0272] Risk information corresponding to the raw image demand information;

[0273] The raw image intention information corresponding to the raw image requirement information;

[0274] A model type of a target raw image model adapted to the raw image requirement information;

[0275] rewriting the raw image requirement information to obtain a target prompt word;

[0276] Reply information to the raw image requirement information;

[0277] The typesetting layout information of the target subject type in the image to be generated by the image generation requirement information.

[0278] The image generation system 1102 is used to generate a target image based on the image generation parameters.

[0279] In the case where the raw image parameters include risk information corresponding to the raw image requirement information, the image generation system determines whether the raw image requirement information has risks based on the risk information corresponding to the raw image requirement information. If there is no risk, subsequent raw image processing is performed; if there is a risk, risk removal processing is performed. The specific method for removing risk information has been explained in the previous text and will not be repeated here.

[0280] When the image generation parameters include the image generation intention information corresponding to the image generation requirement information, the image generation system can obtain the image generation intention clearly stated in the image generation requirement information.

[0281] For example, when the image generation parameters include the model type of the target image generation model of the image generation requirement information, the image generation system calls the model of the corresponding type as the target image generation model for generating the target image.

[0282] In the case where the raw image parameters include a target cue word, the image generation system may input the target cue word into the target raw image model so that the target raw image model can generate a target image according to the target cue word.

[0283] When the raw image information includes the reply information, the image generation system needs to output the reply information to the target object together with the target image generated by the target raw image model.

[0284] In the case where the raw image parameters include the typesetting layout information of the target subject type, the image generation system needs to provide the target raw image model with the typesetting layout information to restrict the typesetting layout of the generated target image.

[0285] In the disclosed embodiment, the use of the prompt engineering system can make the raw image parameters clearer, improve the accuracy of the target image obtained by the image generation system based on the raw image parameters, and be closer to user expectations.

[0286] Based on the same technical concept, the present disclosure also provides an information processing device 1200 for image generation. Fig.12 As shown, including:

[0287] An acquisition unit 1201 is used to acquire image generation requirement information of a target object for generating an image;

[0288] The processing unit 1202 is used to input the raw image requirement information into the prompt engineering system, so as to convert the raw image requirement information into raw image parameters for use by the image generation system based on the prompt engineering system.

[0289] In some embodiments, the image generation parameters include at least one of the following:

[0290] Risk information corresponding to the raw image demand information;

[0291] The raw image intention information corresponding to the raw image requirement information;

[0292] A model type of a target raw image model adapted to the raw image requirement information;

[0293] rewriting the raw image requirement information to obtain a target prompt word;

[0294] Reply information to the raw image requirement information;

[0295] The typesetting layout information of the target subject type in the image to be generated by the image generation requirement information.

[0296] In some embodiments, the processing unit includes:

[0297] A risk identification subunit, used for inputting the requirement text in the raw image requirement information into a security risk identification model to identify the risk information corresponding to the raw image requirement information;

[0298] The risk information includes risk type and / or risk level.

[0299] In some embodiments, the processing unit includes:

[0300] The intention acquisition subunit is used to identify the raw image requirement information based on the multimodal intention understanding model and obtain the raw image intention information.

[0301] In some embodiments, the intention acquisition subunit is specifically used to:

[0302] Constructing text comprehension prompt words based on the requirement text and target prompt template in the raw image requirement information;

[0303] Inputting the text understanding prompt word into the first intention recognition sub-model in the multimodal intention understanding model to obtain at least one of the following information in the raw image intention information: raw image type, image size, and image style;

[0304] The raw image type is a type in a preset type set.

[0305] In some embodiments, the preset type set includes at least one of the following types:

[0306] First-round text-generated image type, first-round image-generated image type, text-generated image multiple-round subsequent type, image-generated image multiple-round subsequent type, image editing type.

[0307] In some embodiments, the system further includes a first training unit for:

[0308] Based on the large language model and target prompt template, a training sample set with different raw image intent labels is distilled;

[0309] Classifying the training sample set based on the raw image intent labels to obtain sample subsets corresponding to the raw image intent labels;

[0310] Based on the sample subsets corresponding to the respective raw image intent labels, supervised training is performed on the pre-trained initial model to obtain the first intent recognition sub-model.

[0311] In some embodiments, the first training unit is specifically used to: obtain an initial sample set;

[0312] Based on the sequence labeling task, a plurality of intermediate samples are screened out from the initial sample set to obtain an intermediate sample set;

[0313] Constructing a sample prompt word based on the target prompt template and the intermediate sample set;

[0314] A large language model is used to generate a raw image intent label for each sample in the intermediate sample set to obtain the training sample set.

[0315] In some embodiments, the device further includes an adjustment unit for:

[0316] In the case of adopting the prompt template to be adjusted, predicting the initial intent label of the target sample set based on the first intent recognition sub-model;

[0317] Using the large language model to determine whether the initial intent label meets multiple preset rule requirements in a rule set;

[0318] In the case that any preset rule requirement is not met, the prompt template to be adjusted is optimized based on the large language model and any preset rule requirement is used as a benchmark to obtain the target prompt template.

[0319] In some embodiments, the intention acquisition subunit is further used to:

[0320] In a case where the raw image requirement information includes a first reference image provided by the target object, obtaining reference image information of the first reference image based on a multimodal large model in the multimodal intent understanding model; the raw image intent information includes the reference image information;

[0321] The reference image information includes at least one of the following information: a first reference subject type, a first reference subject number, and text description information of the first reference image.

[0322] In some embodiments, the intention acquisition subunit is further used to:

[0323] The requirement text in the raw image requirement information is input into the second intent recognition sub-model in the multimodal intent understanding model to obtain the target subject type indicated by the requirement text; the raw image intent information includes the target subject type.

[0324] In some embodiments, the processing unit includes:

[0325] The model determination subunit is used to process the image generation intention information based on the distribution strategy model, and determine the model type of the target image generation model in the image generation system for generating an image for the image generation requirement information.

[0326] In some embodiments, the model type of the target image generation model includes at least one of the following:

[0327] A base model, wherein the base model supports general image generation tasks;

[0328] An accurate model, wherein the accurate model is used to generate a high-fidelity image;

[0329] A highly generalized model, wherein the highly generalized model is used to generate images with fidelity that meets preset requirements and has artistic effects;

[0330] An editing model is used to perform editing operations on a known image.

[0331] In some embodiments, the method further includes a determining unit configured to:

[0332] In the case where the target subject type in the image to be generated by the raw image requirement information is a preset type, the model type of the target raw image model is a security model;

[0333] The security model is used to generate images that avoid public opinion risks.

[0334] In some embodiments, the processing unit includes:

[0335] A processing subunit, used for processing the image generation demand information of the target object based on the distribution strategy model to obtain a processing result;

[0336] A generating subunit, configured to obtain a search statement generated by the distribution strategy model from the processing result when the processing result indicates that search enhancement is required;

[0337] A retrieval subunit, configured to retrieve a second reference image of the image generation requirement information based on the retrieval statement;

[0338] The rewriting subunit is used to rewrite the second reference image and the raw image requirement information based on the prompt word optimization model to obtain the target prompt word adapted to the target raw image model in the image generation system, and generate the reply information.

[0339] In some embodiments, the rewriting subunit is specifically used for:

[0340] Acquire a target dialogue text of a current round of dialogue of the target object from the raw image requirement information;

[0341] Constructing information to be rewritten based on the target dialogue text, the historical dialogue situation in the raw image requirement information, and the second reference image;

[0342] The information to be rewritten is input into the prompt word optimization model to obtain the target prompt word.

[0343] In some embodiments, when the image generation requirement information includes historical conversation records, a deleting unit is further included, which is used to:

[0344] When the historical conversation situation is that the current round of conversation is a successor of multiple rounds of conversations, the text with potential risks in the historical conversation record is deleted.

[0345] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0346] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0347] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0348] Fig.13 A schematic block diagram of an example electronic device 1300 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0349] like Fig.13 As shown, the device 1300 includes a computing unit 1301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1302 or a computer program loaded from a storage unit 1308 into a random access memory (RAM) 1303. In the RAM 1303, various programs and data required for the operation of the device 1300 can also be stored. The computing unit 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0350] A number of components in the device 1300 are connected to the I / O interface 1305, including: an input unit 1306, such as a keyboard, a mouse, etc.; an output unit 1307, such as various types of displays, speakers, etc.; a storage unit 1308, such as a disk, an optical disk, etc.; and a communication unit 1309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1309 allows the device 1300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0351] The computing unit 1301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1301 performs the various methods and processes described above, such as information processing methods for image generation. For example, in some embodiments, the information processing method for image generation may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1308. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1300 via the ROM 1302 and / or the communication unit 1309. When the computer program is loaded into the RAM 1303 and executed by the computing unit 1301, one or more steps of the information processing method for image generation described above may be performed. Alternatively, in other embodiments, the computing unit 1301 may be configured to execute the information processing method for image generation in any other appropriate manner (eg, by means of firmware).

[0352] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0353] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0354] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0355] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0356] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0357] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0358] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0359] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An information processing method for image generation, comprising: Obtaining the raw image requirement information of the target object for generating an image; The raw image requirement information is input into a prompt engineering system, so as to convert the raw image requirement information into raw image parameters for use by an image generation system based on the prompt engineering system.

2. The method according to claim 1, wherein: The image generation parameters include at least one of the following: Risk information corresponding to the raw image demand information; The raw image intention information corresponding to the raw image requirement information; A model type of a target raw image model adapted to the raw image requirement information; rewriting the raw image requirement information to obtain a target prompt word; Reply information to the raw image requirement information; The typesetting layout information of the target subject type in the image to be generated by the image generation requirement information.

3. The method according to claim 2, wherein: The risk information is determined by, among other things: Inputting the requirement text in the raw image requirement information into a security risk identification model to identify the risk information corresponding to the raw image requirement information; The risk information includes risk type and / or risk level.

4. The method according to claim 2, wherein: The image generation intention information is determined by the following methods, including: The raw image requirement information is identified based on a multimodal intent understanding model to obtain the raw image intent information.

5. The method according to claim 4, wherein: The identifying the raw image requirement information based on the multimodal intention understanding model to obtain the raw image intention information includes: Constructing text comprehension prompt words based on the requirement text and target prompt template in the raw image requirement information; Inputting the text understanding prompt word into the first intention recognition sub-model in the multimodal intention understanding model to obtain at least one of the following information in the raw image intention information: raw image type, image size, and image style; The raw image type is a type in a preset type set.

6. The method according to claim 5, wherein the preset type set comprises at least one of the following types: First-round text-generated image type, first-round image-generated image type, text-generated image multiple-round subsequent type, image-generated image multiple-round subsequent type, image editing type.

7. The method according to claim 5, wherein: The first intent recognition sub-model is obtained by the following training method, including: Based on the large language model, we distilled the training sample sets with different raw image intent labels; Classifying the training sample set based on the raw image intent labels to obtain sample subsets corresponding to the raw image intent labels; Based on the sample subsets corresponding to the respective raw image intent labels, supervised training is performed on the pre-trained initial model to obtain the first intent recognition sub-model.

8. The method according to claim 7, wherein: The training sample set of different raw image intent labels distilled based on the large language model includes: Get an initial sample set; Based on the sequence labeling task, a plurality of intermediate samples are screened out from the initial sample set to obtain an intermediate sample set; A large language model is used to generate a raw image intent label for each sample in the intermediate sample set to obtain the training sample set.

9. The method according to claim 7, further comprising: In the case of adopting the prompt template to be adjusted, predicting the initial intent label of the target sample set based on the first intent recognition sub-model; Using the large language model to determine whether the initial intent label meets multiple preset rule requirements in a rule set; In the case that any preset rule requirement is not met, the prompt template to be adjusted is optimized based on the large language model and any preset rule requirement is used as a benchmark to obtain the target prompt template.

10. The method according to claim 4, wherein: The identifying the raw image requirement information based on the multimodal intention understanding model to obtain the raw image intention information includes: In a case where the raw image requirement information includes a first reference image provided by the target object, obtaining reference image information of the first reference image based on a multimodal large model in the multimodal intent understanding model; the raw image intent information includes the reference image information; The reference image information includes at least one of the following information: a first reference subject type, a first reference subject number, and text description information of the first reference image.

11. The method according to claim 4, wherein: The identifying the raw image requirement information based on the multimodal intention understanding model to obtain the raw image intention information includes: The requirement text in the raw image requirement information is input into the second intent recognition sub-model in the multimodal intent understanding model to obtain the target subject type indicated by the requirement text; the raw image intent information includes the target subject type.

12. The method according to claim 2, wherein: The model type of the target image generation model is determined by: The image generation intention information is processed based on the distribution strategy model to determine the model type of the target image generation model in the image generation system that generates the image for the image generation requirement information.

13. The method according to claim 12, wherein: The model type of the target image generation model includes at least one of the following: A base model, wherein the base model supports general image generation tasks; An accurate model for generating high-fidelity images; A highly generalized model, wherein the highly generalized model is used to generate images with fidelity that meets preset requirements and has artistic effects; An editing model is used to perform editing operations on a known image.

14. The method according to claim 13, further comprising: In the case where the target subject type in the image to be generated by the raw image requirement information is a preset type, the model type of the target raw image model is a security model; The security model is used to generate images that avoid public opinion risks.

15. The method according to claim 2, wherein: The target prompt word and the reply information of the raw image requirement information are determined by the following method, including: Processing the image generation demand information of the target object based on the distribution strategy model to obtain a processing result; When the processing result indicates that search enhancement is needed, obtaining a search statement generated by the distribution strategy model from the processing result; Retrieving a second reference image for the raw image requirement information based on the search statement; Based on the prompt word optimization model, the second reference image and the raw image requirement information are rewritten to obtain target prompt words adapted to the target raw image model in the image generation system, and generate the reply information.

16. The method according to claim 15, wherein: The rewriting operation of the second reference image and the raw image requirement information based on the prompt word optimization model to obtain the target prompt word adapted to the target raw image model in the image generation system includes: Acquire a target dialogue text of a current round of dialogue of the target object from the raw image requirement information; Constructing information to be rewritten based on the target dialogue text, the historical dialogue situation in the raw image requirement information, and the second reference image; The information to be rewritten is input into the prompt word optimization model to obtain the target prompt word.

17. The method according to claim 16, wherein: When the image generation requirement information includes historical conversation records, it also includes: When the current round of dialogue is a successor of multiple rounds of dialogue, the text with potential risks in the historical dialogue record is deleted.

18. An information processing device for image generation, comprising: An acquisition unit, used for acquiring image generation requirement information of a target object for generating an image; The processing unit is used to input the raw image requirement information into the prompt engineering system, so as to convert the raw image requirement information into raw image parameters for use by the image generation system based on the prompt engineering system.

19. The device according to claim 18, wherein: The image generation parameters include at least one of the following: Risk information corresponding to the raw image demand information; The raw image intention information corresponding to the raw image requirement information; A model type of a target raw image model adapted to the raw image requirement information; rewriting the raw image requirement information to obtain a target prompt word; Reply information to the raw image requirement information; The typesetting layout information of the target subject type in the image to be generated by the image generation requirement information.

20. The device according to claim 19, wherein The processing unit comprises: A risk identification subunit, used for inputting the requirement text in the raw image requirement information into a security risk identification model to identify the risk information corresponding to the raw image requirement information; The risk information includes risk type and / or risk level.

21. The device according to claim 19, wherein The processing unit comprises: The intention acquisition subunit is used to identify the raw image requirement information based on the multimodal intention understanding model and obtain the raw image intention information.

22. The device according to claim 21, wherein The intention acquisition subunit is specifically used for: Constructing text comprehension prompt words based on the requirement text and target prompt template in the raw image requirement information; Inputting the text understanding prompt word into the first intention recognition sub-model in the multimodal intention understanding model to obtain at least one of the following information in the raw image intention information: raw image type, image size, and image style; The raw image type is a type in a preset type set.

23. The apparatus according to claim 22, wherein the preset type set comprises at least one of the following types: First-round text-generated image type, first-round image-generated image type, text-generated image multiple-round subsequent type, image-generated image multiple-round subsequent type, image editing type.

24. The apparatus according to claim 22, further comprising a first training unit, configured to: Based on the large language model, we distilled the training sample sets with different raw image intent labels; Classifying the training sample set based on the raw image intent labels to obtain sample subsets corresponding to the raw image intent labels; Based on the sample subsets corresponding to the respective raw image intent labels, supervised training is performed on the pre-trained initial model to obtain the first intent recognition sub-model.

25. The device according to claim 24, wherein: The first training unit is specifically used for: Get an initial sample set; Based on the sequence labeling task, a plurality of intermediate samples are screened out from the initial sample set to obtain an intermediate sample set; Constructing a sample prompt word based on the target prompt template and the intermediate sample set; A large language model is used to generate a raw image intent label for each sample in the intermediate sample set to obtain the training sample set.

26. The device according to claim 24, further comprising an adjustment unit, configured to: In the case of adopting the prompt template to be adjusted, predicting the initial intent label of the target sample set based on the first intent recognition sub-model; Using the large language model to determine whether the initial intent label meets multiple preset rule requirements in a rule set; In the case that any preset rule requirement is not met, the prompt template to be adjusted is optimized based on the large language model and any preset rule requirement is used as a benchmark to obtain the target prompt template.

27. The device according to claim 21, wherein The intention acquisition subunit is also used to: In a case where the raw image requirement information includes a first reference image provided by the target object, obtaining reference image information of the first reference image based on a multimodal large model in the multimodal intent understanding model; The image generation intention information includes the reference image information; The reference image information includes at least one of the following information: a first reference subject type, a first reference subject number, and text description information of the first reference image.

28. The device according to claim 21, wherein The intention acquisition subunit is also used to: The requirement text in the raw image requirement information is input into the second intent recognition sub-model in the multimodal intent understanding model to obtain the target subject type indicated by the requirement text; the raw image intent information includes the target subject type.

29. The device according to claim 19, wherein The processing unit comprises: The model determination subunit is used to process the image generation intention information based on the distribution strategy model, and determine the model type of the target image generation model in the image generation system for generating an image for the image generation requirement information.

30. The device according to claim 29, wherein: The model type of the target image generation model includes at least one of the following: A base model, wherein the base model supports general image generation tasks; An accurate model for generating high-fidelity images; A highly generalized model, wherein the highly generalized model is used to generate images with fidelity that meets preset requirements and has artistic effects; An editing model is used to perform editing operations on a known image.

31. The apparatus according to claim 30, further comprising a determining unit, configured to: In the case where the target subject type in the image to be generated by the raw image requirement information is a preset type, the model type of the target raw image model is a security model; in, The security model is used to generate images that avoid public opinion risks.

32. The apparatus of claim 19, wherein: The processing unit comprises: A processing subunit, used for processing the image generation demand information of the target object based on the distribution strategy model to obtain a processing result; A generating subunit, configured to obtain a search statement generated by the distribution strategy model from the processing result when the processing result indicates that search enhancement is required; A retrieval subunit, configured to retrieve a second reference image of the image generation requirement information based on the retrieval statement; The rewriting subunit is used to rewrite the second reference image and the raw image requirement information based on the prompt word optimization model, obtain the target prompt word adapted to the target raw image model in the image generation system, and generate the reply information.

33. The device according to claim 32, wherein: The rewriting subunit is specifically used for: Acquire a target dialogue text of a current round of dialogue of the target object from the raw image requirement information; Constructing information to be rewritten based on the target dialogue text, the historical dialogue situation in the raw image requirement information, and the second reference image; The information to be rewritten is input into the prompt word optimization model to obtain the target prompt word.

34. The device according to claim 33, wherein: In the case where the image generation requirement information includes historical conversation records, a deleting unit is further included, which is used to: When the historical conversation situation is that the current round of conversation is a successor of multiple rounds of conversations, the text with potential risks in the historical conversation record is deleted.

35. An information processing system for image generation, comprising: A prompt engineering system, used for processing raw image requirement information of a target object, so as to convert the raw image requirement information into raw image parameters for use by an image generation system based on the prompt engineering system; The image generation system is used to generate a target image based on the image generation parameters.

36. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 17.

37. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-17.

38. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 17.

Images