Large model-based text-to-image method, device, equipment and storage medium

Abstract

The disclosure provides a large model-based text-to-image method, device, equipment and storage medium, relating to the technical field of artificial intelligence, especially to the fields of deep learning, natural language processing, etc. The method comprises: using at least two feature encoders in a large model to encode features of a target text respectively to obtain at least two target text features; matching the target text with preset candidate attributes to obtain a target attribute in the target text, and obtaining a test result of the target attribute from test results of the candidate attributes; determining a target fusion mode according to the test result of the target attribute, and fusing the at least two target text features by using the target fusion mode to obtain a target fusion feature; and generating a target image according to the target fusion feature.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of computers, especially to the field of artificial intelligence, and relates to the technical fields of deep learning, natural language processing, etc. Specifically, it relates to a text-to-image method and device based on a large model, equipment and a storage medium. BACKGROUND

[0002] AI (Artificial Intelligence) painting, as one of the important application fields of artificial intelligence, has made significant breakthroughs in recent years. It can generate images of various styles according to user input or prompts, providing a powerful tool for artists, designers and creators, and bringing new possibilities to the digital creative field. Therefore, users are no longer limited to existing gallery resources in the process of searching for images, but can create new image content and image styles by generating images using large models according to user needs and creativity.

[0003] Large models refer to deep learning models with parameter sizes exceeding hundreds of millions or even tens of billions. These models have a large number of parameters and complex network structures, capable of capturing complex patterns and distributions, supporting multi-task, multi-modal (such as text, image, audio) learning, and exhibiting strong generalization and emergence capabilities. With the development of AIGC (Artificial Intelligence Generated Content) technology and user search requirements, an AI native product, "Draw a picture", is developed to intelligently generate creative images of different scenes and styles, providing a new image "search" method for users and realizing a generational change in the field of image search. SUMMARY

[0004] The present disclosure provides a text-to-image method and device based on a large model, equipment and a storage medium.

[0005] According to an aspect of the present disclosure, a text-to-image method based on a large model is provided, comprising:

[0006] At least two feature encoders in the large model are used to encode the target text to obtain at least two target text features;

[0007] The target text and the preset candidate attributes are matched to obtain a target attribute in the target text, and the test result of the target attribute is obtained from the test results of the candidate attributes;

[0008] determine a target fusion manner according to the test result of the target attribute, and fuse the at least two target text features by using the target fusion manner to obtain a target fusion feature;

[0009] generate a target image according to the target fusion feature.

[0010] According to an aspect of the present disclosure, a text-to-image device based on a large model is provided, comprising:

[0011] a feature encoding module configured to encode target text by using at least two feature encoders in a large model to obtain at least two target text features;

[0012] a target attribute module configured to match the target text with each candidate attribute to obtain a target attribute in the target text, and obtain a test result of the target attribute from test results of each candidate attribute;

[0013] a feature fusion module configured to determine a target fusion manner according to the test result of the target attribute, and fuse the at least two target text features by using the target fusion manner to obtain a target fusion feature;

[0014] an image generation module configured to generate a target image according to the target fusion feature.

[0015] According to another aspect of the present disclosure, an electronic device is provided, comprising:

[0016] at least one processor; and

[0017] a memory connected with the at least one processor in communication; wherein,

[0018] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method provided by any of the embodiments of the present disclosure.

[0019] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to perform the method provided by any of the embodiments of the present disclosure.

[0020] According to still another aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method provided by any of the embodiments of the present disclosure.

[0021] It should be appreciated that the content described in this section is not intended to identify key or critical features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1a is a flowchart of a text-to-image method based on a large model according to an embodiment of the present disclosure;

[0023] Figure 1b is a schematic diagram of text feature processing according to an embodiment of the present disclosure;

[0024] Figure 1c is a schematic diagram of a target fusion feature injection method according to an embodiment of the present disclosure;

[0025] Figure 1d is a schematic diagram of another target fusion feature injection method according to an embodiment of the present disclosure;

[0026] Figure 2a is a flowchart of another text-to-image method based on a large model according to an embodiment of the present disclosure;

[0027] Figure 2b is a feature fusion schematic diagram of feature vector dimensions according to an embodiment of the present disclosure;

[0028] Figure 2c is a feature fusion schematic diagram of word dimensions according to an embodiment of the present disclosure;

[0029] Figure 3 is a flowchart of still another text-to-image method based on a large model according to an embodiment of the present disclosure;

[0030] Figure 4 is a structural schematic diagram of a text-to-image device based on a large model according to an embodiment of the present disclosure;

[0031] Figure 5 is a block diagram of an electronic device for implementing a text-to-image method based on a large model according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] Figure 1a is a flowchart of a text-to-image method based on a large model according to an embodiment of the present disclosure. The method is applicable to the case of generating various style images according to user input or prompts. The method can be performed by a text-to-image device based on a large model, which can be implemented in software and / or hardware, and can be integrated into an electronic device. As Figure 1a shown, the text-to-image method based on a large model of the present embodiment can include:

[0033] S101, using at least two feature encoders in the large model to respectively encode features of the target text, to obtain at least two target text features;

[0034] S102, matching the target text with each candidate attribute to obtain a target attribute in the target text, and obtaining a test result of the target attribute from test results of each candidate attribute;

[0035] S103, determining a target fusion mode according to the test result of the target attribute, and fusing the at least two target text features using the target fusion mode to obtain a target fusion feature;

[0036] S104, generating a target image according to the target fusion feature.

[0037] In the embodiments of the present disclosure, the text-to-image large model can include at least two feature encoders, a feature fusion module, and a feature injection module. The feature encoders are used to encode features of a target text input by a user to obtain target text features. The feature fusion module is used to fuse the target text features output by various feature encoders to obtain a target fusion feature. The feature injection module is used to inject the target fusion feature into an image generator to inject the target text as a text condition into the image generator, thereby improving the consistency between the generated image and the target text.

[0038] The feature encoders can be divided into different types according to their training methods. Specifically, they can include CLIP (Contrastive Language-Image Pre-training) series feature encoders, LLM (Large Language Model) series feature encoders, and MLLM (Multimodal Large Language Model) series feature encoders. The CLIP series aligns a large amount of text-image pair data through pre-training, so the feature vector of the text encoded by the CLIP series feature encoder can quickly align the image features. However, the control ability of the CLIP series on fine-grained attributes is weak, that is, the CLIP series has prior knowledge of images, but the semantic expression of fine-grained attributes is slightly weak. The LLM series has undergone a large amount of text pre-training and has strong text representation ability, which can distinguish fine-grained attributes such as color, size, shape, etc. The disadvantage is that it takes longer time to align the image features because the LLM series does not undergo text-image training. That is, the LLM series has semantic expression of fine-grained attributes. The MLLM series combines the advantages of the CLIP series and the LLM series, which can represent text features of fine-grained attributes and align image features. However, the model parameter amount is large.

[0039] In an optional implementation, the at least two feature encoders include a feature encoder of the CLIP series and a feature encoder of the LLM series.

[0040] In the embodiments of the present disclosure, the text-to-image large model can use the feature encoders of the CLIP series and the feature encoders of the LLM series, so that the text-to-image large model can take into account the prior knowledge of the image and the semantic expression ability for fine-grained attributes. Figure 1b The target text input by the user in the AI drawing is obtained, the target text is input into the feature encoder of the CLIP series in the text-to-image large model to obtain corresponding target text features, and the target text is input into the feature encoder of the LLM series in the text-to-image large model to obtain corresponding target text features, that is, two kinds of target text features are obtained. In the text-to-image scene in Chinese, a feature encoder in Chinese can be optionally used.

[0041] The candidate attributes are attributes commonly used and focused on in the AI drawing scene, and one candidate attribute can have multiple values. Exemplarily, the candidate attributes can include quantity, size, color, orientation, shape, fatness and height, action, expression, beauty, weather, light, style, age, gender, etc. The values of the orientation attribute can be up, down, left, right, inside or outside, etc., the values of the action attribute can be sitting, lying, standing or push-up, etc., and the values of the style attribute can be stick figure, children's painting, sketch or two-dimensional, etc. The image generation of different candidate attributes can be different; in the case of generating an image using a text containing multiple candidate attributes, some candidate attributes can be missing in the generated image, and although some candidate attributes can exist in the generated image, the image quality of the candidate attributes can not meet the standard. By generating an image using a text containing candidate attributes in advance and testing the generated image for the candidate attributes, a test result of the candidate attributes is obtained. Taking the input text "please generate a red landscape painting with a lying cat in the lower left corner" as an example, if the generated image is not a landscape painting, the test result of the style attribute is missing; if the cat in the generated image is lying, but the action of lying is very stiff and unnatural, the test result of the action attribute is that the quality does not meet the standard.

[0042] Exemplarily, the target text can be matched with each candidate attribute, and the candidate attribute that matches successfully can be taken as a target attribute in the target text, and the test result of the target attribute is obtained from the test results of each candidate attribute. Figure 1bThe target fusion manner can be selected from various candidate fusion manners according to the test result of the target attribute, and the candidate fusion manner can be splicing, addition, maximum pooling or averaging. Since the image generation conditions of different candidate attributes can be different, each target attribute in the target text is obtained by matching the target text with each candidate attribute, and the target fusion manner is determined for the target text in combination with the test result of each target attribute, so that the characteristics of each target attribute in image generation can be fully reflected, and meanwhile the generated target image can be ensured to be highly consistent with the target text. Through the fine fusion strategy, misinterpretation or deviation in the image generation process can be avoided, so as to improve the expression ability of the target image to the target attribute, that is, the out-image correlation is improved.

[0043] In addition, with reference to Figure 1b The at least two target text features can also be fused by a model, for example, the at least two target text features and the test result of the target attribute are input into a feature fusion module based on a Transformer network, the target feature fusion manner is determined by the feature fusion module in combination with the test result of the target attribute, and the at least two target text features are fused by the target feature fusion manner to obtain a target fusion feature.

[0044] Furthermore, the at least two target text features are fused by the target fusion manner to obtain the target fusion feature; and the target fusion feature is injected into an image generator, so as to control the effect of out-image, and a target image is obtained. With reference to Figure 1c In the case where the image generator adopts a DiT (Diffusion Transformer) series network architecture, the target fusion feature can be input into the image generator as one of input conditions of image generation together with a latent variable to obtain the target image; with reference to Figure 1d In the case where the image generator adopts a Unet (U-shaped network) series network architecture, the target fusion feature can be input into the image generator as an external condition through CrossAttention during the denoising process of the latent variable to obtain the target image.

[0045] The technical scheme provided by the embodiments of the present disclosure can avoid misinterpretation or deviation in the image generation process by adopting at least two feature encoders to respectively encode the target text to obtain at least two target text features, matching the target text with each candidate attribute to obtain each target attribute in the target text, and determining the target fusion manner for the target text in combination with the test result of each target attribute, so as to improve the expression ability of the target image to the target attribute, that is, the out-image correlation is improved.

[0046] In an optional implementation, the test result of the candidate attribute includes an existence test result and a quality test result; the existence test result is used to represent whether the candidate attribute is missing in the generated image, and the quality test result is used to represent whether the generated image meets the quality standard on the candidate attribute.

[0047] The existence test result of the candidate attribute is used to represent whether the candidate attribute is missing in the generated image, and the quality test result of the candidate attribute is used to represent whether the generated image meets the quality standard on the candidate attribute. Through the fine analysis of the existence and quality of the candidate attribute, a more accurate test result of the candidate attribute is obtained, the accuracy of feature fusion is significantly enhanced, and the relevance between the image and the text is effectively improved.

[0048] Figure 2a is a flowchart of another text-to-image method based on a large model provided according to an embodiment of the present disclosure. Referring to Figure 2a On the basis of the above-mentioned embodiments, the text-to-image method based on a large model of the present embodiment can include:

[0049] S201, at least two feature encoders in a large model are used to respectively encode features of a target text, to obtain at least two target text features;

[0050] S202, the target text and each candidate attribute are matched to obtain target attributes in the target text, and test results of the target attributes are obtained from test results of each candidate attribute;

[0051] If the existence test result of at least one target attribute is missing, and the quality test result of each target attribute meets the quality standard, S203 is continued to be executed; if the existence test result of each target attribute is present, and the quality test result of at least one target attribute does not meet the quality standard, S204 is executed;

[0052] S203, at least two target text features are fused by using a maximum value pooling method to obtain the target fusion feature;

[0053] S204, at least two target text features are fused by using an average value method to obtain the target fusion feature;

[0054] S205, a target image is generated according to the target fusion feature.

[0055] In the embodiments of the present disclosure, a target text input by a user in AI (Artificial Intelligence) drawing can be acquired, at least two target text features are obtained by inputting the target text into at least two feature encoders in a text-to-image model respectively, the target text is matched with preset candidate attributes, a candidate attribute matched successfully is taken as a target attribute in the target text, and a test result of the target attribute is obtained from test results of the candidate attributes.

[0056] In the test results of the target attributes, if the existence test result of at least one target attribute is missing, and the quality test result of each target attribute meets the quality standard, maximum pooling is taken as the target fusion manner, that is, the maximum pooling manner is adopted to fuse the at least two target text features to obtain a target fusion feature. If the existence test result of any target attribute is missing, the at least two target text features are fused by adopting the maximum pooling, which can avoid the target attribute being offset or submerged by adding or averaging, thereby improving the image generation quality of the target attribute.

[0057] If the existence test result of each target attribute exists, but the quality test result of at least one target attribute does not meet the quality standard, averaging can be taken as the target fusion manner, that is, the at least two target text features are averaged to obtain a target fusion feature. If the quality test result of at least one target attribute does not meet the quality standard, the features are fused by adopting the averaging manner, which can balance the different orders of magnitude or different distributions of feature values generated by different feature encoders to some extent, so that the target fusion feature can better represent the overall characteristics of the original data, thereby improving the image generation quality of the target attribute.

[0058] In an optional implementation, the target fusion manner is determined according to the test results of the target attributes, and the target fusion manner is adopted to fuse the at least two target text features to obtain a target fusion feature, including: if the existence test result of at least one target attribute is missing, and the quality test result of at least one target attribute does not meet the quality standard, a first fusion feature is obtained by adopting the maximum pooling manner to fuse the at least two target text features, and a second fusion feature is obtained by adopting the averaging manner to fuse the at least two target text features; the first fusion feature and the second fusion feature are fused by adopting a basic fusion manner to obtain the target fusion feature.

[0059] The base fusion manner is addition or concatenation. If the existence test result of at least one target attribute is missing, and the quality test result of at least one target attribute does not meet the quality standard, both maximum value pooling and averaging are used as the target fusion manner, maximum value pooling is used to perform feature fusion on at least two target text features to obtain first fusion features, averaging is used to perform feature fusion on at least two target text features to obtain second fusion features, and the first fusion features and the second fusion features are added or concatenated to obtain target fusion features. In the case where the target text includes both missing target attributes and target attributes that do not meet the quality standard, first fusion features and second fusion features are obtained by using maximum value pooling and averaging respectively for feature fusion, and the first fusion features and the second fusion features are added or concatenated to obtain target fusion features, so that the target fusion features can avoid being offset or submerged, and can balance different magnitudes or different distributions of feature values, thereby further improving the image generation quality of the target attribute.

[0060] The technical scheme provided by the embodiments of the present disclosure can select a corresponding feature fusion manner by combining whether the existence test result of the target attribute is missing and whether the quality test result of the target attribute meets the quality standard, so as to avoid offset or submersion of the target attribute caused by addition or averaging, or to balance different magnitudes or different distributions of feature values generated by different feature encoders to some extent, thereby improving the image generation quality of the target attribute.

[0061] In an optional implementation method, the target fusion manner is used to fuse at least two target text features to obtain target fusion features, including: if the feature vector dimensions of the at least two target text features are different, the target fusion manner is used to fuse at least two target text features in the feature vector dimension to obtain target fusion features.

[0062] For example, if the feature vector dimensions (i.e., dim dimensions) of at least two target text features are different, that is, the number of elements or the dimension of the feature space contained in the feature vectors output by the two feature encoders are different, the target fusion manner is used to fuse at least two target text features in the feature vector dimension, and the number of tokens of the target fusion features remains unchanged. Tokens are obtained by segmenting target text and can be words, subwords, characters or other meaningful units. If the feature vector dimensions of each target text feature are different, by fusing each target text feature in the feature vector dimension, compared with performing feature fusion in the token dimension, zero padding of the feature vector of one of the target text features can be avoided, so that the feature vector dimensions of various target text features are the same.

[0063] Reference Figure 2b For example, if the target text includes N tokens, the dim dimension of the target text feature output by the first feature encoder is 128 dimensions, the dim dimension of the target text feature output by the second feature encoder is 256 dimensions, and the target fusion mode is concatenation, the target fusion feature with a dimension of 128+256 is obtained by concatenating in the feature vector dimension without zero padding the feature vector of the target text feature. Reference Figure 2c If concatenation in the token dimension requires the target text feature output by the first feature encoder to be padded from 128 dimensions to 256 dimensions before being concatenated with the target text feature output by the second feature encoder, then if the feature vector dimensions of at least two target text features are different, feature fusion in the feature vector dimension can avoid zero padding, thereby reducing the amount of calculation and improving the generation efficiency of the target image.

[0064] It should be noted that if the feature vector dimensions of at least two target text features are the same, fusion can be performed in the token dimension or in the dim dimension.

[0065] Figure 3 is a flowchart of another text-to-image method based on a large model according to an embodiment of the present disclosure. Referring to Figure 3 The text-to-image method based on a large model of the present embodiment can include:

[0066] S301, constructing each test text based on a prompt word template containing at least two candidate attributes;

[0067] S302, using at least two feature encoders to respectively encode the test text to obtain at least two test text features;

[0068] S303, fusing at least two test text features using a basic fusion mode to obtain test fusion features;

[0069] S304, generating a test image according to the test fusion features;

[0070] S305, inputting the test text and the test image into a pre-set multi-modal large language model to obtain existence test results and quality test results of the candidate attributes in the test text;

[0071] S306, counting the existence test results and the quality test results of each candidate attribute to obtain test results of the candidate attributes;

[0072] S307, using at least two feature encoders in the large model to respectively encode the target text to obtain at least two target text features;

[0073] S308, matching the target text and the preset candidate attributes to obtain a target attribute in the target text, and obtaining a test result of the target attribute from the test results of the candidate attributes;

[0074] S309, determining a target fusion mode according to the test result of the target attribute, and fusing at least two target text features by using the target fusion mode to obtain a target fusion feature;

[0075] S310, generating a target image according to the target fusion feature.

[0076] The prompt word (prompt) refers to an input text or an instruction provided to a large model, which is used to instruct or guide the large model to generate a specific output. The prompt word (prompt) template can include at least two candidate attributes, different prompt word modules can include different candidate attributes, and the values of each candidate attribute are to be filled. For example, _ entity _ action _ background is a prompt word module.

[0077] For example, a plurality of prompt word templates are obtained, and for each prompt word module, the values of the candidate attributes in the prompt word module can be set to obtain a test text. Still taking the prompt word module “_ entity _ action _ background” as an example, the generated test texts can be “three cats lying and playing, with a park as the background”, “two cats drinking water, with a mountain village as the background”, “two monkeys climbing trees, with a sunset as the background”, etc. For each test text, the test text is input into two feature encoders to obtain two test text features; the two test text features are fused by using a basic fusion mode to obtain a test fusion feature, and the test fusion feature is input into an image generator to obtain a test image. The basic fusion mode is addition (add) or concatenation (concat).

[0078] And, input the test text and the corresponding test image into a preset multimodal large language model (MLLM) to obtain the existence test result and the quality test result of each candidate attribute in the test text; and statistically analyze the existence test result and the quality test result of the candidate attribute in each test text, and obtain the test result of the candidate attribute according to the statistical result. By constructing each test text including at least two candidate attributes, generating a test image according to the test text, and determining the existence test result and the quality test result of the candidate attribute in the test text according to the test text and the test image, the existence test result and the quality test result of the test attribute in each test text are statistically analyzed to obtain the final existence test result and the quality test result of the test attribute, i.e. the test result of the test attribute, which improves the coverage of each test attribute and thus improves the accuracy of the test result of each test attribute.

[0079] The multimodal large language model is pre-adjusted by a training set, and the training set includes a text-image pair of a sample text and a sample image. The candidate attributes in the sample text have labeled existence true value results and quality true value results. For example, input the sample text and the sample image into the initial multimodal large language model to obtain the existence test result and the quality test result of the candidate attribute in the sample text; and adjust the multimodal large language model by combining the existence test result and the quality test result of each candidate attribute, and the existence true value result and the quality true value result of each candidate attribute to obtain the adjusted multimodal large language model.

[0080] In the text-to-image process, the target text input by the user in the AI (Artificial Intelligence, AI) drawing can be obtained, and at least two target text features are obtained by inputting the target text into at least two feature encoders in the text-to-image large model. The target text is matched with each candidate attribute, and the candidate attribute that matches successfully is used as a target attribute in the target text. The test result of the target attribute is obtained from the test result of each candidate attribute. According to the test result of the target attribute, a target fusion method is determined, and the target fusion method is used to fuse at least two target text features to obtain a target fusion feature; and a target image is generated according to the target fusion feature.

[0081] The technical scheme provided by the embodiments of the present disclosure comprises: constructing each test text including at least two candidate attributes, generating a test image according to the test text, and determining the existence test result and the quality test result of the candidate attribute in the test text according to the test text and the test image; and statistically processing the existence test result and the quality test result of the test attribute in each test text to obtain the final test result of the test attribute, thereby improving the coverage of each test attribute and the accuracy of the test result of each test attribute.

[0082] In an optional embodiment, the statistical processing of the existence test result and the quality test result of each candidate attribute to obtain the test result of the candidate attribute comprises: if the proportion of any candidate attribute missing in the existence test result corresponding to each test text is greater than a preset proportion threshold, the existence test result of the candidate attribute is determined as attribute missing; and if the proportion of any candidate attribute not meeting the quality standard in the quality test result corresponding to each test text is greater than a preset proportion threshold, the quality test result of the candidate attribute is determined as not meeting the quality standard.

[0083] If the proportion of any candidate attribute missing in the existence test result corresponding to each test text is higher than a preset proportion threshold, the existence test result of the candidate attribute is missing; otherwise, it is existing. If the proportion of any candidate attribute not meeting the quality standard in the quality test result corresponding to each test text is higher than a preset proportion threshold, the quality test result of the candidate attribute is not meeting the quality standard; otherwise, it is meeting the quality standard. The proportion threshold is an empirical value, for example, it can be 60%. By statistically processing the existence test result and the quality test result of the candidate attribute corresponding to each test text to obtain the test result of the candidate attribute, the reliability of the test result of the candidate attribute can be improved.

[0084] In an optional embodiment, the above method further comprises: obtaining a historical search text input by a user; if the historical search text includes at least two candidate attributes, generating an initial text template according to the historical search text; and if the usage frequency of the initial text template reaches a preset frequency threshold, regarding the initial text template as the prompt word template.

[0085] For example, a historical search text input by a user in an AI drawing scene can be obtained, and the historical search text is matched with each candidate attribute to obtain the candidate attributes contained in the historical search text. If the historical search text contains at least two candidate attributes, an initial text template can be obtained by replacing the values of the candidate attributes with wildcards and the candidate attributes. In addition, the usage frequency of each initial text template is counted, and the initial text template with a usage frequency reaching a frequency threshold is regarded as the prompt word template. Through the above processing, the coverage of the prompt word template to the candidate attributes and the universality of the prompt word template can be improved.

[0086] In an optional implementation, the constructing of each test text based on the prompt word template containing at least two candidate attributes comprises: determining, for each candidate attribute in the prompt word template, a value of the candidate attribute to obtain the test text. Through the above processing, the flexibility and coverage of the test text can be improved, and the accuracy of the test result of the candidate attribute is further improved.

[0087] Figure 4 is a structural schematic diagram of a large model-based text-to-image device provided according to an embodiment of the present disclosure. The device is suitable for generating images of various styles according to user input or prompts. The device can be implemented in software and / or hardware, and can be integrated into an electronic device. As shown in Figure 4 The large model-based text-to-image device 400 of the present embodiment can include:

[0088] The feature encoding module 410 is configured to encode the target text using at least two feature encoders in the large model to obtain at least two target text features.

[0089] The target attribute module 420 is configured to match the target text with each candidate attribute to obtain a target attribute in the target text, and obtain a test result of the target attribute from test results of each candidate attribute.

[0090] The feature fusion module 430 is configured to determine a target fusion manner according to the test result of the target attribute, and fuse at least two target text features using the target fusion manner to obtain a target fusion feature.

[0091] The image generation module 440 is configured to generate a target image according to the target fusion feature.

[0092] In an optional implementation, the test result of the candidate attribute includes an existence test result and a quality test result. The existence test result is used to represent whether the candidate attribute is missing in the generated image, and the quality test result is used to represent whether the generated image meets the quality standard on the candidate attribute.

[0093] In an optional implementation, the feature fusion module 430 includes:

[0094] The first fusion unit is configured to, if the existence test result of at least one target attribute is missing and the quality test result of each target attribute meets the quality standard, fuse at least two target text features using a maximum value pooling manner to obtain the target fusion feature.

[0095] a second fusion unit, configured to, if the existence test result of each of the target attributes is existence and the quality test result of at least one of the target attributes is not meeting the quality standard, perform feature fusion on at least two of the target text features by using a mean value calculation manner to obtain the target fusion feature.

[0096] In an optional implementation, the feature fusion module 430 includes a third fusion unit, which is specifically configured to:

[0097] if the existence test result of at least one of the target attributes is missing and the quality test result of at least one of the target attributes is not meeting the quality standard, perform feature fusion on at least two of the target text features by using a maximum value pooling manner to obtain a first fusion feature and perform feature fusion on at least two of the target text features by using a mean value calculation manner to obtain a second fusion feature;

[0098] perform fusion on the first fusion feature and the second fusion feature by using a basic fusion manner to obtain the target fusion feature.

[0099] In an optional implementation, the text-to-image device based on a large model 400 described above further includes a candidate attribute test module, which includes:

[0100] a test text unit, configured to construct each test text based on a prompt word template containing at least two candidate attributes;

[0101] a test encoding unit, configured to perform feature encoding on the test text by using at least two of the feature encoders respectively to obtain at least two test text features;

[0102] a test fusion unit, configured to perform fusion on at least two of the test text features by using a basic fusion manner to obtain a test fusion feature;

[0103] a test image unit, configured to generate a test image according to the test fusion feature;

[0104] an attribute test unit, configured to input the test text and the test image into a preset multi-modal large language model to obtain an existence test result and a quality test result of a candidate attribute in the test text;

[0105] a test result unit, configured to perform statistics on the existence test result and the quality test result of each candidate attribute to obtain a test result of the candidate attribute.

[0106] In an optional implementation, the test result unit is specifically configured to:

[0107] If the proportion of any of the candidate attributes missing in the existence test results corresponding to the test texts is greater than a preset proportion threshold, it is determined that the existence test result of the candidate attribute is attribute missing.

[0108] If the proportion of any of the candidate attributes not meeting the quality standard in the quality test results corresponding to the test texts is greater than a preset proportion threshold, it is determined that the quality test result of the candidate attribute is not meeting the quality standard.

[0109] In an optional implementation, the candidate attribute test module further includes a template construction unit, and the template construction unit is specifically configured to:

[0110] obtain historical search texts input by a user;

[0111] if the historical search texts include at least two candidate attributes, generate an initial text template according to the historical search texts;

[0112] if the use frequency of the initial text template reaches a preset frequency threshold, the initial text template is used as the prompt word template.

[0113] In an optional implementation, the test text unit is specifically configured to:

[0114] for each candidate attribute in the prompt word template, randomly determine the value of the candidate attribute to obtain the test text.

[0115] In an optional implementation, the at least two feature encoders include a feature encoder of a CLIP series and a feature encoder of a LLM series.

[0116] In an optional implementation, the feature fusion module 430 is specifically configured to:

[0117] if the feature vector dimensions of the at least two target text features are different, the target fusion manner is used to fuse the at least two target text features in the feature vector dimension to obtain a target fusion feature.

[0118] The text-to-image device based on a large model provided in the embodiments of the present application can execute the text-to-image method based on a large model provided in any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0119] In the technical solutions of the present disclosure, the acquisition, storage and application of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0120] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0121] Figure 5 is a block diagram of an electronic device for implementing a large model-based text-to-image method according to an embodiment of the disclosure. Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown in the figures, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0122] As shown in Figure 5 , the electronic device 500 includes a computing unit 501 that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded into a random access memory (RAM) 503 from a storage unit 508. Various programs and data required for the operation of the electronic device 500 can also be stored in the RAM 503. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0123] Various components in the electronic device 500 are connected to the I / O interface 505, including an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, a speaker, etc.; a storage unit 508, such as a magnetic disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the electronic device 500 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0124] The computing unit 501 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as the large-model-based Vincent diagram method. For example, in some embodiments, the large-model-based Vincent diagram method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the large-model-based Vincent diagram method described above can be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute the large model-based Vincent graph method in any other appropriate manner (eg, by means of firmware).

[0125] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0126] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. Various programs and data required for the operation of the electronic device 500 can also be stored in the RAM 503. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0127] A plurality of components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a magnetic disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the electronic device 500 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0128] The computing unit 501 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 501 performs various methods and processes described above, such as the large model-based text-to-image method. For example, in some embodiments, the large model-based text-to-image method can be implemented as a computer software program, which is tangibly embodied in a machine-readable medium, such as the storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded onto the RAM 503 and executed by the computing unit 501, one or more steps of the large model-based text-to-image method described above can be performed. Alternatively, in other embodiments, the computing unit 501 can be configured to perform the large model-based text-to-image method by any other appropriate means, such as by means of firmware.

[0129] Various implementations of the systems and techniques described above herein can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0130] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package, or entirely on a remote machine or server.

[0131] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0132] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0133] The systems and techniques described here can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0134] The computer system can include clients and servers. The clients and the servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0135] Artificial intelligence is a discipline that studies enabling computers to simulate some thinking processes and intelligent behaviors of people (such as learning, reasoning, thinking, planning, etc.), both hardware and software technologies. Artificial intelligence hardware technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing, etc.; artificial intelligence software technology mainly includes computer vision technology, audio recognition technology, natural language processing technology, machine learning / deep learning technology, big data processing technology, knowledge graph technology, etc.

[0136] Cloud computing refers to accessing elastic and scalable shared physical or virtual resource pools through a network, and the resources can include servers, operating systems, networks, software, applications, and storage devices, etc., and the resources can be deployed and managed in a self-service manner as needed. Through cloud computing technology, efficient and powerful data processing capabilities can be provided for artificial intelligence, blockchain, and other technology applications and model training.

[0137] It should be understood that various forms of flow shown above can be used to reorder, add or delete steps. For example, the steps described in the present disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions of the present disclosure can be achieved, and the present disclosure is not limited herein.

[0138] The above detailed description does not limit the scope of the disclosure. Various modifications, combinations, sub-combinations and alternatives can be made to the detailed description. Any modification, equivalent replacement and improvement etc. made within the spirit and principle of the disclosure shall be included in the scope of the disclosure.

Claims

1. The large-scale model-based cultural graph method includes: Using at least two feature encoders in the large model to respectively encode the target text, thereby obtaining at least two target text features; Matching the target text with each preset candidate attribute to obtain the target attribute in the target text, and obtaining the test result of the target attribute from the test results of each candidate attribute; Determining a target fusion method according to a test result of the target attribute, and fusing at least two target text features using the target fusion method to obtain a target fusion feature; generating a target image according to the target fusion features; The method further comprises: Construct each test text based on a prompt word template containing at least two candidate attributes; Using at least two of the feature encoders to perform feature encoding on the test text respectively to obtain at least two test text features; Fusing at least two of the test text features using a basic fusion method to obtain a test fusion feature; generating a test image according to the test fusion features; Inputting the test text and the test image into a preset multimodal large language model to obtain an existence test result and a quality test result of the candidate attributes in the test text; The existence test results and quality test results of each candidate attribute are statistically analyzed to obtain the test results of the candidate attribute.

2. The method according to claim 1, wherein The test results of the candidate attributes include an existence test result and a quality test result; the existence test result is used to indicate whether the candidate attribute is missing in the generated image, and the quality test result is used to indicate whether the generated image meets the quality standard for the candidate attribute.

3. The method according to claim 1 or 2, wherein: The step of determining a target fusion method according to the test result of the target attribute, and fusing at least two target text features using the target fusion method to obtain a target fusion feature, includes: If the existence test result of at least one of the target attributes is missing, and the quality test results of each of the target attributes all meet the quality standard, then using the maximum pooling method to fuse the at least two target text features to obtain the target fused feature; If the existence test results of each of the target attributes are all existence, and the quality test result of at least one of the target attributes is not meeting the quality standard, then at least two of the target text features are feature fused by averaging to obtain the target fusion feature.

4. The method according to claim 1 or 2, wherein: The step of determining a target fusion method according to the test result of the target attribute, and fusing at least two target text features using the target fusion method to obtain a target fusion feature, includes: If the existence test result of at least one of the target attributes is missing, and the quality test result of at least one of the target attributes is not meeting the quality standard, then the at least two target text features are subjected to feature fusion by maximum pooling to obtain a first fused feature, and the at least two target text features are subjected to feature fusion by averaging to obtain a second fused feature; The first fusion feature and the second fusion feature are fused in a basic fusion manner to obtain the target fusion feature.

5. The method according to claim 1, wherein The existence test results and quality test results of each candidate attribute are statistically analyzed to obtain the test results of the candidate attribute, including: If the proportion of missing attributes in the existence test results corresponding to each of the test texts is greater than a preset proportion threshold, the existence test result of the candidate attribute is determined to be attribute missing; If the proportion of any candidate attribute that does not meet the quality standard in the quality test results corresponding to each test text is greater than a preset proportion threshold, it is determined that the quality test result of the candidate attribute does not meet the quality standard.

6. The method according to claim 1, further comprising: Get the historical search text entered by the user; If the historical search text includes at least two candidate attributes, generating an initial text template according to the historical search text; If the usage frequency of the initial text template reaches a preset frequency threshold, the initial text template is used as the prompt word template.

7. The method according to claim 1, wherein The step of constructing each test text based on a prompt word template containing at least two candidate attributes includes: For each candidate attribute in the prompt word template, a value of the candidate attribute is randomly determined to obtain the test text.

8. The method according to claim 1, wherein The at least two feature encoders include a CLIP series feature encoder and an LLM series feature encoder.

9. The method according to claim 1, wherein The step of fusing at least two target text features using the target fusion method to obtain a target fusion feature includes: If the feature vector dimensions of the at least two target text features are different, the target fusion method is used to splice the at least two target text features in the feature vector dimension to obtain a target fusion feature.

10. Large-scale model-based cultural image installation, including: A feature encoding module, configured to respectively perform feature encoding on the target text using at least two feature encoders in the large model to obtain at least two target text features; A target attribute module is configured to match the target text with preset candidate attributes to obtain the target attribute in the target text, and obtain a test result of the target attribute from the test results of the candidate attributes; A feature fusion module is used to determine a target fusion method according to the test result of the target attribute, and fuse at least two target text features using the target fusion method to obtain a target fusion feature; An image generation module, configured to generate a target image according to the target fusion features; The device further includes a candidate attribute testing module, wherein the candidate attribute testing module includes: A test text unit, configured to construct each test text based on a prompt word template containing at least two candidate attributes; a test encoding unit, configured to perform feature encoding on the test text using at least two feature encoders to obtain at least two test text features; A test fusion unit, configured to fuse at least two of the test text features using a basic fusion method to obtain a test fusion feature; A test image unit, configured to generate a test image according to the test fusion feature; An attribute testing unit, configured to input the test text and the test image into a preset multimodal large language model to obtain an existence test result and a quality test result of candidate attributes in the test text; The test result unit is used to collect statistics on the existence test results and quality test results of each candidate attribute to obtain the test results of the candidate attribute.

11. The device according to claim 10, wherein The test results of the candidate attributes include an existence test result and a quality test result; the existence test result is used to indicate whether the candidate attribute is missing in the generated image, and the quality test result is used to indicate whether the generated image meets the quality standard for the candidate attribute.

12. The device according to claim 10 or 11, wherein The feature fusion module includes: a first fusion unit, configured to fuse at least two target text features using a maximum pooling method to obtain the target fused feature if the existence test result of at least one target attribute is missing and the quality test results of each target attribute all meet the quality standard; The second fusion unit is used to fuse at least two of the target text features by averaging to obtain the target fusion feature if the existence test results of each of the target attributes are all existence and the quality test result of at least one of the target attributes does not meet the quality standard.

13. The device according to claim 10 or 11, wherein The feature fusion module includes a third fusion unit, and the third fusion unit is specifically configured to: If the existence test result of at least one of the target attributes is missing, and the quality test result of at least one of the target attributes is not meeting the quality standard, then the at least two target text features are subjected to feature fusion by maximum pooling to obtain a first fused feature, and the at least two target text features are subjected to feature fusion by averaging to obtain a second fused feature; The first fusion feature and the second fusion feature are fused in a basic fusion manner to obtain the target fusion feature.

14. The device according to claim 10, wherein The test result unit is specifically used for: If the proportion of missing attributes in the existence test results corresponding to each of the test texts is greater than a preset proportion threshold, the existence test result of the candidate attribute is determined to be attribute missing; If the proportion of any candidate attribute that does not meet the quality standard in the quality test results corresponding to each test text is greater than a preset proportion threshold, it is determined that the quality test result of the candidate attribute does not meet the quality standard.

15. The apparatus according to claim 10, wherein the candidate attribute testing module further comprises a template construction unit, wherein the template construction unit is specifically configured to: Get the historical search text entered by the user; If the historical search text includes at least two candidate attributes, generating an initial text template according to the historical search text; If the usage frequency of the initial text template reaches a preset frequency threshold, the initial text template is used as the prompt word template.

16. The device according to claim 10, wherein The test text unit is specifically used for: For each candidate attribute in the prompt word template, a value of the candidate attribute is randomly determined to obtain the test text.

17. The device according to claim 10, wherein The at least two feature encoders include a CLIP series feature encoder and an LLM series feature encoder.

18. The device according to claim 10, wherein The feature fusion module is specifically used for: If the feature vector dimensions of the at least two target text features are different, the target fusion method is used to splice the at least two target text features in the feature vector dimension to obtain a target fusion feature.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 9.

21. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images