Picture generation method, device and system and medium

By extending and offset filtering the basic description text, a target description text collection is generated, and a diversified picture is generated in combination with the diffusion model, the problem of insufficient diversity in image generation in the existing technology is solved and efficient satisfaction of business needs is achieved.

CN118967887APending Publication Date: 2024-11-15PING AN TECH (SHENZHEN) CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411185562.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-27
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Existing image generation technology is difficult to generate diversity and randomness that meets business needs in the production of promotional posters and marketing posters, which leads to business parties need to frequently iterate input to meet the needs of creative design and promotional copywriting.

Method used

By obtaining the basic description text, performing preset expansion strategy expansion, filtering the target description text collection based on the offset degree, and inputting it into the diffusion model to generate a diverse collection of pictures to ensure that the pictures match business needs.

Benefits of technology

It improves the diversity and randomness of image generation results, reduces the iterative input frequency of business parties, and meets the diversified needs of creative design and promotional copywriting.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118967887A_ABST
    Figure CN118967887A_ABST
Patent Text Reader

Abstract

The invention discloses a picture generation method, device and system and a medium. The method comprises the following steps: acquiring a basic description text; expanding the basic description text according to a preset expansion strategy to obtain a plurality of expanded description texts; filtering the plurality of extended description texts according to the offset degree of the plurality of extended description texts and the basic description text to obtain a target description text set; and inputting the target description text set into a diffusion model, and generating a picture set corresponding to the target description text set. By expanding the basic description text and filtering the basic description text on the basis of the offset degree, the target description text set with diversity and randomness is obtained, so that various pictures highly related to services can be automatically generated in combination with a diffusion model, and the diversity of picture generation results is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of financial technology, and in particular to a method, device, system and medium for generating an image. Background Art

[0002] Image generation is a very active research in the field of computer vision and has many practical application scenarios. The images generated by the diffusion model are more diverse and of higher quality than those generated by GAN. The training process of the diffusion model is also more stable, making image generation services such as promotion and marketing, creative design, etc. a reality. However, due to the limitations of diffusion itself, such as the weak relevance of training data to the business and the dependence of output results on prompt quality, it is difficult to generate the images required for the business in one step, and usually multiple iterations are required.

[0003] However, in the process of making promotional posters and marketing posters, the business side usually requires automatic generation of optional materials as backgrounds, subjects, etc. for business design, and requires seamless iterative updates, that is, no additional frequent input from the business side is required for iterative generation. This requires that the generated results must be random or diverse while meeting business requirements to meet the business side's needs for continuous creative design and iterative promotional copywriting. Therefore, how to improve the diversity of image generation results is an urgent problem to be solved. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, an object of the present invention is to provide an image generation method, device, system and medium that can be applied to financial technology or other related fields, and its main purpose is to improve the diversity of image generation results.

[0005] The technical solution of the present invention is as follows:

[0006] A first aspect of the present invention provides a method for generating an image, comprising:

[0007] Get the basic description text;

[0008] Expanding the basic description text according to a preset expansion strategy to obtain multiple extended description texts;

[0009] Filtering the multiple extended description texts according to the offset degree between the multiple extended description texts and the basic description text to obtain a target description text set;

[0010] The target description text set is input into a diffusion model to generate a picture set corresponding to the target description text set.

[0011] In one embodiment, the basic description text is extended according to a preset extension strategy to obtain multiple extended description texts, including:

[0012] Translating the basic description text between Chinese and English for multiple times to obtain multiple different translated texts;

[0013] The basic description text and the multiple mutually translated texts are all subjected to minimum semantic arrangement and combination to obtain multiple extended description texts.

[0014] In one embodiment, the basic description text is extended according to a preset extension strategy to obtain multiple extended description texts, including:

[0015] Inputting the basic description text into a trained text expansion model to expand the basic description text;

[0016] The length of the output result of the text expansion model is controlled to obtain a plurality of extended description texts of different lengths.

[0017] In one embodiment, filtering the multiple extended description texts according to the offset degree between the multiple extended description texts and the basic description text to obtain the target description text set includes:

[0018] Calculating corresponding cosine distance values ​​according to the text embedding vectors of the multiple extended description texts and the text embedding vector of the basic description text;

[0019] Calculate the z score of the corresponding extended description text according to each cosine distance value;

[0020] The extended description texts whose z scores are within a preset filtering range are filtered out from the multiple extended description texts to obtain a target description text set.

[0021] In one embodiment, the step of inputting the target description text set into a diffusion model to generate a picture set corresponding to the target description text set includes:

[0022] Generate corresponding prompt texts according to the multiple target description texts in the target description text set;

[0023] The prompt text is input into the diffusion model, and the random sampling result of adding noise in the forward process and the random intensity of noise reduction in the reverse process of the diffusion model are adjusted to generate a set of pictures corresponding to the multiple prompt texts.

[0024] In one embodiment, after inputting the target description text set into a diffusion model to generate a picture set corresponding to the target description text set, the method further includes:

[0025] Determine, in the picture set, a reference picture having the shortest distance from the basic description text;

[0026] The other pictures in the picture set are filtered according to the degree of deviation between the other pictures and the reference picture to obtain a target picture set.

[0027] In one embodiment, filtering other pictures in the picture set according to the degree of deviation between the other pictures and the reference picture to obtain the target picture set includes:

[0028] Calculate corresponding cosine distance values ​​according to the picture embedding vectors of other pictures in the picture set and the picture embedding vector of the reference picture;

[0029] Calculate the z score of the corresponding image according to each cosine distance value;

[0030] The pictures whose z scores are within a preset filtering range in the picture set are filtered out to obtain a target picture set.

[0031] A second aspect of the present invention provides a picture generating device, comprising:

[0032] A text acquisition module is used to obtain basic description text;

[0033] A text extension module, used to extend the basic description text according to a preset extension strategy to obtain multiple extended description texts;

[0034] A text filtering module, used for filtering the multiple extended description texts according to the offset degree between the multiple extended description texts and the basic description text to obtain a target description text set;

[0035] The image generation module is used to input the target description text set into the diffusion model to generate an image set corresponding to the target description text set.

[0036] A third aspect of the present invention provides a picture generation system, the system comprising at least one processor; and

[0037] a memory communicatively connected to the at least one processor; wherein,

[0038] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above-mentioned picture generation method.

[0039] A fourth aspect of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors can execute the above-mentioned image generation method.

[0040] Beneficial effects: The present invention discloses a method, device, system and medium for generating pictures. Compared with the prior art, the embodiments of the present invention obtain a basic description text; expand the basic description text according to a preset expansion strategy to obtain a plurality of extended description texts; filter the plurality of extended description texts according to the degree of deviation between the plurality of extended description texts and the basic description text to obtain a target description text set; input the target description text set into a diffusion model to generate a picture set corresponding to the target description text set. By expanding the basic description text and filtering based on the degree of deviation, a target description text set with diversity and randomness is obtained, so that a variety of pictures that are strongly related to the business can be automatically generated in combination with the diffusion model, thereby improving the diversity of picture generation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0042] Figure 1 A flow chart of a method for generating an image provided by an embodiment of the present invention;

[0043] Figure 2 A flow chart of step S102 in the picture generation method provided in an embodiment of the present invention;

[0044] Figure 3 Another flow chart of step S102 in the picture generating method provided by an embodiment of the present invention;

[0045] Figure 4 A flow chart of step S103 in the picture generation method provided in an embodiment of the present invention;

[0046] Figure 5 A flow chart of step S104 in the picture generation method provided in an embodiment of the present invention;

[0047] Figure 6 A schematic diagram of functional modules of a picture generating device provided by an embodiment of the present invention;

[0048] Figure 7 A schematic diagram of the hardware structure of the image generation system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. The embodiments of the present invention are described below in conjunction with the accompanying drawings.

[0050] Image generation is a very active research in the field of computer vision and has many practical application scenarios. The images generated by the diffusion model are more diverse and of higher quality than those generated by GAN. The training process of the diffusion model is also more stable, making image generation services such as promotion and marketing, creative design, etc. a reality. However, due to the limitations of diffusion itself, such as the weak relevance of training data to the business and the dependence of output results on prompt quality, it is difficult to generate the images required for the business in one step, and usually multiple iterations are required.

[0051] However, in the process of making promotional posters and marketing posters, the business side usually requires automatic generation of optional materials as backgrounds, subjects, etc. for business design, and requires seamless iterative updates, that is, no additional frequent input from the business side is required for iterative generation. This requires that the generated results must be random or diverse while meeting business requirements to meet the business side's needs for continuous creative design and iterative promotional copywriting. Therefore, how to improve the diversity of image generation results is an urgent problem to be solved.

[0052] In order to solve the above problems, the present invention proposes a method for generating an image. Figure 1 , Figure 1 A flowchart of an embodiment of the image generation method provided by the present invention. The system architecture of the image generation method provided in this embodiment may include a first terminal device, a second terminal device, a third terminal device, a network and a server. The network is used to provide a medium for a communication link between the first terminal device, the second terminal device, the third terminal device and the server. The network may include various connection types, such as wired and / or wireless communication links, etc.

[0053] The user may use the first terminal device, the second terminal device, and the third terminal device to interact with the server through the network to receive or send messages, etc. Various communication client applications may be installed on the first terminal device, the second terminal device, and the third terminal device, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (only as examples).

[0054] The first terminal device, the second terminal device, and the third terminal device may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0055] The server may be a server that provides various services, such as a background management server that provides support for the content browsed by the user using the first terminal device, the second terminal device, and the third terminal device (for example only). The background management server may analyze and process the received data such as the user request, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the terminal device. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server for a distributed system, or a server combined with a blockchain.

[0056] It should be noted that the picture generation method provided in the embodiment of the present invention can generally be executed by the first terminal device, the second terminal device or the third terminal device. Correspondingly, the picture generation device provided in the embodiment of the present invention can also be set in the first terminal device, the second terminal device or the third terminal device.

[0057] Alternatively, the image generation method provided in the embodiment of the present invention may also be generally executed by a server. Accordingly, the image generation device provided in the embodiment of the present invention may generally be arranged in a server. The image generation method provided in the embodiment of the present invention may also be executed by a server or a server cluster that is different from the server and can communicate with the first terminal device, the second terminal device, the third terminal device and / or the server. Accordingly, the image generation device provided in the embodiment of the present invention may also be arranged in a server or a server cluster that is different from the server and can communicate with the first terminal device, the second terminal device, the third terminal device and / or the server.

[0058] It should be understood that the numbers of the above terminal devices, networks and servers are only illustrative. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0059] like Figure 1 As shown, the picture generation method provided in this embodiment specifically includes the following steps:

[0060] S101. Obtain basic description text.

[0061] In this embodiment, the basic description text is a descriptive text pre-generated based on the business product requirements. Specifically, the basic description text can be manually written or automatically generated using natural language generation technology based on information such as product features, target customer groups, and market positioning provided by the business party. For example, 5 to 10 basic description texts that are strongly related to the business product can be generated based on product requirements to ensure that the generated material matches the product positioning and market demand, providing a data basis for subsequent processing, so that processing can be performed based on a small amount of basic description text to meet a variety of business image generation needs.

[0062] S102: Expand the basic description text according to a preset expansion strategy to obtain multiple extended description texts.

[0063] In this embodiment, a small amount of basic description text cannot meet the needs of the business party for continuous creative design and iterative promotional copywriting. Therefore, after obtaining the basic description text, it is subjected to text expansion processing according to a preset expansion strategy to obtain multiple extended description texts. Specifically, multiple different description texts can be generated through different text expansion techniques, such as synonym replacement, sentence reorganization, and insertion of modifiers, etc., to increase the diversity of materials and provide richer visual elements and concepts for the design.

[0064] Since the image generation in this embodiment is based on the diffusion model, that is, the extension model, and the diffusion model is a multimodal model, the semantic control is mainly achieved through the input text and the diffusion process, and the quality of the generated results is also affected by these two processes. Therefore, this embodiment improves the diversity of image generation results by automatically expanding the basic description text according to the preset extension strategy.

[0065] In one embodiment, Figure 2 As shown, the basic description text is extended according to the preset extension strategy to obtain multiple extended description texts, including:

[0066] S201, performing multiple Chinese-English translations on the basic description text to obtain multiple different translated texts;

[0067] S202: performing minimum semantic arrangement and combination on the basic description text and the multiple mutually translated texts to obtain multiple extended description texts.

[0068] In this embodiment, the expansion strategy for the basic description text is to first translate the basic description text into Chinese and English multiple times to obtain multiple different translated texts. For example, a machine translation tool is used to translate the basic description text from Chinese to English, and then the English is translated back to Chinese, and this process is repeated multiple times. Since each translation may introduce new expressions or changes, a random and diverse description angle is brought to the description text used to generate the picture.

[0069] In addition, for the basic description text of the original input and the mutual translation results, that is, multiple mutual translation texts, the minimum semantic permutation and combination is also performed to obtain multiple extended description texts. The minimum semantic permutation and combination refers to the reordering or combination of keywords or tags in the text to generate new text variants while keeping the basic meaning of the text unchanged. By permuting and combining the original text in different ways, texts with different expressions but similar semantics can be generated, thereby providing more diverse inputs for the image generation model; at the same time, when generating images, different text arrangements may affect the weight or importance of each element in the generated image. For example, "red car" and "car red" are similar in semantics, but may have different effects on the emphasis of color and objects when generating images, thereby increasing the diversity and complexity of image generation.

[0070] Specifically, we can first extract key elements, that is, identify key elements or concepts such as nouns and adjectives from the basic description text and each translated text; then arrange and combine these key elements in different orders to form a new text expression. This arrangement and combination can be a simple reordering or a more complex structural change; then maintain minimal semantic changes. Although the order of the text has changed, the goal is to maintain the consistency of the overall semantics, that is, the new text arrangement can still convey the basic meaning of the original text; finally, generate variants, that is, generate multiple variants for each basic description text and translated text, so as to provide a wider context and perspective for image generation. Through the above process, a large number of text variants can be created from limited original text, which not only improves the diversity and creativity of image generation, but also ensures that the generated images can better reflect the semantic content of the text.

[0071] In one embodiment, Figure 3 As shown, the basic description text is extended according to the preset extension strategy to obtain multiple extended description texts, including:

[0072] S301, inputting the basic description text into a trained text expansion model to expand the basic description text;

[0073] S302: Control the output result length of the text extension model to obtain a plurality of extended description texts of different lengths.

[0074] In this embodiment, the expansion strategy for the basic description text is to expand it through a pre-trained text expansion model, that is, to use a large amount of text data to train a text expansion model in advance, and the model can generate new text content according to a given text. Specifically, for example, a ChatGLM model or other large language models can be used, and the obtained basic description text is input into the text expansion model to expand the basic description text. At the same time, the output result length of the text expansion model is controlled, that is, the length limit of the model output text is set. For example, the minimum and maximum number of characters of the generated text can be set to control the length of the output result, and multiple extended description texts of different lengths can be obtained. Through the combination of the text expansion model and the output length control, extended description texts of different lengths can be generated to provide diversified choices for picture generation, adapting to different design requirements and application scenarios.

[0075] S103: Filter the multiple extended description texts according to the offset degree between the multiple extended description texts and the basic description text to obtain a target description text set.

[0076] In this embodiment, by expanding the basic description text through the corresponding preset extension strategy, an extended description text of more than a hundred times can be obtained. For multiple extended description texts, text filtering is further performed based on the degree of offset between each extended description text and the basic description text, that is, through text similarity calculation, such as cosine similarity or Jaccard similarity, the similarity between the extended text and the basic text is evaluated, and texts within a certain range are selected as the target description text set, and texts in the extended description text that are too offset from the basic description text are filtered out, thereby ensuring the similarity between the extended description text and the basic description text, ensuring that the generated material has a certain degree of innovation and diversity while maintaining the original concept, avoiding deviation from the core concept of the product, and ensuring the accuracy and reliability of subsequent image generation.

[0077] In one embodiment, Figure 4 As shown, the filtering of the multiple extended description texts according to the offset degree between the multiple extended description texts and the basic description text to obtain a target description text set includes:

[0078] S401, calculating corresponding cosine distance values ​​according to the text embedding vectors of the multiple extended description texts and the text embedding vector of the basic description text;

[0079] S402, calculating the z score of the corresponding extended description text according to each cosine distance value;

[0080] S403 , filtering out extended description texts whose z-scores are within a preset filtering range from the multiple extended description texts to obtain a target description text set.

[0081] In this embodiment, when filtering the extended description text, the basic description text and all the extended description texts are first converted into embedded vectors using text embedding technology (such as open-clip-g, Word2Vec, BERT, GPT, etc.), and the cosine distance of each pair of embedded vectors of the basic description text and the extended description text is calculated to measure the similarity between them. At the same time, the cosine distance values ​​of all the extended description texts are standardized, and the z score (standard score) of each distance value is calculated to evaluate their deviation from the average distance. According to business needs and an acceptable degree of deviation, a z-score filtering interval is set to filter out extended description texts with z scores in a preset filtering interval. For example, if the filtering interval is set to be greater than 3 and less than -3, it means that the extended description texts with z scores between -3 and 3 are retained. According to the z-score filtering interval, the extended description texts that meet the conditions can be screened out to obtain a target description text set for subsequent image generation.

[0082] This embodiment provides a quantitative method to evaluate and compare the similarity between texts through cosine distance and z-score, and the introduction of z-score makes the filtering process more accurate and can exclude outliers based on statistical principles. Extended description texts that are too far away from the basic description text are filtered out through cosine distance and z-score, which improves the diversity of image generation while also ensuring relevance.

[0083] S104: Input the target description text set into a diffusion model to generate a picture set corresponding to the target description text set.

[0084] In this embodiment, the image generation process is realized by a diffusion model. The core idea of ​​the diffusion model is to convert data into Gaussian distributed noise through a process of gradually adding noise, and then recover data from noise through learning the inverse process to realize data generation. In image generation, the diffusion model first gradually adds noise to the image through a forward process until the image is completely converted into noise. Subsequently, the inverse process gradually predicts and removes noise through a neural network, and finally generates the target image.

[0085] After being trained with a large number of prompt texts and images, the diffusion model can generate corresponding images through the input text. Therefore, after obtaining a target description text set with diverse texts, this embodiment can input it into the trained diffusion model to guide the image generation process, thereby generating corresponding high-quality, high-resolution images according to each extended description text in the target description text set, forming an image set corresponding to the target description text set. By expanding the basic description text and filtering based on the degree of offset, a target description text set with diversity and randomness is obtained, so that a variety of images that are strongly related to the business can be automatically generated in combination with the diffusion model, thereby improving the diversity of image generation results.

[0086] In one embodiment, Figure 5 As shown, the target description text set is input into the diffusion model to generate a picture set corresponding to the target description text set, including:

[0087] S501, generating corresponding prompt texts according to a plurality of target description texts in the target description text set;

[0088] S502: Input the prompt text into a diffusion model, and adjust the random sampling result of adding noise in the forward process and the random intensity of noise reduction in the reverse process of the diffusion model to generate a set of pictures corresponding to the multiple prompt texts.

[0089] In this embodiment, corresponding prompt texts are generated based on multiple target description texts in the target description text set in combination with the prompt template of the diffusion model, so that the multiple target description texts can be converted into prompt texts that can be understood by the diffusion model and can accurately reflect the text content and intention. Afterwards, the generated prompt text can be preprocessed to adapt to the input requirements of the diffusion model, such as word segmentation, removal of stop words, etc., and an encoder (such as CLIP's text encoder) is used to convert the preprocessed prompt text into a vector form that can be understood by the model. Since diffusion is a multimodal model, semantic control is mainly achieved through the input text and the diffusion process. Therefore, on the basis of automatically expanding the input text, the diversity of image generation results can be further improved by controlling the random parameters in the diffusion process.

[0090] The diffusion image generation process mainly includes two processes, one is the forward process of adding noise by random sampling from the normal distribution as the original noise input diffusion, and the other is the reverse process of iterative denoising to generate the final result by combining the prompt text and its control strength. This embodiment adjusts the random sampling result of adding noise in the forward process of the diffusion model, for example, multiplying the random sampling result by a random value in the range of 0.995-1.005 to increase the diversity of random sampling; at the same time, the random intensity of denoising of the model in the reverse process is adjusted, for example, the control intensity of the denoising process is randomly selected in the range of 1.5-4.5, so as to control the diffusion denoising process to increase diversity. By adjusting the parameters of the forward and reverse processes, the image generation effect can be better controlled, and adjusting the random sampling and denoising intensity can further increase the diversity of the generated image.

[0091] In one embodiment, after inputting the target description text set into a diffusion model to generate a picture set corresponding to the target description text set, the method further includes:

[0092] Determine, in the picture set, a reference picture having the shortest distance from the basic description text;

[0093] The other pictures in the picture set are filtered according to the degree of deviation between the other pictures and the reference picture to obtain a target picture set.

[0094] In this embodiment, in order to ensure the relevance between the image and the original text while generating diversity of images, the generated image set is further screened, specifically by first determining the reference image with the smallest distance from the basic description text in the image set, for example, using an image recognition model (such as CNN) to extract image feature vectors from the image set, and using the feature vectors to calculate the similarity between each image and the basic description text, the text description can be converted into an image, that is, the corresponding original image is generated by the basic description text, and then the distance between them (such as cosine similarity) is calculated, and the image with the smallest distance from the basic description text is selected from the image set as the reference image. For other images in the image set, the distance between them and the reference image is calculated, and the same image feature vector and similarity calculation method can be used to filter images based on the degree of offset between other images in the image set and the reference image, and the images with excessive offset from the reference image are filtered out, thereby improving the relevance of the image set to the marketing theme and ensuring that the target image set is consistent with the basic description text in style and content.

[0095] In one embodiment, filtering other pictures in the picture set according to the degree of deviation between the other pictures and the reference picture to obtain the target picture set includes:

[0096] Calculate corresponding cosine distance values ​​according to the picture embedding vectors of other pictures in the picture set and the picture embedding vector of the reference picture;

[0097] Calculate the z score of the corresponding image according to each cosine distance value;

[0098] The pictures whose z scores are within a preset filtering range in the picture set are filtered out to obtain a target picture set.

[0099] In this embodiment, similar to the filtering method of extended text, when filtering pictures in a picture collection, a pre-trained image feature extraction model (such as open-clip-g, etc.) is first used to convert each picture in the picture collection into an embedding vector. Based on the benchmark picture with the smallest distance from the basic description text, the cosine distance between the embedding vector of each picture in the picture collection and the benchmark picture embedding vector is calculated. At the same time, the cosine distances of all pictures are standardized and converted into z scores to evaluate the relative distance between each picture and the benchmark picture. According to business needs and acceptable deviations, a z-score filtering interval is set to determine whether the picture is similar enough to the benchmark picture, and to filter out pictures with z scores in the preset filtering interval. For example, if the filtering interval is set to be greater than 3 and less than -3, it means that pictures with z scores between -3 and 3 are retained. According to the z-score filtering interval, eligible target pictures can be screened out to obtain a target picture collection. By combining cosine distance and z-score to filter images, images with large visual differences from benchmark images can be filtered out. This can more accurately control the consistency and diversity of the image collection, ensure that the images in the target image collection are consistent with the benchmark images or business descriptions in terms of visual themes, and obtain diverse images that are strongly related to business products to meet diverse business needs.

[0100] It should be noted that there is not necessarily a certain order between the above-mentioned steps. A person skilled in the art can understand, based on the description of the embodiments of the present invention, that in different embodiments, the above-mentioned steps may have different execution orders, that is, they may be executed in parallel, may be executed interchangeably, and so on.

[0101] Another embodiment of the present invention provides a picture generating device, such as Figure 6 As shown, the device 1 comprises:

[0102] A text acquisition module 11 is used to acquire basic description text;

[0103] A text expansion module 12, used for expanding the basic description text according to a preset expansion strategy to obtain a plurality of extended description texts;

[0104] A text filtering module 13, configured to filter the plurality of extended description texts according to the degree of deviation between the plurality of extended description texts and the basic description text, to obtain a target description text set;

[0105] The picture generation module 14 is used to input the target description text set into a diffusion model to generate a picture set corresponding to the target description text set.

[0106] The module referred to in the present invention refers to a series of computer program instruction segments that can perform specific functions, which are more suitable for describing the image generation execution process than programs. For the specific implementation of each module, please refer to the corresponding method embodiment above, which will not be repeated here.

[0107] In one embodiment, the text expansion module 12 includes:

[0108] A translation unit, used for performing multiple Chinese-English translations on the basic description text to obtain multiple different translated texts;

[0109] The arrangement and combination unit is used to perform minimum semantic arrangement and combination on the basic description text and multiple mutually translated texts to obtain multiple extended description texts.

[0110] In one embodiment, the text expansion module 12 includes:

[0111] An input unit, used for inputting the basic description text into a trained text expansion model to expand the basic description text;

[0112] The length control unit is used to control the length of the output result of the text expansion model to obtain a plurality of extended description texts of different lengths.

[0113] In one embodiment, the text filtering module 13 includes:

[0114] A first calculation unit, configured to calculate corresponding cosine distance values ​​according to the text embedding vectors of the multiple extended description texts and the text embedding vector of the basic description text;

[0115] A second calculation unit is used to calculate the z score of the corresponding extended description text according to each cosine distance value;

[0116] The text filtering unit is used to filter out the extended description texts whose z scores are within a preset filtering range from the multiple extended description texts to obtain a target description text set.

[0117] In one embodiment, the picture generation module 14 includes:

[0118] A prompt text generating unit, used for generating corresponding prompt texts according to a plurality of target description texts in the target description text set;

[0119] The image generation unit is used to input the prompt text into the diffusion model, and adjust the random sampling result of adding noise in the forward process and the random intensity of noise reduction in the reverse process of the diffusion model to generate a set of images corresponding to multiple prompt texts.

[0120] In one embodiment, the device 1 further comprises:

[0121] A picture determination module, used to determine a reference picture with the shortest distance from the basic description text in the picture set;

[0122] The image filtering module is used to filter other images in the image set according to the degree of deviation between other images and the reference image to obtain a target image set.

[0123] In one embodiment, the image filtering module includes:

[0124] A third calculation unit, configured to calculate corresponding cosine distance values ​​according to the picture embedding vectors of other pictures in the picture set and the picture embedding vector of the reference picture;

[0125] A fourth calculation unit, used to calculate a z score of a corresponding image according to each cosine distance value;

[0126] The image filtering unit is used to filter out the images whose z scores are within a preset filtering range in the image set to obtain a target image set.

[0127] Another embodiment of the present invention provides a picture generation system, such as Figure 7 As shown, the system 10 includes:

[0128] One or more processors 110 and memory 120, Figure 7 A processor 110 is used as an example for description. The processor 110 and the memory 120 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0129] The processor 110 is used to complete various control logics of the system 10, and it can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of these components. In addition, the processor 110 can also be any traditional processor, microprocessor or state machine. The processor 110 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP and / or any other such configuration.

[0130] The memory 120 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the image generation method in the embodiment of the present invention. The processor 110 executes various functional applications and data processing of the system 10 by running the non-volatile software programs, instructions and units stored in the memory 120, that is, implements the image generation method in the above method embodiment.

[0131] The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required by at least one function; the data storage area may store data created according to the use of the system 10, etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include a memory remotely arranged relative to the processor 110, and these remote memories may be connected to the system 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0132] One or more units are stored in the memory 120, and when executed by one or more processors 110, the image generation method in any of the above method embodiments is executed, for example, the following steps are executed:

[0133] Get the basic description text;

[0134] Expanding the basic description text according to a preset expansion strategy to obtain multiple extended description texts;

[0135] Filtering the multiple extended description texts according to the offset degree between the multiple extended description texts and the basic description text to obtain a target description text set;

[0136] The target description text set is input into a diffusion model to generate a picture set corresponding to the target description text set.

[0137] In one embodiment, the basic description text is extended according to a preset extension strategy to obtain multiple extended description texts, including:

[0138] Translating the basic description text between Chinese and English for multiple times to obtain multiple different translated texts;

[0139] The basic description text and the multiple mutually translated texts are all subjected to minimum semantic arrangement and combination to obtain multiple extended description texts.

[0140] In one embodiment, the basic description text is extended according to a preset extension strategy to obtain multiple extended description texts, including:

[0141] Inputting the basic description text into a trained text expansion model to expand the basic description text;

[0142] The length of the output result of the text expansion model is controlled to obtain a plurality of extended description texts of different lengths.

[0143] In one embodiment, filtering the multiple extended description texts according to the offset degree between the multiple extended description texts and the basic description text to obtain the target description text set includes:

[0144] Calculating corresponding cosine distance values ​​according to the text embedding vectors of the multiple extended description texts and the text embedding vector of the basic description text;

[0145] Calculate the z score of the corresponding extended description text according to each cosine distance value;

[0146] The extended description texts whose z scores are within a preset filtering range are filtered out from the multiple extended description texts to obtain a target description text set.

[0147] In one embodiment, the step of inputting the target description text set into a diffusion model to generate a picture set corresponding to the target description text set includes:

[0148] Generate corresponding prompt texts according to the multiple target description texts in the target description text set;

[0149] The prompt text is input into the diffusion model, and the random sampling result of adding noise in the forward process and the random intensity of noise reduction in the reverse process of the diffusion model are adjusted to generate a set of pictures corresponding to the multiple prompt texts.

[0150] In one embodiment, after inputting the target description text set into a diffusion model to generate a picture set corresponding to the target description text set, the method further includes:

[0151] Determine, in the picture set, a reference picture having the shortest distance from the basic description text;

[0152] The other pictures in the picture set are filtered according to the degree of deviation between the other pictures and the reference picture to obtain a target picture set.

[0153] In one embodiment, filtering other pictures in the picture set according to the degree of deviation between the other pictures and the reference picture to obtain the target picture set includes:

[0154] Calculate corresponding cosine distance values ​​according to the picture embedding vectors of other pictures in the picture set and the picture embedding vector of the reference picture;

[0155] Calculate the z score of the corresponding image according to each cosine distance value;

[0156] The pictures whose z scores are within a preset filtering range in the picture set are filtered out to obtain a target picture set.

[0157] An embodiment of the present invention provides a non-volatile computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the image generation method in any of the above method embodiments is executed, for example, the following steps are performed:

[0158] Get the basic description text;

[0159] Expanding the basic description text according to a preset expansion strategy to obtain multiple extended description texts;

[0160] Filtering the multiple extended description texts according to the offset degree between the multiple extended description texts and the basic description text to obtain a target description text set;

[0161] The target description text set is input into a diffusion model to generate a picture set corresponding to the target description text set.

[0162] In one embodiment, the basic description text is extended according to a preset extension strategy to obtain multiple extended description texts, including:

[0163] Translating the basic description text between Chinese and English for multiple times to obtain multiple different translated texts;

[0164] The basic description text and the multiple mutually translated texts are all subjected to minimum semantic arrangement and combination to obtain multiple extended description texts.

[0165] In one embodiment, the basic description text is extended according to a preset extension strategy to obtain multiple extended description texts, including:

[0166] Inputting the basic description text into a trained text expansion model to expand the basic description text;

[0167] The length of the output result of the text expansion model is controlled to obtain a plurality of extended description texts of different lengths.

[0168] In one embodiment, filtering the multiple extended description texts according to the offset degree between the multiple extended description texts and the basic description text to obtain the target description text set includes:

[0169] Calculating corresponding cosine distance values ​​according to the text embedding vectors of the multiple extended description texts and the text embedding vector of the basic description text;

[0170] Calculate the z score of the corresponding extended description text according to each cosine distance value;

[0171] The extended description texts whose z scores are within a preset filtering range are filtered out from the multiple extended description texts to obtain a target description text set.

[0172] In one embodiment, the step of inputting the target description text set into a diffusion model to generate a picture set corresponding to the target description text set includes:

[0173] Generate corresponding prompt texts according to the multiple target description texts in the target description text set;

[0174] The prompt text is input into the diffusion model, and the random sampling result of adding noise in the forward process and the random intensity of noise reduction in the reverse process of the diffusion model are adjusted to generate a set of pictures corresponding to the multiple prompt texts.

[0175] In one embodiment, after inputting the target description text set into a diffusion model to generate a picture set corresponding to the target description text set, the method further includes:

[0176] Determine, in the picture set, a reference picture having the shortest distance from the basic description text;

[0177] The other pictures in the picture set are filtered according to the degree of deviation between the other pictures and the reference picture to obtain a target picture set.

[0178] In one embodiment, filtering other pictures in the picture set according to the degree of deviation between the other pictures and the reference picture to obtain the target picture set includes:

[0179] Calculate corresponding cosine distance values ​​according to the picture embedding vectors of other pictures in the picture set and the picture embedding vector of the reference picture;

[0180] Calculate the z score of the corresponding image according to each cosine distance value;

[0181] The pictures whose z scores are within a preset filtering range in the picture set are filtered out to obtain a target picture set.

[0182] As an example, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM, (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0183] In summary, the present invention discloses a method, device, system and medium for generating pictures, wherein the method includes: obtaining a basic description text; expanding the basic description text according to a preset expansion strategy to obtain a plurality of extended description texts; filtering the plurality of extended description texts according to the degree of deviation between the plurality of extended description texts and the basic description text to obtain a target description text set; inputting the target description text set into a diffusion model to generate a picture set corresponding to the target description text set. By expanding the basic description text and filtering based on the degree of deviation, a target description text set with diversity and randomness is obtained, so that a variety of pictures that are strongly related to the business can be automatically generated in combination with the diffusion model, thereby improving the diversity of picture generation results.

[0184] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, and the computer program can include the processes of the above-mentioned method embodiments when executed. The storage medium can be a memory, a disk, a floppy disk, a flash memory, an optical storage device, etc.

[0185] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for generating an image, characterized in that: include: Get the basic description text; Expanding the basic description text according to a preset expansion strategy to obtain multiple extended description texts; Filtering the multiple extended description texts according to the offset degree between the multiple extended description texts and the basic description text to obtain a target description text set; The target description text set is input into a diffusion model to generate a picture set corresponding to the target description text set.

2. The image generation method according to claim 1, characterized in that: The basic description text is extended according to a preset extension strategy to obtain multiple extended description texts, including: Translating the basic description text between Chinese and English for multiple times to obtain multiple different translated texts; The basic description text and the multiple mutually translated texts are all subjected to minimum semantic arrangement and combination to obtain multiple extended description texts.

3. The image generation method according to claim 1, characterized in that: The basic description text is extended according to a preset extension strategy to obtain multiple extended description texts, including: Inputting the basic description text into a trained text expansion model to expand the basic description text; The length of the output result of the text expansion model is controlled to obtain a plurality of extended description texts of different lengths.

4. The image generation method according to claim 1, characterized in that: The filtering of the multiple extended description texts according to the offset degree between the multiple extended description texts and the basic description text to obtain a target description text set includes: Calculating corresponding cosine distance values ​​according to the text embedding vectors of the multiple extended description texts and the text embedding vector of the basic description text; Calculate the z score of the corresponding extended description text according to each cosine distance value; The extended description texts whose z scores are within a preset filtering range are filtered out from the multiple extended description texts to obtain a target description text set.

5. The image generation method according to claim 1, characterized in that: The step of inputting the target description text set into a diffusion model to generate a picture set corresponding to the target description text set includes: Generate corresponding prompt texts according to the multiple target description texts in the target description text set; The prompt text is input into the diffusion model, and the random sampling result of adding noise in the forward process and the random intensity of noise reduction in the reverse process of the diffusion model are adjusted to generate a set of pictures corresponding to the multiple prompt texts.

6. The image generation method according to claim 1, characterized in that: After inputting the target description text set into the diffusion model to generate a picture set corresponding to the target description text set, the method further includes: Determine, in the picture set, a reference picture having the shortest distance from the basic description text; The other pictures in the picture set are filtered according to the degree of deviation between the other pictures and the reference picture to obtain a target picture set.

7. The image generation method according to claim 6, characterized in that: The filtering other pictures in the picture set according to the degree of deviation between the other pictures and the reference picture to obtain the target picture set includes: Calculate corresponding cosine distance values ​​according to the picture embedding vectors of other pictures in the picture set and the picture embedding vector of the reference picture; Calculate the z score of the corresponding image according to each cosine distance value; The pictures whose z scores are within a preset filtering range in the picture set are filtered out to obtain a target picture set.

8. A picture generating device, characterized in that: include: A text acquisition module is used to obtain basic description text; A text extension module, used to extend the basic description text according to a preset extension strategy to obtain multiple extended description texts; A text filtering module, used for filtering the multiple extended description texts according to the offset degree between the multiple extended description texts and the basic description text to obtain a target description text set; The image generation module is used to input the target description text set into the diffusion model to generate an image set corresponding to the target description text set.

9. A picture generation system, characterized in that: The system includes at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the picture generation method described in any one of claims 1-7.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to execute the image generation method described in any one of claims 1-7.

Citation Information

Cited By

  • Text generation image optimization method and device based on AI feedback and medium

    CN119784894A