Intelligent drawing method, system and equipment based on label matching and storage medium

Abstract

The invention discloses an intelligent drawing method, system and device based on label matching and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the steps of performing word segmentation on an input text through a preset recognition network, and recognizing related tags; determining an AI image generation model corresponding to the label according to the related label; generating a plurality of candidate images corresponding to the label through an AI image generation model corresponding to the label; scoring the image quality of the plurality of candidate images, and selecting the image with the highest score as a target image; enabling the labels to be in one-to-one correspondence with the corresponding target images, constructing a label-image pair data set, and dividing the label-image pair data set into a training set and a verification set; constructing a CLIP model, and training the model by using the training set; and inputting the test set into the trained CLIP model to generate an image. According to the method, the AI drawing algorithm based on the keyword can be optimized, the accuracy of the image generated by the text is improved, and the presentation effect of the image is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an intelligent drawing method, system, device and storage medium based on tag matching. Background Art

[0002] With the rapid development of electronic products, people have more personalized needs for the image display of electronic products (such as wallpapers or lock screen images of mobile phones and tablet devices). Text-generated images need to generate images that are as consistent as possible with the meaning of the text based on the given text. Natural language processing technology and computer vision technology are used to extract semantic information from the text and generate images using generative models such as diffusion models / autoregression.

[0003] When people talk about "AI painting", they mostly refer to computer programs that automatically draw pictures based on deep learning models. On the AI ​​painting application platform, users can enter various painting keywords including style, theme, tone, etc., and then AI searches and learns from the vast resources and materials on the Internet based on the keywords, and finally blends and splices them into paintings that meet the requirements for users to choose and refer to.

[0004] The generation method of images such as wallpapers is to manually design or use Generative Adversarial Network (GAN) to generate pictures. These methods have shortcomings such as high labor cost, poor presentation effect of GAN-generated images, and low success rate. In actual use, a single method or a single model generates images corresponding to texts, and the accuracy of the generated images is low. The degree of matching with user needs is still unsatisfactory, and it is difficult to obtain accurate usage scenarios.

[0005] Therefore, it is an urgent problem for those skilled in the art to propose an intelligent drawing method, system, device and storage medium based on tag matching to solve the problems existing in the prior art. Summary of the invention

[0006] In view of this, the present invention provides an intelligent drawing method, system, device and storage medium based on tag matching, which can optimize the keyword-based AI drawing algorithm, improve the accuracy of images generated from text, and improve the image presentation effect.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] An intelligent drawing method based on label matching, comprising:

[0009] S1. Segment the input text through a preset recognition network, classify the variables corresponding to different fields in the text data, obtain variables belonging to different types, and identify related labels;

[0010] S2. According to the identified relevant labels, determine the AI ​​image generation model corresponding to the label from multiple candidate AI image generation models;

[0011] S3, generating multiple candidate images corresponding to the labels through an AI image generation model corresponding to the labels;

[0012] S4, scoring the image quality of multiple candidate images, and selecting the image with the highest image quality score as the target image;

[0013] S5, matching the labels with the corresponding target images one by one, constructing a label-image pair dataset, and dividing the label-image pair dataset into a training set and a validation set;

[0014] S6, constructing a CLIP model, and using the training set to train the model to obtain a trained CLIP model;

[0015] S7. Input the test set into the trained CLIP model to generate images.

[0016] In the above method, optionally, in S1, the input text includes: text directly input by the user, and conversation text obtained by collecting the user's conversation voice and performing voice recognition;

[0017] The input text is segmented to obtain candidate keywords and the weight of each candidate keyword in the text description information, and the relevant tags are identified according to the weight of the keyword.

[0018] In the above method, optionally, in S2, the input image type is obtained according to the identified relevant label, and the AI ​​image generation model corresponding to the label is determined from multiple candidate AI image generation models according to the image type;

[0019] Multiple candidate AI image generation models include: image-text similarity model and diffusion model, text semantic similarity model and image-text similarity model, diffusion model, and autoregressive model.

[0020] In the above method, optionally, in S3, the image generated by the AI ​​image generation model is screened and denoised to obtain a plurality of candidate images corresponding to the labels;

[0021] The screening process includes: determining whether the generated image meets the resolution requirements, and if not, deleting the image; identifying and deleting the generated image containing sensitive content;

[0022] The denoising process includes: adding T-step random noise to the screened image to obtain the image to be denoised, where T is a positive integer; determining the text vector corresponding to the label, so that the diffusion model performs T-step denoising on the reference image to be denoised based on the cross attention between the text vector and the image to be denoised at each step; and generating multiple candidate images based on the results after T-step denoising.

[0023] In the above method, optionally, in S4, multiple candidate images and corresponding labels are respectively input into the human preference evaluation HPS v2 model to obtain an image quality score for each candidate image.

[0024] An intelligent drawing system based on tag matching, executing any one of the intelligent drawing methods based on tag matching described above, comprising:

[0025] The tag acquisition module obtains the relevant tags in the input text;

[0026] An AI image generation model determination module identifies relevant labels, obtains the input image type, and determines an AI image generation model corresponding to the label from multiple candidate AI image generation models;

[0027] The screening module screens and denoises the images generated by the AI ​​image generation model to obtain multiple candidate images;

[0028] An evaluation module scores the image quality of multiple candidate images through an image quality evaluation model;

[0029] The image generation module inputs the test set of the label-image pair dataset into the trained CLIP model to generate an image.

[0030] An intelligent drawing device based on tag matching, comprising: a processor, a memory and a communication interface;

[0031] The memory stores executable instructions for the processor;

[0032] When the processor executes the instruction, it implements any one of the above-mentioned intelligent drawing methods based on tag matching.

[0033] A computer-readable storage medium stores executable instructions, and when the instructions are executed by a processor, any one of the above-mentioned intelligent drawing methods based on tag matching is implemented.

[0034] It can be seen from the above technical solutions that, compared with the prior art, the present invention provides an intelligent drawing method, system, device and storage medium based on label matching, which has the following beneficial effects: the present invention can generate an image corresponding to the text description information according to the labels in the text description information and the advantages of different models, with higher accuracy; using the image quality evaluation model, multiple candidate images generated by the AI ​​image generation model are evaluated, and the image with the highest image quality score is selected as the target image, which can improve the image presentation effect and obtain an image with a high degree of match with the user's expected presentation effect; the images generated by the AI ​​image generation model are screened and denoised, which effectively improves the quality of the training data, thereby improving the accuracy of the generated image. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0036] Figure 1 A flowchart of an intelligent drawing method based on tag matching provided by the present invention;

[0037] Figure 2 A structural schematic diagram of an intelligent drawing system based on tag matching provided by the present invention;

[0038] Figure 3 A schematic diagram of the structure of an intelligent drawing device based on tag matching provided by the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0041] Reference Figure 1 As shown, the present invention discloses an intelligent drawing method based on label matching, comprising:

[0042] S1. Segment the input text through a preset recognition network, classify the variables corresponding to different fields in the text data, obtain variables belonging to different types, and identify related labels;

[0043] S2. According to the identified relevant labels, determine the AI ​​image generation model corresponding to the label from multiple candidate AI image generation models;

[0044] S3, generating multiple candidate images corresponding to the labels through an AI image generation model corresponding to the labels;

[0045] S4, scoring the image quality of multiple candidate images, and selecting the image with the highest image quality score as the target image;

[0046] S5, matching the labels with the corresponding target images one by one, constructing a label-image pair dataset, and dividing the label-image pair dataset into a training set and a validation set;

[0047] S6, constructing a CLIP model, and using the training set to train the model to obtain a trained CLIP model;

[0048] S7. Input the test set into the trained CLIP model to generate images.

[0049] Furthermore, in S1, the input text includes: text directly input by the user, and conversation text obtained by collecting the user's conversation voice and performing voice recognition;

[0050] Segment the input text, obtain candidate keywords and the weight of each candidate keyword in the text description information, and identify relevant tags based on the keyword weight;

[0051] Specifically, the text can be segmented and the first label can be identified through the existing natural language processing-based word segmentation model. For example, the dictionary-based method is a classic traditional word segmentation method. This method is very intuitive. The word segmentation vocabulary is extracted from a large-scale training corpus, and the word frequency is counted at the same time. The sentence can be segmented through reverse maximum matching, N-shortest path and other word segmentation methods; the dictionary-based word segmentation method is very intuitive, and the final word segmentation effect can be easily adjusted by adding or removing dictionaries. For example, when it is found that a new noun cannot be correctly segmented, it can be directly added to the dictionary to achieve the purpose of correct segmentation; similarly, over-reliance on dictionaries also causes this method to not handle unregistered words very well, and when common strings appear in the words in the dictionary, there will be problems with ambiguous segmentation. This requires a rich enough corpus so that the frequency of each word can be well set, but it is not limited to this.

[0052] Further, in S2, the input image type is obtained according to the identified relevant label, and the AI ​​image generation model corresponding to the label is determined from multiple candidate AI image generation models according to the image type;

[0053] Specifically, the image configuration information can be determined according to the user's description of the target image requirements. The image configuration information may include a target image type and multiple description tags, wherein the image type may also be understood as the theme of the image, such as natural scenery, pets, animation, etc. The description tag may be understood as a descriptive word for the image, or may also be referred to as a prompt word, which may be used to describe the main elements, image style, image details, etc. desired by the user. The user may input one or more description tags, determine the AI ​​image generation model according to the image configuration information, and generate multiple candidate images corresponding to the image configuration information through the AI ​​image generation model;

[0054] Multiple candidate AI image generation models include: image-text similarity model and diffusion model, text semantic similarity model and image-text similarity model, diffusion model, and autoregressive model;

[0055] Specifically, the image-text similarity model and the diffusion model determine a reference image from a candidate image library corresponding to the text description information according to the image-text similarity model, and generate an image corresponding to the text description information based on the reference image and the text description information based on the diffusion model;

[0056] A text semantic similarity model and a picture-text similarity model, determining a target candidate image from a candidate image library corresponding to the text description information according to the text semantic similarity model, and determining an image corresponding to the text description information from the target candidate image according to the picture-text similarity model;

[0057] A diffusion model is used to input the text description information into the diffusion model to obtain an image corresponding to the text description information;

[0058] The autoregressive model is used to input the text description information into the autoregressive model to obtain an image corresponding to the text description information.

[0059] Furthermore, in S3, the images generated by the AI ​​image generation model are screened and denoised to obtain multiple candidate images corresponding to the labels;

[0060] The screening process includes: determining whether the generated image meets the resolution requirements, and if not, deleting the image; identifying and deleting the generated image containing sensitive content;

[0061] The denoising process includes: adding T-step random noise to the screened image to obtain the image to be denoised, where T is a positive integer; determining the text vector corresponding to the label, so that the diffusion model performs T-step denoising on the reference image to be denoised based on the cross attention between the text vector and the image to be denoised at each step; and generating multiple candidate images based on the results after T-step denoising.

[0062] Furthermore, in S4, multiple candidate images and corresponding labels are respectively input into the human preference evaluation HPS v2 model to obtain the image quality score of each candidate image;

[0063] Specifically, the Human Preference Score (HPS v2) model is obtained by training the HPD v2 (Human Preference Dataset). It has good generalization ability and can be used as an evaluation indicator for the Vincent graph model. The model can imitate human aesthetic preferences to objectively score the effects of Vincent graphs and reflect the quality of the generated results. By inputting multiple candidate images and description labels, it can automatically output the relative scores of all candidate images.

[0064] Reference Figure 2 As shown, an intelligent drawing system based on tag matching executes any one of the intelligent drawing methods based on tag matching described above, including:

[0065] The tag acquisition module obtains the relevant tags in the input text;

[0066] An AI image generation model determination module identifies relevant labels, obtains the input image type, and determines an AI image generation model corresponding to the label from multiple candidate AI image generation models;

[0067] The screening module screens and denoises the images generated by the AI ​​image generation model to obtain multiple candidate images;

[0068] An evaluation module scores the image quality of multiple candidate images through an image quality evaluation model;

[0069] The image generation module inputs the test set of the label-image pair dataset into the trained CLIP model to generate an image.

[0070] Reference Figure 3 As shown, an intelligent drawing device based on tag matching includes: a processor, a memory and a communication interface;

[0071] The memory stores executable instructions for the processor;

[0072] When the processor executes the instruction, it implements any one of the above-mentioned intelligent drawing methods based on tag matching.

[0073] A computer-readable storage medium stores executable instructions, and when the instructions are executed by a processor, any one of the above-mentioned intelligent drawing methods based on tag matching is implemented.

[0074] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0075] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An intelligent drawing method based on label matching, characterized in that: include: S1. Segment the input text through a preset recognition network, classify the variables corresponding to different fields in the text data, obtain variables belonging to different types, and identify related labels; S2. According to the identified relevant labels, determine the AI ​​image generation model corresponding to the label from multiple candidate AI image generation models; S3, generating multiple candidate images corresponding to the labels through an AI image generation model corresponding to the labels; S4, scoring the image quality of multiple candidate images, and selecting the image with the highest image quality score as the target image; S5, matching the labels with the corresponding target images one by one, constructing a label-image pair dataset, and dividing the label-image pair dataset into a training set and a validation set; S6, constructing a CLIP model, and using the training set to train the model to obtain a trained CLIP model; S7. Input the test set into the trained CLIP model to generate images.

2. The intelligent drawing method based on label matching according to claim 1, characterized in that: In S1, the input text includes: text directly input by the user, and conversation text obtained by collecting the user's conversation voice and performing voice recognition; The input text is segmented to obtain candidate keywords and the weight of each candidate keyword in the text description information, and the relevant tags are identified according to the weight of the keyword.

3. The intelligent drawing method based on label matching according to claim 1, characterized in that: In S2, according to the identified relevant label, the input image type is obtained, and according to the image type, an AI image generation model corresponding to the label is determined from multiple candidate AI image generation models; Multiple candidate AI image generation models include: image-text similarity model and diffusion model, text semantic similarity model and image-text similarity model, diffusion model, and autoregressive model.

4. The intelligent drawing method based on label matching according to claim 1, characterized in that: In S3, the images generated by the AI ​​image generation model are screened and denoised to obtain multiple candidate images corresponding to the labels; The screening process includes: determining whether the generated image meets the resolution requirements, and if not, deleting the image; identifying and deleting the generated image containing sensitive content; The denoising process includes: adding T-step random noise to the screened image to obtain the image to be denoised, where T is a positive integer; determining the text vector corresponding to the label, so that the diffusion model performs T-step denoising on the reference image to be denoised based on the cross attention between the text vector and the image to be denoised at each step; and generating multiple candidate images based on the results after T-step denoising.

5. The intelligent drawing method based on label matching according to claim 1, characterized in that: In S4, multiple candidate images and corresponding labels are respectively input into the human preference evaluation HPS v2 model to obtain the image quality score of each candidate image.

6. An intelligent drawing system based on tag matching, executing the intelligent drawing method based on tag matching according to any one of claims 1 to 5, characterized in that: include: Tag acquisition module, which obtains relevant tags in the input text; An AI image generation model determination module identifies relevant labels, obtains the input image type, and determines an AI image generation model corresponding to the label from multiple candidate AI image generation models; The screening module screens and denoises the images generated by the AI ​​image generation model to obtain multiple candidate images; An evaluation module scores the image quality of multiple candidate images through an image quality evaluation model; The image generation module inputs the test set of the label-image pair dataset into the trained CLIP model to generate an image.

7. An intelligent drawing device based on tag matching, characterized in that: include: processor, memory and communication interface; The memory stores executable instructions for the processor; When the processor executes the instructions, an intelligent drawing method based on tag matching as described in any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores executable instructions, and when the instructions are executed by the processor, an intelligent drawing method based on tag matching as described in any one of claims 1 to 5 is implemented.

Images