Method and device for generating cue word of stable diffusion model and electronic equipment
By supervising and fine-tuning training of large language models in specific domain data sets, the problem of insufficient accuracy in prompt word generation in the prior art is solved, and more efficient and accurate image generation is achieved.
Patent Information
- Application Number
- CN202411814382.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-11
- Publication Date
- 2025-04-25
AI Technical Summary
The existing stable diffusion model prompt word generation methods rely on manual writing or fixed templates, making it difficult to generate accurate and complex natural language descriptions, resulting in a significant gap between the generated prompt words and the target image.
Generate domain-specific prompt words by supervising fine-tuning training of large language models in specific domain datasets. The method includes collecting text descriptions and image data related to the target image generation domain, establishing a data set for supervising fine-tuning training, and selecting pre-trained large language models for fine-tuning training.
Improve the efficiency and accuracy of image generation, enable more accurate understanding of complex natural language descriptions, and generate prompt words that are more consistent with user intentions.
Smart Images
Figure CN119886090A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of computer image generation and processing, and in particular relates to a prompt word generation method, device and electronic equipment of a stable diffusion model. Background Art
[0002] In the field of image generation and creativity, the stable diffusion model has become one of the advanced tools for generating high-quality images. By learning a large amount of image data, this type of model can correspond the prompt word to the specific image content and generate the corresponding image based on the input description. However, the existing stable diffusion model prompt word generation method faces many challenges:
[0003] Limitations of manually written prompt words: In many applications, users need to manually write detailed and precise prompt words to generate target images. However, manually writing prompt words is time-consuming and complex, and requires users to have professional image description and model interaction skills.
[0004] The current existing technologies are some prompt word generation tools that use predefined templates or simplified natural language processing. These tools mostly rely on fixed templates to combine prompt words, and the generated prompt words lack an accurate understanding of complex and detailed natural language descriptions, resulting in a significant gap with the target image. The main reason is that the prompt word generation is not accurate enough. Fixed templates and simplified natural language processing methods are difficult to accurately understand complex descriptions, resulting in the generated prompt words failing to accurately reflect the user's intentions.
[0005] Therefore, this solution is improved with respect to the above-mentioned shortcomings. Summary of the invention
[0006] Purpose of the invention: In order to overcome the deficiencies in the prior art, the present invention provides a prompt word generation method, device and electronic device of a stable diffusion model, which can perform supervised fine-tuning training on a large language model in a specific field data set, and generate prompt words based on the specific field through the large language model, thereby obtaining prompt words in the specific field and improving the efficiency and accuracy of image generation.
[0007] Technical solution: To achieve the above purpose, the technical solution of the present invention is as follows:
[0008] A prompt word generation method for a stable diffusion model, comprising:
[0009] Pre-training large language models with supervised fine-tuning using domain-specific datasets;
[0010] receiving a user's natural language description of a target image, and converting the input natural language description into text;
[0011] The large language model trained by supervised fine-tuning generates domain-specific prompt words based on the text described by the user and the knowledge learned during the supervised fine-tuning training process.
[0012] Furthermore, the supervised fine-tuning training includes the following steps:
[0013] Collect text descriptions and image data related to the target image generation domain and build a dataset for supervised fine-tuning training;
[0014] A pre-trained large language model is selected as the base model and fine-tuned using a domain-specific dataset.
[0015] Furthermore, the VLLM deployment framework is used to allocate system resources, which specifically includes the following steps:
[0016] Dynamically allocate computing resources to ensure that each user request can be fully processed within the specified time;
[0017] Monitor the status and progress of each user's request and arrange the priority of reasoning tasks through the VLLM reasoning model.
[0018] Furthermore, we collect user feedback, continuously learn user preferences, and optimize the quality and accuracy of prompt word generation, including:
[0019] The generated prompt words are semantically and structurally optimized so that the prompt words are consistent with the user's initial description.
[0020] Furthermore, collecting user feedback includes: collecting and identifying the difference between the final image generated by the user and the initial expected image, and feeding back the degree of the difference to the large language model for supervised fine-tuning training.
[0021] Furthermore, unsafe words are defined, and after the prompt words are generated, the unsafe words are screened and filtered, and after the unsafe words are eliminated, the prompt words that are finally applied to the stable diffusion model are output.
[0022] Furthermore, it also includes:
[0023] Establishing a template library of predefined prompt words, which at least includes a combination of keywords of theme, style and image composition;
[0024] After the user inputs a natural language description and converts it into text, the user's description text is matched with the templates in the predefined template library through keyword matching technology to generate auxiliary words;
[0025] The prompt words generated by the large language model and the auxiliary words generated by the template library are summarized, and the prompt word or prompt word combination that best matches the user description is selected according to a predetermined priority strategy.
[0026] A prompt word generating device of a stable diffusion model, comprising:
[0027] The input module is used to receive the natural language description of the target image input by the user, parse and extract the image attributes, theme and style information, and format it into a format that can be recognized and understood by the large language model;
[0028] Large language model module, which is used to perform supervised fine-tuning training on large language models and generate prompt words suitable for stable diffusion models based on the trained model and user descriptions;
[0029] The concurrent performance improvement module includes a VLLM reasoning module and a resource scheduling and management module, wherein the VLLM reasoning module is used to reason about the production of promotion words, and the resource scheduling and management module is used to dynamically allocate and manage computing resources to ensure priority allocation during multi-task concurrency.
[0030] Furthermore, it also includes a prompt word optimization and feedback module for optimizing the generated prompt words to reflect the user's intention and meet the requirements of image generation, as well as collecting user feedback on the generated images and using it to improve the prompt word generation strategy.
[0031] An electronic device, comprising:
[0032] Memory, used to store computer programs;
[0033] The processor is used to execute the computer program to implement the prompt word generation method of the stable diffusion model.
[0034] Beneficial effects: The prompt word generation method of the stable diffusion model of the present invention can supervise fine-tune the large language model in a specific field data set, and generate prompt words based on the specific field through the large language model, thereby obtaining prompt words in the specific field with ultra-high accuracy, and improving the efficiency and accuracy of image generation. The present invention is suitable for various application scenarios that require the generation of high-quality images, such as artistic creation, advertising design, virtual reality and entertainment content production.
[0035] In the prompt word generation device and electronic device of the stable diffusion model, all the beneficial effects of the prompt word generation method can also be achieved. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Attached Figure 1 It is a schematic diagram of the overall process of the present invention. DETAILED DESCRIPTION
[0037] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0038] The terms used in the present disclosure are for the purpose of describing specific embodiments only and are not intended to limit the present disclosure. The singular forms "a", "said" and "the" used in the present disclosure and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.
[0039] The specific implementation of the present invention is further described in detail below in conjunction with the drawings and examples.
[0040] As attached Figure 1 As shown, a prompt word generation method of a stable diffusion model includes:
[0041] A large language model is pre-trained with supervised fine-tuning using a domain-specific dataset; a natural language description of a target image from a user is received and the input natural language description is converted into text; the large language model after supervised fine-tuning training generates prompt words based on the specific domain based on the text described by the user and the knowledge learned during the supervised fine-tuning training process.
[0042] In order to improve the accuracy and matching degree of prompt word generation, a domain-specific data set is used to supervise the fine-tuning training of LLM. This process allows the model to learn to generate prompt words that are more in line with the target model and user descriptions based on user habits and needs in the target domain. After receiving the natural language description used and converting it into text, the text is processed by the LLM large language model to generate prompt words in a specific field, thereby improving the efficiency and accuracy of image generation. The present invention is suitable for various application scenarios that require the generation of high-quality images, such as artistic creation, advertising design, virtual reality, and entertainment content production.
[0043] In some specific embodiments, the supervised fine-tuning training (SFT) includes the following steps:
[0044] Collect text descriptions and image data related to the target image generation domain and establish a dataset for supervised fine-tuning training; select a pre-trained large language model as the base model and use the domain-specific dataset for fine-tuning training.
[0045] In this process, we collect text descriptions and image data related to the target image generation domain to build a dataset for supervised fine-tuning training. We select a pre-trained large language model (such as GPT-4) as the base model and fine-tune it with the help of a domain-specific dataset.
[0046] Supervised fine-tuning training SFT training process: Through supervised fine-tuning training, the model learns data in a specific field based on existing knowledge in order to more accurately understand and generate prompt words related to image generation. The training process includes data annotation, model parameter adjustment, and performance evaluation.
[0047] The LLM module parses the preprocessed natural language input and identifies key topics, styles, and other attributes in the user description. Based on the knowledge from SFT training, the model converts the parsed user description into the prompt words required by the stable diffusion model. The prompt words include keywords and structured descriptions that are directly related to image generation.
[0048] Although large language models (such as the GPT series) have shown significant potential in prompt word generation, their application in existing systems is still limited by computing resources, inference speed, and concurrency performance, and cannot meet high concurrency requirements, affecting user experience. For this reason, the VLLM deployment framework is used to allocate system resources.
[0049] In some specific implementation methods, the use of the VLLM deployment framework to allocate system resources specifically includes the following steps: dynamically allocating computing resources to ensure that each user request can be fully processed within the specified time; monitoring the status and progress of each user request, and arranging the priority of reasoning tasks through the VLLM reasoning model.
[0050] Use vLLM for efficient model reasoning to ensure high performance in multi-user concurrency. This module dynamically allocates computing resources to ensure that each user request is fully processed. Monitor the status and progress of each user's request, reasonably arrange the priority of reasoning tasks, and reduce response time. Through vLLM's efficient reasoning technology, the model processing capability in concurrent situations is improved to ensure that the system always maintains fast and accurate prompt word generation.
[0051] In some specific embodiments, user feedback is collected, user preferences are continuously learned, and the quality and accuracy of the generated prompt words are optimized, including: semantic and structural optimization of the generated prompt words so that the prompt words conform to the user's initial description. By optimizing the generated prompt words to improve their semantic accuracy and readability, it is ensured that the stable diffusion model can understand and generate high-quality images accordingly.
[0052] Collecting user feedback includes collecting and identifying the differences between the final images generated by the user and the initial expected images, and feeding the degree of difference back to the large language model for supervised fine-tuning training.
[0053] The match between the user-generated image and the original description is used to optimize and improve the prompt word generation strategy. The system continuously adjusts the model parameters and prompt word generation logic based on user feedback to ensure that the prompt word more accurately reflects the user's intention.
[0054] Insufficient filtering of unsafe words in existing technologies: Existing tools cannot effectively filter unsafe words, which may lead to risks of not complying with ethical and legal norms. Therefore, unsafe words are defined to optimize prompt words.
[0055] Unsafe words are defined, and after the prompt words are generated, the unsafe words are screened and filtered. After the unsafe words are eliminated, the prompt words that are finally applied to the stable diffusion model are output.
[0056] In some specific embodiments, the method further includes: establishing a template library of predefined prompt words, which at least includes a combination of keywords of theme, style and image composition;
[0057] After the user inputs a natural language description and converts it into text, the user's description text is matched with the templates in the predefined template library through keyword matching technology to generate auxiliary words;
[0058] The prompt words generated by the large language model and the auxiliary words generated by the template library are summarized, and the prompt word or prompt word combination that best matches the user description is selected according to a predetermined priority strategy.
[0059] This method uses a predefined prompt word template library. If only this method is used to generate prompt words, it is obviously not personalized enough and the prompt words are relatively limited. However, it has the advantage of very high accuracy, which can meet the user's needs for image generation to a certain extent. It can be used as an auxiliary method for the LLM large language model. By combining the predefined prompt word template library with the LLM, more optional prompt words are provided for the final prompt word phrase, thereby increasing the accuracy of the final prompt word and then improving the accuracy of image generation.
[0060] A prompt word generating device of a stable diffusion model, comprising:
[0061] The input module is used to receive the natural language description of the target image input by the user, parse and extract the image attributes, theme and style information, and format it into a format that can be recognized and understood by the large language model;
[0062] Input description: The user inputs a natural language description of the target image through the system's input module. The description can include requirements for the image's theme, color, style, composition, etc.
[0063] Text preprocessing: The input module performs basic preprocessing on the user's input text, including removing stop words, processing spelling errors, and normalizing words to ensure that the model understands the core meaning of the user's description more accurately.
[0064] Formatting: Convert the natural language description entered by the user into a standardized format so that it can be directly processed by the LLM module.
[0065] Large language model module, which is used to perform supervised fine-tuning training on large language models and generate prompt words suitable for stable diffusion models based on the trained model and user descriptions;
[0066] By collecting text descriptions and image data related to the target image generation domain, a dataset for supervised fine-tuning training is established. A pre-trained large language model (such as GPT-4) is selected as the base model and fine-tuned with the help of a domain-specific dataset. Through supervised fine-tuning training, the model learns data in a specific domain based on existing knowledge in order to more accurately understand and generate prompt words related to image generation. The training process includes data annotation, model parameter adjustment, and performance evaluation.
[0067] The concurrent performance improvement module includes a VLLM reasoning module and a resource scheduling and management module, wherein the VLLM reasoning module is used to reason about the production of promotion words, and the resource scheduling and management module is used to dynamically allocate and manage computing resources to ensure priority allocation during multi-task concurrency.
[0068] Introducing vLLM technology for efficient model reasoning, while maintaining the accuracy of prompt word generation, significantly improves multi-user concurrent performance. Dynamically allocate computing resources to ensure that each user request can be fully processed within the specified time. Monitor the status and progress of each user request, reasonably arrange task priorities, and reduce response time.
[0069] In some specific embodiments, a prompt word optimization and feedback module is also included to optimize the generated prompt words to reflect the user's intention and meet the requirements of image generation, and to collect user feedback on the generated images and to improve the prompt word generation strategy.
[0070] An electronic device, comprising:
[0071] Memory, used to store computer programs;
[0072] The processor is used to execute the computer program to implement the prompt word generation method of the stable diffusion model.
[0073] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information.
[0074] In the description of the present invention, it is necessary to understand that the terms "upper", "lower", "front", "back", "left", "right", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.
[0075] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A prompt word generation method for a stable diffusion model, characterized in that: include: Pre-training large language models with supervised fine-tuning using domain-specific datasets; receiving a user's natural language description of a target image, and converting the input natural language description into text; The large language model trained by supervised fine-tuning generates domain-specific prompt words based on the text described by the user and the knowledge learned during the supervised fine-tuning training process.
2. The method for generating prompt words for a stable diffusion model according to claim 1, characterized in that: The supervised fine-tuning training includes the following steps: Collect text descriptions and image data related to the target image generation domain and build a dataset for supervised fine-tuning training; A pre-trained large language model is selected as the base model and fine-tuned using a domain-specific dataset.
3. The method for generating prompt words for a stable diffusion model according to claim 1, characterized in that: The VLLM deployment framework is used to allocate system resources, which includes the following steps: Dynamically allocate computing resources to ensure that each user request can be fully processed within the specified time; Monitor the status and progress of each user's request and arrange the priority of reasoning tasks through the VLLM reasoning model.
4. The method for generating prompt words of a stable diffusion model according to claim 1, characterized in that: Collect user feedback, continuously learn user preferences, and optimize the quality and accuracy of prompt word generation, including: The generated prompt words are semantically and structurally optimized so that the prompt words are consistent with the user's initial description.
5. The method for generating prompt words of a stable diffusion model according to claim 1, characterized in that: Collecting user feedback includes collecting and identifying the differences between the final images generated by the user and the initial expected images, and feeding the degree of difference back to the large language model for supervised fine-tuning training.
6. The method for generating prompt words for a stable diffusion model according to claim 1, characterized in that: Unsafe words are defined, and after the prompt words are generated, the unsafe words are screened and filtered. After the unsafe words are eliminated, the prompt words that are finally applied to the stable diffusion model are output.
7. The method for generating prompt words for a stable diffusion model according to claim 1, characterized in that: Also includes: Establishing a template library of predefined prompt words, which at least includes a combination of keywords of theme, style and image composition; After the user inputs a natural language description and converts it into text, the user's description text is matched with the templates in the predefined template library through keyword matching technology to generate auxiliary words; The prompt words generated by the large language model and the auxiliary words generated by the template library are summarized, and the prompt word or prompt word combination that best matches the user description is selected according to a predetermined priority strategy.
8. A prompt word generating device of a stable diffusion model, characterized in that: include: The input module is used to receive the natural language description of the target image input by the user, parse and extract the image attributes, theme and style information, and format it into a format that can be recognized and understood by the large language model; Large language model module, which is used to perform supervised fine-tuning training on large language models and generate prompt words suitable for stable diffusion models based on the trained model and user descriptions; The concurrent performance improvement module includes a VLLM reasoning module and a resource scheduling and management module, wherein the VLLM reasoning module is used to reason about the production of promotion words, and the resource scheduling and management module is used to dynamically allocate and manage computing resources to ensure priority allocation during multi-task concurrency.
9. The prompt word generating device of the stable diffusion model according to claim 8, characterized in that: It also includes a prompt word optimization and feedback module, which is used to optimize the generated prompt words to reflect the user's intention and meet the requirements of image generation, and to collect user feedback on the generated images and use it to improve the prompt word generation strategy.
10. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor is used to execute the computer program to implement the prompt word generation method of the stable diffusion model according to any one of claims 1 to 7.