Icon generation method and device, icon generation model training method and device, equipment and medium

Abstract

The embodiment of the invention provides an icon generation method and device, an icon generation model training method and device, equipment and a medium. The icon generation method comprises the following steps: acquiring a style reference icon and text description information input by a user; inputting the style reference icon into an image prompt adapter to obtain a style feature vector; inputting the text description information into a text encoder of a stable diffusion model to obtain a text feature vector; inputting the style feature vector and the text feature vector into a U-Net module of a stable diffusion model to obtain an icon prediction vector output by the U-Net module; analyzing the icon prediction vector to obtain at least one target icon consistent with the content of the text description information; the style of the target icon is consistent with that of the style reference icon. Therefore, the high-quality icon meeting the personalized requirements of the user can be generated.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular to methods, devices, equipment and media for icon generation and icon generation model training. Background Art

[0002] In the modern information society, personalized and beautiful icons (ICON) play an important role in various applications, websites and smart device interfaces. With the diversification and personalization of user needs, automatically generating high-quality, personalized icons has become an important research topic.

[0003] In terms of automatic image generation, deep learning technologies such as Generative Adversarial Networks (GAN) and Variational Auto-Encoder (VAE) have been widely used. However, these generative models perform poorly in terms of personalization and fine control, and the generated icons often lack details and have a single style. In addition, when these models generate icons that match the text description, the quality of the generated results is often unstable.

[0004] Based on this, how to automatically generate high-quality icons that meet users' personalized needs has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of this, an embodiment of this specification provides an icon generation method, and an embodiment of this specification also provides a method for training an icon generation model. One or more embodiments of this specification also relate to an icon generation device, a training device for an icon generation model, a computing device, and a computer-readable storage medium to solve the technical defects existing in the prior art.

[0006] According to a first aspect of an embodiment of this specification, there is provided an icon generation method, including: Get the style reference icon and text description information entered by the user; Inputting the style reference icon into the image prompt adapter to obtain a style feature vector; Inputting the text description information into the text encoder of the stable diffusion model to obtain a text feature vector; Inputting the style feature vector and the text feature vector into the U-Net module of the stable diffusion model to obtain an icon prediction vector output by the U-Net module; The icon prediction vector is parsed to obtain at least one target icon that is consistent with the content of the text description information; the style of the target icon is consistent with the style of the style reference icon.

[0007] According to a second aspect of an embodiment of this specification, a method for training an icon generation model is provided, comprising: Acquire a stable diffusion model integrated with an image prompt adapter; the stable diffusion model is used to generate an icon corresponding to the text description information based on the text description information; Acquire a first training sample set; a training sample in the first training sample set includes a first icon, first text description information, and a second icon; the first text description information is description information for the content in the first icon; the style of the second icon is consistent with the style of the first icon; The first training sample set is input into the stable diffusion model integrated with the image prompt adapter, and the parameters in the image prompt adapter are adjusted according to the output of the stable diffusion model to obtain a trained icon generation model.

[0008] According to a third aspect of the embodiments of this specification, there is provided an icon generating device, including: An acquisition module is used to obtain the style reference icon and text description information input by the user; A first input module, used for inputting the style reference icon into the image prompt adapter to obtain a style feature vector; A second input module, used for inputting the text description information into the text encoder of the stable diffusion model to obtain a text feature vector; A third input module, used to input the style feature vector and the text feature vector into the U-Net module of the stable diffusion model to obtain an icon prediction vector output by the U-Net module; A parsing module is used to parse the icon prediction vector to obtain at least one target icon that is consistent with the content of the text description information; the style of the target icon is consistent with the style of the style reference icon.

[0009] According to a fourth aspect of the embodiments of this specification, there is provided a training device for an icon generation model, comprising: A first acquisition module is used to acquire a stable diffusion model integrated with an image prompt adapter; the stable diffusion model is used to generate an icon corresponding to the text description information based on the text description information; A second acquisition module is used to acquire a first training sample set; a training sample in the first training sample set includes a first icon, a first text description information and a second icon; the first text description information is description information for the content in the first icon; the style of the second icon is consistent with the style of the first icon; A parameter adjustment module is used to input the first training sample set into the stable diffusion model integrated with the image prompt adapter, and adjust the parameters in the image prompt adapter according to the output of the stable diffusion model to obtain a trained icon generation model.

[0010] According to a fifth aspect of an embodiment of this specification, a computing device is provided, including: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned icon generation method are implemented, or the steps of the above-mentioned icon generation model training method are implemented.

[0011] According to the sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned icon generation method, or implement the steps of the above-mentioned icon generation model training method.

[0012] At least one embodiment provided in this specification can achieve the following beneficial effects: after the user inputs the style reference icon and text description information into the stable diffusion model integrated with the image prompt adapter, the image prompt adapter integrated in the stable diffusion model can extract the style features of the style reference icon input by the user, so that the model can output an icon consistent with the style of the style reference icon, thereby generating an icon of the same style based on the reference style provided by the user to meet the personalized needs of the user. Moreover, after the stable diffusion model is trained with icon samples, it can also ensure the generation of high-quality icons, thereby using the stable diffusion model integrated with the image prompt adapter to automatically generate high-quality icons that meet the personalized needs of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a flowchart of an icon generation method provided by an embodiment of this specification; Figure 2 It is a model architecture diagram of an icon generation model provided by an embodiment of this specification; Figure 3 It is a flowchart of a training method for an icon generation model provided by an embodiment of this specification; Figure 4 It is a model architecture diagram of a stable diffusion model provided by an embodiment of this specification; Figure 5 is a structural schematic diagram of an icon generating device provided by an embodiment of this specification; Figure 6It is a structural schematic diagram of a training device for an icon generation model provided by an embodiment of this specification; Figure 7 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0014] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0015] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0016] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0017] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0018] In this specification, an icon generation method is provided, and an embodiment of this specification also provides a method for training an icon generation model. One or more embodiments of this specification simultaneously involve an icon generation device, a training device for an icon generation model, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0019] See also Figure 1 , Figure 1 This is a flowchart of an icon generation method provided by an embodiment of this specification. From a program perspective, the execution subject of the process can be an icon generation program. From a hardware perspective, the execution subject of the process can be a device or equipment equipped with an icon generation program. The icon generation method uses a stable diffusion model integrated with an image prompt adapter.

[0020] like Figure 1 As shown, the process may include the following steps: Step 102: Obtain the style reference icon and text description information input by the user.

[0021] An icon is a small image or symbol used to represent a program, file, function, or other object on a computer interface or other device screen. In the user interface of computers and mobile devices, icons are often used to provide graphical visual clues so that users can more easily understand and operate system functions.

[0022] In the embodiments of this specification, the stable diffusion model integrated with the image prompt adapter may also be referred to as an "icon generation model". In the following discussion and explanation, some content will use "icon generation model" to refer to the "stable diffusion model integrated with the image prompt adapter".

[0023] In the embodiments of this specification, the style reference icon input by the user into the model can be an icon of the style of the icon image that the user wants the model to generate. For example, if the user wants the model to generate an icon in the style of a stick figure, then the style reference icon input by the user can be in the style of a stick figure. In actual applications, the user can input the style reference icon of the icon style that he or she wants, and let the model output an icon that is consistent with the style of the style reference icon, thereby meeting the user's personalized needs.

[0024] In the embodiments of this specification, the style reference icon can be a complete icon image, or the style reference icon can be a partial area in a complete icon image. In actual applications, the style reference icon input by the user can be obtained from an icon database, or can be an icon image generated using an icon generation model, or can be obtained by searching on a web page, or can be drawn by the user. The method for obtaining the style reference icon is not specifically limited here.

[0025] In the embodiments of this specification, the text description information input by the user into the model can be a character, a word or several words, or a sentence or several sentences, and there is no specific limitation on this. The content of the icon output by the icon generation model is consistent with the content of the text description information input by the user into the model. For example, if the user wants the model to generate an icon with a stick figure style and a puppy content, the user can input a stick figure style reference icon and a text description information with the content "a puppy" into the model.

[0026] Step 104: input the style reference icon into the image prompt adapter to obtain a style feature vector.

[0027] In the embodiments of this specification, the Image Prompt Adapter (IP-Adapter) is an efficient and lightweight adapter designed to provide image prompting capabilities for a pre-trained text-to-image diffusion model. The core technology of the Image Prompt Adapter lies in its ability to combine image prompts with text prompts, thereby achieving high-quality image generation while keeping the model lightweight. The Image Prompt Adapter is essentially a lightweight module that can be integrated into a Stable Diffusion Model. The Image Prompt Adapter can guide the Stable Diffusion Model to generate images with specific features or styles, so that the Image Prompt Adapter has significant advantages in generating personalized content.

[0028] In actual applications, after the style reference icon is input into the image prompt adapter, the image encoder of the image prompt adapter can extract the style features in the style reference icon to obtain the style feature vector. These style feature vectors will be input into the stable diffusion model as the style reference for the stable diffusion model to generate new icons.

[0029] Step 106: Input the text description information into the text encoder of the stable diffusion model to obtain a text feature vector.

[0030] In the embodiments of this specification, the Stable Diffusion Model is a generative model based on the diffusion process, which is mainly used for image generation and image processing tasks. It uses the diffusion process to gradually transform random noise into a clear image. The Stable Diffusion Model performs well in image generation tasks and can generate detailed and natural images. The Stable Diffusion Model can generate high-quality images under given conditions (such as text descriptions) by learning the distribution of data. The Stable Diffusion Model has received widespread attention for its ability to generate high-quality images and handle diverse tasks, and has been significantly applied in artistic creation, game development and other fields.

[0031] In the embodiments of this specification, a text encoder is a model or algorithm for converting text data into a numerical format. This conversion enables computers to understand and process text information. Text encoders are commonly used in natural language processing (NLP) tasks such as text classification, sentiment analysis, machine translation, question-answering systems, etc. The main function of a text encoder is to map vocabulary (such as words or phrases) in a language into a high-dimensional vector space so that it can be effectively used in a machine learning model. By using a text encoder, text can be converted into a numerical format suitable for various machine learning and deep learning tasks, thereby improving the performance of the model.

[0032] In practical applications, after the text description information is input into the stable diffusion model, the text encoder of the stable diffusion model can convert the text information in the text description information into text feature vectors that can be understood and processed by computers. These text feature vectors will be transmitted to the U-Net module in the stable diffusion model as a content reference for the stable diffusion model to generate new icons.

[0033] Step 108: Input the style feature vector and the text feature vector into the U-Net module of the stable diffusion model to obtain the icon prediction vector output by the U-Net module.

[0034] In the embodiments of this specification, in the stable diffusion model, the U-Net module is used as the core network structure, which can be responsible for the processes such as feature extraction, feature fusion calculation and noise removal in the image generation process. The stable diffusion model can learn how to gradually restore a clear image from random noise through the U-Net module. Due to its efficient structure and good ability to retain details, the U-Net module plays a vital role in the stable diffusion model, enabling the model to generate fine and highly realistic images.

[0035] In practical applications, after the style feature vector and the text feature vector are input into the U-Net module of the stable diffusion model, the U-Net module can perform feature fusion calculation and noise removal on the style feature vector and the text feature vector, and finally output an icon prediction vector. The icon prediction vector can also be a feature vector, and the icon image can be obtained after decoding.

[0036] Step 110: Parse the icon prediction vector to obtain at least one target icon that is consistent with the content of the text description information; the style of the target icon is consistent with the style of the style reference icon.

[0037] Specifically, the icon prediction vector can be parsed using a decoder, and other programs or modules with decoding functions can also be used to parse the icon prediction vector, without specific limitation. Among them, the decoder is an important component in the field of machine learning and deep learning, usually opposite to the encoder. The main function of the decoder is to convert the compression or representation generated by the encoder (such as feature vectors, potential vectors) into raw data or other interpretable output forms. In different models and tasks, the specific implementation and function of the decoder may vary.

[0038] In actual applications, the target icon generated by the icon generation model can be one or more, and there is no specific limitation on this. The style of the target icon generated by the icon generation model is consistent with the style of the style reference icon. In addition, since the content in the target icon is generated based on the text description information, the content of the target icon is consistent with the content of the text description information. For example: the style of the style reference icon input by the user to the icon generation model is a stick figure style, and the input text description information is "puppy", then the target icon output by the icon generation model contains a puppy in the content, and the style is a stick figure style. Since the input text description information does not specify the breed and number of the puppy, the number of puppies in the generated target icon may be only one or more, and the breeds of the puppies may also be diverse.

[0039] Figure 1 According to the method in the embodiment, after the user inputs the style reference icon and text description information into the stable diffusion model integrated with the image prompt adapter, the image prompt adapter integrated in the stable diffusion model can extract the style features of the style reference icon input by the user, so that the model can output an icon with the same style as the style reference icon, thereby generating an icon of the same style based on the reference style provided by the user to meet the personalized needs of the user. Moreover, after the stable diffusion model is trained with icon samples, it can also ensure the generation of high-quality icons, thereby the stable diffusion model integrated with the image prompt adapter can be used to automatically generate high-quality icons that meet the personalized needs of users.

[0040] based on Figure 1 The method in this specification also provides some specific implementation plans of the method, which are described below.

[0041] As an implementation mode, optionally, in the embodiments of this specification, the image prompt adapter may include an image encoder and an adapter layer; the adapter layer is embedded in the U-Net module; the U-Net module may include a cross attention layer and the adapter layer; Correspondingly, the step of inputting the style reference icon into the image prompt adapter to obtain the style feature vector may specifically include: The style reference icon is input into the image encoder of the image prompt adapter to obtain a style feature vector.

[0042] Correspondingly, the step of inputting the style feature vector and the text feature vector into the U-Net module of the stable diffusion model to obtain the icon prediction vector output by the U-Net module may specifically include: Input the text feature vector and the random Gaussian noise matrix into the cross attention layer to obtain a first feature vector output by the cross attention layer; The first feature vector and the style feature vector are input into the adapter layer to obtain the icon prediction vector output by the adapter layer.

[0043] In the embodiments of this specification, the Cross-Attention Layer is a mechanism used in deep learning, especially when processing complex input data, such as when processing natural language processing (NLP) and computer vision (CV) tasks. This mechanism allows the model to establish connections between different parts of the input sequence to help better understand and process information. In generative models, such as stable diffusion models, the Cross-Attention Layer can be used to connect text features and image features to generate high-quality images related to a given description.

[0044] In the embodiments of this specification, the adapter layer (Adapter Layer) of the image prompt adapter is a model structure design that can effectively improve and adjust the performance of pre-trained neural networks, especially when dealing with new tasks or data sets. Adapter layers are often embedded in large pre-trained models to fine-tune specific tasks without completely retraining the entire model. Adapter layers are usually relatively simple sub-networks with a small number of parameters. These adapter layers are usually inserted between feature extraction layers to adjust and enhance the representation capabilities of the model. When dealing with tasks such as image generation, recognition, or conversion, the adapter layer can provide the model with the necessary adjustments so that it can better adapt to the specific needs of the new task, thereby improving performance.

[0045] Figure 2 This is a model architecture diagram of an icon generation model provided by an embodiment of this specification. Figure 2As shown, the icon generation model is a stable diffusion model integrated with an image prompt adapter, wherein the image prompt adapter includes an image encoder and an adapter layer, and the image prompt adapter is integrated into the stable diffusion model by embedding the adapter layer of the image prompt adapter into the U-Net module of the stable diffusion model. After the adapter layer of the image prompt adapter is embedded in the U-Net module of the stable diffusion model, the U-Net module includes a cross attention layer and an adapter layer. In addition, the stable diffusion model can also include an image encoder, a text encoder, and a decoding module.

[0046] In the embodiments of this specification, an image encoder is an algorithm or device that converts image data into a specific format for the purpose of compressing, storing or transmitting images. Image encoders effectively utilize storage space and bandwidth by reducing the size of image files while maintaining the quality of the image as much as possible. In the fields of computer vision and machine learning, image encoders can also refer to models used to extract image features, such as convolutional neural networks (CNNs). These encoders can convert images into low-dimensional feature representations to facilitate subsequent classification, detection and other tasks. In actual applications, the style reference icon is input into the image encoder of the image prompt adapter, and after the style feature extraction processing of the image encoder, a style feature vector can be output.

[0047] In the embodiments of this specification, the random Gaussian noise matrix is ​​a matrix generated by a Gaussian distribution (or normal distribution), in which each element is independently randomly generated and follows a specific Gaussian distribution. The characteristic of the Gaussian distribution is that its shape is defined by the mean and standard deviation. Each element in the random Gaussian noise matrix comes from a specified Gaussian distribution and can be used to simulate random noise in an image or signal. In practical applications, the numpy.random.normal function of the NumPy library can be used to generate a random Gaussian noise matrix, and other methods or functions can also be used to generate a random Gaussian noise matrix, which is not specifically limited.

[0048] In practical applications, after the text feature vector and the random Gaussian noise matrix are input into the cross-attention layer of the U-Net module, after calculation and denoising processing in the cross-attention layer, the first feature vector can be output, where the first feature vector contains text feature information.

[0049] Subsequently, the first feature vector output by the cross attention layer and the style feature vector output by the image encoder of the image prompt adapter are input into the adapter layer, and after further calculation and denoising in the adapter layer, an icon prediction vector can be output. The icon prediction vector output by the adapter layer is input into the decoding module for decoding, and the target icon can be obtained.

[0050] As an implementation manner, optionally, in the embodiment of this specification, the style of the target icon is consistent with the style of the style reference icon, which specifically means that: the target icon and the style reference icon meet at least one of the first condition and the second condition; The first condition includes: a difference between a first color feature value of the target icon and a second color feature value of the style reference icon is less than a first preset threshold; The second condition includes: a difference between a first line feature value of the target icon and a second line feature value of the style reference icon is smaller than a second preset threshold; and the line feature value is used to reflect a width value of a line.

[0051] In the embodiments of the present specification, the consistency between icon styles can be measured based on color features and line features. When the difference between the color feature value of the target icon and the color feature value of the style reference icon is less than a preset threshold, the style of the target icon can be considered to be consistent with the style of the style reference icon; or, when the difference between the line feature value of the target icon and the line feature value of the style reference icon is less than a preset threshold, the style of the target icon can also be considered to be consistent with the style of the style reference icon. In practical applications, it can also be that when the difference between the color feature value of the target icon and the color feature value of the style reference icon is less than a first preset threshold, and the difference between the line feature value of the target icon and the line feature value of the style reference icon is less than a second preset threshold, the style of the target icon is considered to be consistent with the style of the style reference icon. The specific evaluation criteria can be set and adjusted according to actual needs, and no specific limitation is made to this. Among them, the first preset threshold and the second preset threshold can both be set and adjusted according to actual needs, and the numerical value corresponding to the first preset threshold and the numerical value corresponding to the second preset threshold can be the same or different, and no specific limitation is made to this.

[0052] In the embodiments of this specification, color feature values ​​may refer to numerical values ​​or feature vectors used to describe and represent the color features of an image or object in computer vision, image processing, and pattern recognition. These feature values ​​can help computers understand and identify different color information in an image. Common methods for calculating color feature values ​​are as follows: (1) Through a color histogram, a color histogram is a chart that describes the frequency of occurrence of each color in an image. It divides the color space of an image into multiple color intervals and calculates the number of pixels in each interval. (2) Color features obtained by calculating the color values ​​of all pixels in an image (usually RGB or HSV), which are usually used to represent the overall hue of the image. (3) Color moments, features obtained by calculating the statistical data of a color histogram (such as mean, variance, skewness, kurtosis), which can be used to extract higher-level color information. (4) Principal component analysis (PCA), by performing principal component analysis on image colors, the main color features can be obtained. This method can reduce the dimensionality of color data, thereby highlighting the key color information in the image. (5) Use different color models (such as RGB, HSV, LAB, etc.) to represent colors and extract color features from different perspectives. (6) Use a clustering algorithm (such as K-means) to identify the main colors in the image and use them as feature values ​​for subsequent recognition and classification tasks. In practical applications, color feature values ​​can be calculated by any one of the above methods or a combination of multiple methods, or other methods can be used to calculate color feature values, which are not specifically limited.

[0053] In the embodiments of this specification, the line feature value may refer to a numerical value or feature vector used to describe and represent the line features in an image in image processing and computer vision. These features are typically used in applications such as image analysis, object recognition, and shape detection. Specifically, the line feature value may be the width value of the line. For example: Taking the second preset threshold of 5 as an example, if the width value of the line in icon A is 18, the width value of the line in icon B is 21, and the width value of the line in icon C is 55, then it can be considered that the line style of icon A is consistent with the line style of icon B, and the line style of icon C is inconsistent with the line style of both icon A and icon B.

[0054] See also Figure 3 , Figure 3 This is a flowchart of a training method for an icon generation model provided by an embodiment of this specification. From a program perspective, the execution subject of the process can be a program installed on a server or training platform. From a hardware perspective, the execution subject of the process can be a server or training platform that can perform model training.

[0055] like Figure 3 As shown, the process may include the following steps: Step 302: Acquire a stable diffusion model integrated with an image prompt adapter; the stable diffusion model is used to generate an icon corresponding to the text description information based on the text description information.

[0056] In the embodiment of this specification, the image prompt adapter, the stable diffusion model, the icon and the text description information can be Figure 1 The meanings of the terms mentioned in the embodiments of the method are consistent. Since the meanings of these terms have been explained in detail above, they will not be repeated here.

[0057] In the embodiments of this specification, the stable diffusion model is a model pre-trained using icon samples, so the stable diffusion model has the ability to generate high-quality icons corresponding to text description information based on text description information. After the stable diffusion model is pre-trained and has the ability to generate high-quality icons, it can be integrated with an image prompt adapter to obtain an initial icon generation model. The initial icon generation model needs to be trained again later to obtain a trained icon generation model.

[0058] Step 304: Obtain a first training sample set; a training sample in the first training sample set includes a first icon, first text description information and a second icon; the first text description information is description information for the content in the first icon; the style of the second icon is consistent with the style of the first icon.

[0059] In the embodiment of this specification, the first training sample set is used to train the initial icon generation model. In the first training sample set, each training sample contains two icons of the same style and text description information for the content of one of the two icons. Since the specific method of obtaining the first training sample set will be explained in detail in the subsequent embodiments of the embodiment of this specification, it will not be repeated here.

[0060] Step 306: Input the first training sample set into the stable diffusion model integrated with the image prompt adapter, adjust the parameters in the image prompt adapter according to the output of the stable diffusion model, and obtain a trained icon generation model.

[0061] In the embodiments of this specification, since the stable diffusion model is already a pre-trained model with high-quality icon generation capabilities, when using the first training sample set to train the initial icon generation model, the parameters in the stable diffusion model can be locked, and the parameters in the image prompt adapter can be mainly adjusted. After training and tuning the parameters in the image prompt adapter, the trained icon generation model can be obtained. Since there is no need to tune the parameters in the stable diffusion model, it is beneficial to the rapid convergence of the model during training, which is beneficial to improving the model training efficiency and shortening the model training cycle. Since the subsequent embodiments in the embodiments of this specification will explain in detail the process of adjusting the parameters in the image prompt adapter according to the output of the stable diffusion model, it will not be repeated here.

[0062] Figure 3 The method in , by integrating the image prompt adapter into the pre-trained stable diffusion model, obtains the initial icon generation model, then uses the training samples to train the initial icon generation model, and tunes the parameters in the image prompt adapter to obtain the trained icon generation model. On the one hand, the icon generation model can be made capable of generating high-quality icons, and on the other hand, the image prompt adapter can extract style features from the style reference icon input by the user, so that the model can output icons that are consistent with the style reference icon, thereby generating icons of the same style based on the reference style provided by the user to meet the personalized needs of the user.

[0063] based on Figure 3 The method in this specification also provides some specific implementation plans of the method, which are described below.

[0064] As an implementation manner, optionally, in the embodiments of this specification, before acquiring the stable diffusion model integrated with the image prompt adapter, the following steps may also be included: Acquire an original data set; the original data set includes at least one icon and original description data; one icon corresponds to one piece of original description data; Using a natural language model, extended description data corresponding to the original description data of at least one icon is generated to obtain the first training sample set; one icon corresponds to at least one extended description data; the original description data and the extended description data corresponding to the same icon are semantically consistent but have different text lengths.

[0065] In practical applications, the acquired original data set may contain data with problems such as poor quality, noise, duplication, etc. Therefore, a series of predefined filtering and cleaning rules can be used to remove low-quality, duplicate and irrelevant data.

[0066] In the embodiments of this specification, the original data set may include multiple icons and an original description data corresponding to each icon. In addition, the original data set may also include icon groups that have been grouped according to style, each icon group may include at least two icons, and the style of the icons in each icon group is consistent. In practical applications, the original data set can be obtained from multiple public icon libraries or design websites by using web crawler technology, and the original data set can also be obtained by other means, which are not specifically limited.

[0067] In the embodiments of this specification, natural language models are computing models used to understand, generate and process natural languages ​​(such as English, Chinese, etc.). These models use machine learning and deep learning methods to learn the structure, grammar and semantics of languages ​​from large amounts of text data, so that they can perform various language-related tasks.

[0068] In practical applications, the natural language model can be a GPT-4 model or other natural language models, such as a recurrent neural network, a GloVe model, a Sentence-BERT model, and other models. In order to expand the data in the original data set to increase the training samples of the model, the natural language model can be used to expand the original description data to obtain expanded description data. Since the expanded description data is semantically consistent with the original description data but has a different text length, training the icon generation model with training samples containing the expanded description data can improve the generalization ability of the icon generation model and avoid the situation where the icon generation model cannot understand the output error caused by only using samples containing original description data with a single text length to train the icon generation model, once the input text length changes.

[0069] In actual applications, the expanded description data and the original description data not only have different text lengths, but also different grammatical units, where grammatical units may include: words, phrases, sentences, paragraphs, etc. Specifically, for one original description data, the natural language model can expand it into one or more expanded description data. When the original description data is one word or multiple words, the corresponding expanded description data can be a word, a sentence, or a paragraph. For example: if the original description data is "child, red, an apple", the expanded description data expanded by the natural language model can be "a child holding a red apple" or "a child holding a red apple".

[0070] In practical applications, an icon may correspond to an original description data and at least one extended description data. The original description data and extended description data corresponding to the same icon may constitute a text description information set of the icon. In the training samples in the first training sample set, the text description information corresponding to the icon may be a description data randomly selected from the text description information set of the icon. That is, when determining the training samples in the first training sample set, a description data may be randomly selected from the original description data and at least one extended description data corresponding to the icon as the text description information of the icon in the sample. For example, if the original description data corresponding to icon A is a, and it also corresponds to three extended description data, namely b, c and d, then a description data may be randomly selected from a, b, c and d (if c is selected) as the text description information of icon A. After selecting an icon B with the same style as icon A from the original data set, the three data items of icon A, text description information c and icon B may constitute a training sample in the first training sample set.

[0071] Specifically, in order to further increase the number of training samples, the icons in the obtained original data set can also be subjected to color transformation processing. For example, by adjusting the color-related parameters such as the hue, saturation and brightness of the icon, the style of the icon can be changed to obtain more available samples. This allows the data set to cover icon images of different styles and application scenarios, which is conducive to improving the breadth and depth of icon generation model training and making it more adaptable to diverse personalized generation needs.

[0072] As an implementation manner, optionally, in the embodiments of this specification, the obtaining of a stable diffusion model integrated with an image prompt adapter may specifically include: Obtain an initial stable diffusion model; The initial stable diffusion model is pre-trained using a second training sample set to obtain a stable diffusion model; a training sample in the second training sample set includes a third icon and third text description information; the third text description information is description information for the content in the third icon; An image prompt adapter is integrated with the stable diffusion model to obtain the stable diffusion model integrated with the image prompt adapter; the image prompt adapter is used to provide the feature vector extracted from the icon to the U-Net module of the stable diffusion model.

[0073] In practical applications, since the stable diffusion model is rarely used to generate icons, the icon samples used to train the stable diffusion model are also relatively small. Therefore, the icons generated by the initial stable diffusion model are of low quality. It is necessary to pre-train it using icon samples to obtain a stable diffusion model that can generate high-quality icons.

[0074] In an embodiment of the present specification, the icons and text description information corresponding to the icons included in the training samples in the second training sample set may be directly selected from the original data set. In this case, the text description information corresponding to the icon corresponds to the original description data corresponding to the icon in the original data set; or, after obtaining the extended description data set using a natural language model, the second training sample set is determined based on the original data set and the extended description data set. In this case, the text description information corresponding to the icon corresponds to a description data randomly selected from the text description information set of the icon, wherein the text description information set of the icon is composed of an original description data corresponding to the icon and at least one extended description data.

[0075] As an implementation manner, optionally, in the embodiments of this specification, the pre-training of the initial stable diffusion model using the second training sample set to obtain the stable diffusion model may specifically include: Inputting the third icon into the image encoder of the initial stable diffusion model to obtain a third icon feature vector; Inputting the third text description information into the text encoder of the initial stable diffusion model to obtain a third text feature vector; Inputting the third icon feature vector and the third text feature vector into the U-Net module of the initial stable diffusion model to obtain a second feature vector output by the U-Net module; parsing the second characteristic vector to obtain a first prediction icon output by the initial stable diffusion model; According to a first loss value between the first predicted icon and the third icon, parameters in the initial stable diffusion model are adjusted to obtain a trained stable diffusion model.

[0076] The adjusting the parameters in the initial stable diffusion model according to the first loss value between the first predicted icon and the third icon to obtain the trained stable diffusion model may specifically include: Determine a first loss value between the first prediction icon and the third icon; the first loss value is used to reflect the size of the difference between the first prediction icon and the third icon; Taking minimizing the first loss value as an adjustment target, the parameters in the initial stable diffusion model are adjusted to obtain a trained stable diffusion model.

[0077] Figure 4is a model architecture diagram of a stable diffusion model provided in one embodiment of this specification. In the embodiment of this specification, the model architecture of the initial stable diffusion model is consistent with the model architecture of the stable diffusion model. Figure 4 The model architecture of the stable diffusion model in the above example is used to illustrate the model architecture of the initial stable diffusion model. Figure 4 As shown, the stable diffusion model includes an image encoder, a text encoder, a U-Net module and a decoding module, wherein the image encoder and the text encoder are respectively connected to the U-Net module, and the U-Net module is connected to the decoding module.

[0078] In the embodiment of this specification, the third icon is input into the image encoder of the initial stable diffusion model, and the image encoder can extract the image features in the third icon to obtain the third icon feature vector. The third text description information is input into the text encoder of the initial stable diffusion model, and the text encoder can extract the text features in the third text description information to obtain the third text feature vector.

[0079] Furthermore, after the third icon feature vector and the third text feature vector are input into the U-Net module of the initial stable diffusion model, the U-Net module can perform feature fusion calculation and noise removal on the third icon feature vector and the third text feature vector, and finally output a second feature vector. The second feature vector contains the image feature information of the third icon and the content information corresponding to the third text description information.

[0080] Subsequently, after the U-Net module outputs the second feature vector, the second feature vector may be input into a decoding module for decoding processing to obtain a first prediction icon output by the initial stable diffusion model.

[0081] Specifically, a preset loss function can be used to calculate the first loss value between the first prediction icon and the third icon, and the parameters in the initial stable diffusion model are adjusted with the minimum first loss value as the adjustment target to obtain a trained stable diffusion model. In addition, the first loss value is used to reflect the difference between the first prediction icon and the third icon, that is, the smaller the difference between the first prediction icon output by the model and the third icon input to the model, the better the stable diffusion model is.

[0082] As an implementation manner, optionally, in the embodiment of this specification, the inputting the first training sample set into the stable diffusion model integrated with the image prompt adapter specifically includes: For a training sample in the first training sample set, inputting the first icon into an image encoder of the stable diffusion model; Inputting the first text description information into a text encoder of the stable diffusion model; The second icon is input to an image encoder of the image prompt adapter.

[0083] In the embodiment of the present specification, a training sample in the first training sample set includes a first icon, a first text description information and a second icon. And these three items of information in the same training sample are input into different modules in the icon generation model: the first icon is input into the image encoder of the stable diffusion model, the first text description information corresponding to the first icon is input into the text encoder of the stable diffusion model, and the second icon consistent with the style of the first icon is input into the image encoder of the image prompt adapter.

[0084] As an implementation mode, optionally, in the embodiments of this specification, the image prompt adapter may include an image encoder and an adapter layer; the adapter layer is embedded in the U-Net module of the stable diffusion model; the U-Net module may include a cross-attention layer and the adapter layer.

[0085] Correspondingly, after inputting the first training sample set into the stable diffusion model integrated with the image prompt adapter, the method may further include: Acquire a first icon feature vector output by the image encoder of the stable diffusion model and a first text feature vector output by the text encoder of the stable diffusion model; Inputting the first icon feature vector and the first text feature vector into the cross attention layer to obtain a third feature vector output by the cross attention layer; Obtaining a second icon feature vector output by the image encoder of the image prompt adapter; Inputting the third feature vector and the second icon feature vector into the adapter layer to obtain an icon prediction vector output by the adapter layer; The icon prediction vector is parsed to obtain a second prediction icon output by the stable diffusion model.

[0086] In the embodiments of this specification, since the above content has been based on Figure 2 The model architecture of the icon generation model has been described in detail, so it will not be repeated here. After the first icon is input into the image encoder of the stable diffusion model, the image encoder can extract the image features in the first icon to obtain a first icon feature vector. The first text description information is input into the text encoder of the stable diffusion model, and the text encoder can extract the text features in the first text description information to obtain a first text feature vector.

[0087] In practical applications, after the image encoder outputs the first icon feature vector, the first icon feature vector can also be denoised, that is, a random Gaussian noise matrix is ​​added to obtain the noisy first icon feature vector, and then the noisy first icon feature vector and the first text feature vector are input into the cross-attention layer of the U-Net module. After fusion calculation and denoising in the cross-attention layer, a third feature vector can be output, wherein the third feature vector contains the image feature information of the first icon and the content information corresponding to the first text description information.

[0088] In the embodiment of the present specification, after the second icon is input into the image encoder of the image prompt adapter, the image encoder of the image prompt adapter can extract the style feature information of the second icon to obtain the second icon feature vector. Subsequently, the third feature vector output by the cross-attention layer and the second icon feature vector output by the image encoder of the image prompt adapter are input into the adapter layer, and after further calculation and denoising processing in the adapter layer, an icon prediction vector can be output. The icon prediction vector output by the adapter layer is input into the decoding module for decoding processing, or the icon prediction vector is directly parsed, and the second prediction icon output by the stable diffusion model can be obtained.

[0089] As an implementation manner, optionally, in the embodiments of this specification, adjusting the parameters in the image prompt adapter according to the output of the stable diffusion model to obtain the trained icon generation model may specifically include: Determine a second loss value between the second prediction icon output by the stable diffusion model and the first icon; the second loss value is used to reflect the size of the difference between the second prediction icon and the first icon; Taking minimizing the second loss value as an adjustment target, the parameters in the image prompt adapter are adjusted to obtain a trained icon generation model.

[0090] In the embodiments of this specification, a preset loss function can be used to calculate the second loss value between the second predicted icon and the first icon, and the parameters in the initial icon generation model are adjusted with the minimum second loss value as the adjustment target to obtain a trained icon generation model. In addition, the second loss value is used to reflect the difference between the second predicted icon and the first icon, that is, the smaller the difference between the second predicted icon output by the model and the first icon input to the model, the better the icon generation model is.

[0091] In practical applications, since the stable diffusion model is a model that has been pre-trained and tuned, during the training of the icon generation model, the parameters in the stable diffusion model can be frozen, and only the parameters in the image prompt adapter can be adjusted. More preferably, since the parameters of the image encoder of the image prompt adapter have little effect on the performance of the icon generation model, the parameters of the image encoder of the image prompt adapter can also be frozen, and only the parameters of the adapter layer of the image prompt adapter can be adjusted, which is beneficial to the rapid convergence of the model during training, improves the model training efficiency, and shortens the model training cycle.

[0092] Corresponding to the above method embodiment, this specification also provides an icon generation device embodiment, Figure 5 FIG. 1 shows a schematic diagram of the structure of an icon generation device provided by an embodiment of the present specification. The device uses a stable diffusion model integrated with an image prompt adapter, such as Figure 5 As shown, the device may include: The acquisition module 502 is used to acquire the style reference icon and text description information input by the user.

[0093] The first input module 504 is used to input the style reference icon into the image prompt adapter to obtain a style feature vector.

[0094] The second input module 506 is used to input the text description information into the text encoder of the stable diffusion model to obtain a text feature vector.

[0095] The third input module 508 is used to input the style feature vector and the text feature vector into the U-Net module of the stable diffusion model to obtain the icon prediction vector output by the U-Net module.

[0096] The parsing module 510 is used to parse the icon prediction vector to obtain at least one target icon that is consistent with the content of the text description information; the style of the target icon is consistent with the style of the style reference icon.

[0097] Optionally, the image prompt adapter includes an image encoder and an adapter layer; the adapter layer is embedded in the U-Net module; the U-Net module includes a cross attention layer and the adapter layer; correspondingly, the first input module 504 may specifically include: The first input unit is used to input the style reference icon into the image encoder of the image prompt adapter to obtain a style feature vector.

[0098] Correspondingly, the third input module 508 may specifically include: The first input unit is used to input the text feature vector and the random Gaussian noise matrix into the cross-attention layer to obtain the first feature vector output by the cross-attention layer.

[0099] The second input unit is used to input the first feature vector and the style feature vector into the adapter layer to obtain the icon prediction vector output by the adapter layer.

[0100] Optionally, the style of the target icon is consistent with the style of the style reference icon, which may specifically mean that: the target icon and the style reference icon satisfy at least one of a first condition and a second condition; The first condition may include: a difference between a first color feature value of the target icon and a second color feature value of the style reference icon is smaller than a first preset threshold.

[0101] The second condition may include: a difference between a first line feature value of the target icon and a second line feature value of the style reference icon is smaller than a second preset threshold; and the line feature value is used to reflect a width value of a line.

[0102] The above is a schematic scheme of an icon generation device of this embodiment. It should be noted that the technical scheme of the icon generation device and the technical scheme of the above-mentioned icon generation method belong to the same concept, and the details of the technical scheme of the icon generation device that are not described in detail can all be referred to the description of the technical scheme of the above-mentioned icon generation method.

[0103] Corresponding to the above method embodiment, this specification also provides an embodiment of a training device for an icon generation model, Figure 6 FIG. 1 is a schematic diagram showing a structure of a training device for an icon generation model provided by an embodiment of the present specification. Figure 6 As shown, the device may include: The first acquisition module 602 is used to acquire a stable diffusion model integrated with an image prompt adapter; the stable diffusion model is used to generate an icon corresponding to the text description information based on the text description information.

[0104] The second acquisition module 604 is used to acquire a first training sample set; a training sample in the first training sample set includes a first icon, a first text description information and a second icon; the first text description information is the description information for the content in the first icon; the style of the second icon is consistent with the style of the first icon.

[0105] The parameter adjustment module 606 is used to input the first training sample set into the stable diffusion model integrated with the image prompt adapter, and adjust the parameters in the image prompt adapter according to the output of the stable diffusion model to obtain a trained icon generation model.

[0106] Optionally, the device may further include: The third acquisition module is used to acquire an original data set; the original data set includes at least one icon and original description data; one icon corresponds to one piece of original description data.

[0107] A data expansion module is used to use a natural language model to generate expanded description data corresponding to the original description data of at least one icon to obtain the first training sample set; one icon corresponds to at least one expanded description data; the original description data and the expanded description data corresponding to the same icon are semantically consistent and have different text lengths.

[0108] Optionally, the first acquisition module 602 may specifically include: The acquisition unit is used to obtain an initial stable diffusion model.

[0109] A pre-training unit is used to pre-train the initial stable diffusion model using a second training sample set to obtain a stable diffusion model; a training sample in the second training sample set includes a third icon and third text description information; the third text description information is description information for the content in the third icon.

[0110] A model integration unit is used to integrate an image prompt adapter with the stable diffusion model to obtain the stable diffusion model integrated with the image prompt adapter; the image prompt adapter is used to provide a feature vector extracted from the icon to a U-Net module of the stable diffusion model.

[0111] Optionally, the pre-training unit may specifically include: The first input subunit is used to input the third icon into the image encoder of the initial stable diffusion model to obtain a third icon feature vector.

[0112] The second input subunit is used to input the third text description information into the text encoder of the initial stable diffusion model to obtain a third text feature vector.

[0113] The third input subunit is used to input the third icon feature vector and the third text feature vector into the U-Net module of the initial stable diffusion model to obtain a second feature vector output by the U-Net module.

[0114] The parsing subunit is used to parse the second feature vector to obtain a first prediction icon output by the initial stable diffusion model.

[0115] The parameter adjustment subunit is used to adjust the parameters in the initial stable diffusion model according to the first loss value between the first prediction icon and the third icon to obtain a trained stable diffusion model.

[0116] Optionally, the parameter adjustment module 606 may specifically include: The first input unit is configured to input the first icon to an image encoder of the stable diffusion model for a training sample in the first training sample set.

[0117] The second input unit is used to input the first text description information into the text encoder of the stable diffusion model.

[0118] The third input unit is used to input the second icon to the image encoder of the image prompt adapter.

[0119] Optionally, the image prompt adapter includes an image encoder and an adapter layer; the adapter layer is embedded in a U-Net module of the stable diffusion model; the U-Net module includes a cross attention layer and the adapter layer; correspondingly, the device may further include: A fourth acquisition module is used to acquire a first icon feature vector output by the image encoder of the stable diffusion model and a first text feature vector output by the text encoder of the stable diffusion model.

[0120] The first input module is used to input the first icon feature vector and the first text feature vector into the cross attention layer to obtain a third feature vector output by the cross attention layer.

[0121] The fifth acquisition module is used to acquire the second icon feature vector output by the image encoder of the image prompt adapter.

[0122] The second input module is used to input the third feature vector and the second icon feature vector into the adapter layer to obtain the icon prediction vector output by the adapter layer.

[0123] The parsing module is used to parse the icon prediction vector to obtain a second predicted icon output by the stable diffusion model.

[0124] Optionally, the parameter adjustment module 606 may further include: A loss value determination unit is used to determine a second loss value between the second prediction icon output by the stable diffusion model and the first icon; the second loss value is used to reflect the size of the difference between the second prediction icon and the first icon.

[0125] A parameter adjustment unit is used to adjust the parameters in the image prompt adapter with the second loss value being minimized as an adjustment target to obtain a trained icon generation model.

[0126] The above is a schematic scheme of a training device for an icon generation model of this embodiment. It should be noted that the technical scheme of the training device for the icon generation model and the technical scheme of the training method for the icon generation model described above belong to the same concept, and the details of the technical scheme of the training device for an icon generation model that are not described in detail can all be referred to the description of the technical scheme of the training method for the icon generation model described above.

[0127] Figure 7 The block diagram of a computing device provided by an embodiment of the present specification is shown. The components of the computing device 700 include but are not limited to a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and the database 750 is used to store data.

[0128] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0129] In one embodiment of the present specification, the above components of the computing device 700 and Figure 7 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 7 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0130] The computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 700 may also be a mobile or stationary server.

[0131] The processor 720 is used to execute the following computer executable instructions, which, when executed by the processor, implement the steps of the above-mentioned icon generation method or icon generation model training method.

[0132] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned icon generation method or icon generation model training method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above-mentioned icon generation method or icon generation model training method.

[0133] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned icon generation method, or implement the steps of the above-mentioned icon generation model training method.

[0134] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the icon generation method or the training method of the icon generation model are of the same concept, and the details not described in detail in the technical scheme of the storage medium can be found in the description of the technical scheme of the icon generation method or the training method of the icon generation model.

[0135] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0136] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0137] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0138] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0139] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can understand and use this specification well. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A method for generating an icon, characterized in that: The method uses a stable diffusion model integrated with an image cueing adapter, and the method includes: Get the style reference icon and text description information entered by the user; Inputting the style reference icon into the image prompt adapter to obtain a style feature vector; Inputting the text description information into the text encoder of the stable diffusion model to obtain a text feature vector; Inputting the style feature vector and the text feature vector into the U-Net module of the stable diffusion model to obtain an icon prediction vector output by the U-Net module; The icon prediction vector is parsed to obtain at least one target icon that is consistent with the content of the text description information; the style of the target icon is consistent with the style of the style reference icon.

2. The method according to claim 1, characterized in that: The image prompt adapter includes an image encoder and an adapter layer; the adapter layer is embedded in the U-Net module; The U-Net module includes a cross attention layer and the adapter layer; The step of inputting the style reference icon into the image prompt adapter to obtain a style feature vector specifically includes: Inputting the style reference icon into the image encoder of the image prompt adapter to obtain a style feature vector; The step of inputting the style feature vector and the text feature vector into the U-Net module of the stable diffusion model to obtain the icon prediction vector output by the U-Net module specifically includes: Input the text feature vector and the random Gaussian noise matrix into the cross attention layer to obtain a first feature vector output by the cross attention layer; The first feature vector and the style feature vector are input into the adapter layer to obtain the icon prediction vector output by the adapter layer.

3. The method according to claim 1, characterized in that The style of the target icon is consistent with the style of the style reference icon, which specifically means that: the target icon and the style reference icon meet at least one of the first condition and the second condition; The first condition includes: a difference between a first color feature value of the target icon and a second color feature value of the style reference icon is less than a first preset threshold; The second condition includes: a difference between a first line feature value of the target icon and a second line feature value of the style reference icon is smaller than a second preset threshold; and the line feature value is used to reflect a width value of a line.

4. A training method for an icon generation model, characterized in that: The method comprises: Acquire a stable diffusion model integrated with an image prompt adapter; the stable diffusion model is used to generate an icon corresponding to the text description information based on the text description information; Acquire a first training sample set; a training sample in the first training sample set includes a first icon, first text description information, and a second icon; the first text description information is description information for the content in the first icon; the style of the second icon is consistent with the style of the first icon; The first training sample set is input into the stable diffusion model integrated with the image prompt adapter, and the parameters in the image prompt adapter are adjusted according to the output of the stable diffusion model to obtain a trained icon generation model.

5. The method according to claim 4, characterized in that Before acquiring the stable diffusion model integrated with the image prompt adapter, the method further includes: Acquire an original data set; the original data set includes at least one icon and original description data; one icon corresponds to one piece of original description data; Using a natural language model, extended description data corresponding to the original description data of at least one icon is generated to obtain the first training sample set; one icon corresponds to at least one extended description data; the original description data and the extended description data corresponding to the same icon are semantically consistent but have different text lengths.

6. The method according to claim 4, characterized in that The step of acquiring a stable diffusion model integrated with an image prompt adapter specifically includes: Obtain an initial stable diffusion model; The initial stable diffusion model is pre-trained using a second training sample set to obtain a stable diffusion model; a training sample in the second training sample set includes a third icon and third text description information; the third text description information is description information for the content in the third icon; An image prompt adapter is integrated with the stable diffusion model to obtain the stable diffusion model integrated with the image prompt adapter; the image prompt adapter is used to provide the feature vector extracted from the icon to the U-Net module of the stable diffusion model.

7. The method according to claim 6, characterized in that The method of pre-training the initial stable diffusion model using the second training sample set to obtain the stable diffusion model specifically includes: Inputting the third icon into the image encoder of the initial stable diffusion model to obtain a third icon feature vector; Inputting the third text description information into the text encoder of the initial stable diffusion model to obtain a third text feature vector; Inputting the third icon feature vector and the third text feature vector into the U-Net module of the initial stable diffusion model to obtain a second feature vector output by the U-Net module; parsing the second characteristic vector to obtain a first prediction icon output by the initial stable diffusion model; According to a first loss value between the first predicted icon and the third icon, parameters in the initial stable diffusion model are adjusted to obtain a trained stable diffusion model.

8. The method according to claim 4, characterized in that The step of inputting the first training sample set into the stable diffusion model integrated with the image prompt adapter specifically includes: For a training sample in the first training sample set, inputting the first icon into an image encoder of the stable diffusion model; Inputting the first text description information into a text encoder of the stable diffusion model; The second icon is input to an image encoder of the image prompt adapter.

9. The method according to claim 8, characterized in that The image prompt adapter includes an image encoder and an adapter layer; the adapter layer is embedded in the U-Net module of the stable diffusion model; The U-Net module includes a cross attention layer and the adapter layer; After inputting the first training sample set into the stable diffusion model integrated with the image prompt adapter, the method further includes: Acquire a first icon feature vector output by the image encoder of the stable diffusion model and a first text feature vector output by the text encoder of the stable diffusion model; Inputting the first icon feature vector and the first text feature vector into the cross attention layer to obtain a third feature vector output by the cross attention layer; Obtaining a second icon feature vector output by the image encoder of the image prompt adapter; Inputting the third feature vector and the second icon feature vector into the adapter layer to obtain an icon prediction vector output by the adapter layer; The icon prediction vector is parsed to obtain a second prediction icon output by the stable diffusion model.

10. The method according to claim 9, characterized in that The adjusting the parameters in the image prompt adapter according to the output of the stable diffusion model to obtain a trained icon generation model specifically includes: Determine a second loss value between the second prediction icon output by the stable diffusion model and the first icon; the second loss value is used to reflect the size of the difference between the second prediction icon and the first icon; Taking minimizing the second loss value as an adjustment target, the parameters in the image prompt adapter are adjusted to obtain a trained icon generation model.

11. An icon generating device, characterized in that: The device uses a stable diffusion model integrated with an image prompt adapter, and the device includes: An acquisition module is used to obtain the style reference icon and text description information input by the user; A first input module, used for inputting the style reference icon into the image prompt adapter to obtain a style feature vector; A second input module, used for inputting the text description information into the text encoder of the stable diffusion model to obtain a text feature vector; A third input module, used to input the style feature vector and the text feature vector into the U-Net module of the stable diffusion model to obtain an icon prediction vector output by the U-Net module; A parsing module is used to parse the icon prediction vector to obtain at least one target icon that is consistent with the content of the text description information; the style of the target icon is consistent with the style of the style reference icon.

12. A training device for an icon generation model, characterized in that: The device comprises: A first acquisition module is used to acquire a stable diffusion model integrated with an image prompt adapter; the stable diffusion model is used to generate an icon corresponding to the text description information based on the text description information; A second acquisition module is used to acquire a first training sample set; a training sample in the first training sample set includes a first icon, a first text description information and a second icon; the first text description information is description information for the content in the first icon; the style of the second icon is consistent with the style of the first icon; A parameter adjustment module is used to input the first training sample set into the stable diffusion model integrated with the image prompt adapter, and adjust the parameters in the image prompt adapter according to the output of the stable diffusion model to obtain a trained icon generation model.

13. A computing device, characterized in that: include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the icon generation method described in any one of claims 1 to 3 are implemented, or the steps of the icon generation model training method described in any one of claims 4 to 10 are implemented.

14. A computer-readable storage medium, characterized in that: It stores computer executable instructions, which, when executed by a processor, implement the steps of the icon generation method described in any one of claims 1 to 3, or implement the steps of the icon generation model training method described in any one of claims 4 to 10.

Images