Personalized image generation method based on diffusion model and related equipment

Abstract

The invention belongs to the field of artificial intelligence and big data, is applied to the field of smart cities, and relates to a personalized image generation method based on a diffusion model, and the method comprises the steps: obtaining a personalized reference image and a reference prompt word; based on the personalized reference image and the reference cue word, performing first training on a pre-training diffusion model by using a personalized method to obtain a first diffusion model; performing second training on the first diffusion model by using a fractional sampling method to obtain a second diffusion model; and performing personalized image generation processing on a target personalized prompt word through the second diffusion model to obtain a target personalized image. The invention further provides a personalized image generation device based on the diffusion model, computer equipment and a storage medium. According to the method, the problems that stylization and other factors which are difficult to maintain exist in an existing personalized image generation method, and the personalized ability or alignment with complex text prompts is affected are solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence and big data technology, and in particular to a personalized image generation method based on a diffusion model and related equipment. Background Art

[0002] Text-to-image models have shown remarkable capabilities in generating diverse images in a variety of contexts (place, time, style, and appearance) including specific subjects. Recently, personalized approaches have even allowed specific subjects (objects, animals, or people) to be included in the generated images. Personalized approaches fine-tune a pre-trained text-to-image model on a set of personal images while relying heavily on regularization. However, it compromises the capture of the target subject's identifying features. On the other hand, insisting on recognition accuracy may hinder hinting capabilities. In general, the trade-off between identity preservation and hint alignment is a core challenge of personalized approaches. Content creators and AI artists often strive to create personalized images using personal subjects, which is beyond the capabilities of traditional text-to-image models.

[0003] Existing personalized image generation methods suffer from difficult-to-maintain stylization and other factors that affect personalization capabilities or alignment with complex text cues. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a personalized image generation method and related equipment based on a diffusion model to solve the problem that the existing personalized image generation methods have difficult-to-maintain stylization and other factors that affect the personalization ability or alignment with complex text prompts. Its main purpose is to enable the generation of target subjects by simultaneously optimizing two goals - personalization and prompt alignment - while ensuring that noise prediction is aligned with text prompts.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a personalized image generation method based on a diffusion model, which adopts the following technical solution:

[0006] Get personalized reference images and reference prompts;

[0007] Based on the personalized reference image and the reference prompt word, a pre-trained diffusion model is first trained using a personalized method to obtain a first diffusion model;

[0008] Performing a second training on the first diffusion model using a fractional sampling method to obtain a second diffusion model;

[0009] The personalized image generation process is performed on the target personalized prompt word through the second diffusion model to obtain a target personalized image.

[0010] Furthermore, the first training of the pre-trained diffusion model using a personalized method based on the personalized reference image and the reference prompt word to obtain a first diffusion model includes:

[0011] Performing a preprocessing operation on the personalized reference image to obtain a standard reference image;

[0012] Inputting the standard reference image and the reference prompt word into a pre-trained diffusion model;

[0013] The pre-trained diffusion model is first trained using a personalized method. During the training process, the network weights of the pre-trained model are fine-tuned. After the training is completed, a first diffusion model is obtained.

[0014] Furthermore, in the first training process, the pre-trained model is constrained using a loss function, and the formula of the loss function is as follows:

[0015]

[0016] Among them, y is the true value of the image, N is the number of samples, and Υ is the predicted value of the image.

[0017] Furthermore, the network weights of the pre-trained model include the weights of the self-attention layer and the weights of the cross-attention layer. The network weights of the pre-trained model are fine-tuned, and the training is completed to obtain a first diffusion model, including:

[0018] Determine a first update matrix of a weight matrix of a self-attention layer, and perform a first low-rank decomposition on the first update matrix to obtain a first decomposition matrix;

[0019] Determine a second update matrix for the weight matrix of the cross attention layer, and perform a second low-rank decomposition on the second update matrix to obtain a second decomposition matrix;

[0020] The first decomposition matrix and the second decomposition matrix are first trained using the standard reference image and the reference prompt words to obtain a first diffusion model.

[0021] Further, performing a first low-rank decomposition on the first update matrix to obtain a first decomposition matrix includes:

[0022] Determine the rank of the first update matrix using a singular value decomposition method, wherein the rank of the first update matrix is ​​smaller than the rows and columns of the weight matrix of the self-attention layer;

[0023] Perform a first low-rank decomposition on the first update matrix according to the rank of the first update matrix to obtain a first decomposition matrix.

[0024] Further, performing a second low-rank decomposition on the second update matrix to obtain a second decomposition matrix includes:

[0025] Determine the rank of the second update matrix using a principal component analysis method, wherein the rank of the second update matrix is ​​less than the rows and columns of the weight matrix of the cross-attention layer;

[0026] Perform a second low-rank decomposition on the second update matrix according to the rank of the second update matrix to obtain a second decomposition matrix.

[0027] Furthermore, the using a fractional sampling method to train the first diffusion model to obtain a second diffusion model includes:

[0028] Inputting the standard reference image and the reference prompt word into the first diffusion model;

[0029] The first diffusion model is subjected to a second training using a fractional sampling method. During the training process, the prediction is pushed toward the target prompt using fractional sampling. After the training is completed, a second diffusion model is obtained.

[0030] In order to solve the above technical problems, the present application also provides a personalized image generation device based on a diffusion model, which adopts the following technical solution:

[0031] An acquisition module, used for acquiring personalized reference images and reference prompt words;

[0032] A first training module, configured to perform a first training on the pre-trained diffusion model using a personalized method based on the personalized reference image and the reference prompt word to obtain a first diffusion model;

[0033] A second training module, configured to perform a second training on the first diffusion model using a fractional sampling method to obtain a second diffusion model;

[0034] The generation processing module is used to perform personalized image generation processing on the target personalized prompt word through the second diffusion model to obtain a target personalized image.

[0035] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0036] The computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the method for generating a personalized image based on a diffusion model described in any one of the embodiments of the present application are implemented.

[0037] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0038] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the method for generating a personalized image based on a diffusion model described in any one of the embodiments of the present invention are implemented.

[0039] Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects: obtaining a personalized reference image and a reference prompt word; based on the personalized reference image and the reference prompt word, performing a first training on the pre-trained diffusion model using a personalized method to obtain a first diffusion model; performing a second training on the first diffusion model using a fractional sampling method to obtain a second diffusion model; performing personalized image generation processing on the target personalized prompt word through the second diffusion model to obtain a target personalized image, which solves the problem that the existing personalized image generation method has difficult-to-maintain stylization and other factors that affect the personalization ability or alignment with complex text prompts. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0042] Figure 2 is a flow chart of an embodiment of a method for generating personalized images based on a diffusion model according to the present application;

[0043] Figure 3 yes Figure 2 A flowchart of a specific implementation of step S202;

[0044] Figure 4 yes Figure 3 A flowchart of a specific implementation of step S2023;

[0045] Figure 5 yes Figure 4 A flowchart of a specific implementation method of step S20231;

[0046] Figure 6 yes Figure 4 A flowchart of a specific implementation method of step S20232;

[0047] Figure 7 yes Figure 2A flowchart of a specific implementation method of step S203;

[0048] Figure 8 It is a structural framework diagram of a personalized image generation method based on a diffusion model according to the present application;

[0049] Fig. 9 is a structural schematic diagram of a specific implementation of a personalized image generation device 800 based on a diffusion model according to the present application;

[0050] Fig.10 yes Fig. 9 A schematic diagram of a structure of a specific implementation of the first training module 902;

[0051] Fig.11 yes Fig.10 A structural diagram of a specific implementation of the first training submodule 9023;

[0052] Fig.12 yes Fig.11 A schematic structural diagram of a specific implementation of the first low-rank decomposition unit 90231;

[0053] Fig.13 yes Fig.11 A schematic structural diagram of a specific implementation of the second low-rank decomposition unit 90232;

[0054] Fig.14 yes Fig. 9 A structural diagram of a specific implementation of the second training module 903;

[0055] Fig.15 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0057] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0058] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0059] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0060] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0061] The terminal devices 101, 102, and 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0062] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0063] It should be noted that the personalized image generation method based on the diffusion model provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the personalized image generation device based on the diffusion model is generally arranged in the server / terminal device.

[0064] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0065] Continue to refer Figure 2 , shows a flow chart of an embodiment of a personalized image generation method based on a diffusion model according to the present application. The personalized image generation method based on a diffusion model comprises the following steps:

[0066] Step S201: Obtain personalized reference images and reference prompt words.

[0067] In this embodiment, the method for generating a personalized image based on a diffusion model is executed on an electronic device (eg Figure 1 The server / terminal device shown in the figure can receive the personalized image generation request based on the diffusion model from the terminal device through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, Wi-Fi connection, Bluetooth connection, Wi MAX connection, Zigbee connection, UWB (Ultra Wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0068] The above-mentioned terminal devices may be terminal devices equipped with remote interaction functions such as personal assistants and intelligent customer service.

[0069] The above-mentioned personalized image generation method based on diffusion model can be applied to online document image generation, online document management and other businesses.

[0070] Specifically, the above electronic equipment can be used in different industry types such as medical image generation, property insurance image generation, financial image generation, store image generation, etc.

[0071] The personalized reference image is a personalized image with personalized characteristics generated according to the reference prompt word. The personalized reference image can be a group or a single image.

[0072] The reference words may be any descriptive language, such as "beach under sunset", "snow scene in winter" and so on.

[0073] It should be noted that this application allows combining different themes into coherent scenes or using a single artwork as a reference for generating artistic inspiration images.

[0074] Step S202: Based on the personalized reference image and the reference prompt word, a diffusion model is pre-trained using a personalized method to obtain a first diffusion model.

[0075] In this embodiment, after obtaining the personalized reference image and the reference prompt word, the electronic device may train the pre-trained diffusion model using a personalized method, thereby obtaining a first diffusion model.

[0076] The above pre-trained diffusion model is a tool for generating high-quality images. The diffusion model is a machine learning model used to process the relationship between images and text. The diffusion model can convert text information into image information, and also convert image information into text information.

[0077] The above personalization method adjusts the pre-trained text-to-image model on a set of personal images, while relying heavily on regularization. Personalization methods can transform or select the structure and input features of the model according to the characteristics and distribution of the data to improve the model's fit to the data; they can systematically adjust the model's hyperparameters, such as learning rate, regularization coefficient, etc., to find the best parameter combination and improve model performance; they can enhance the training data by performing operations such as rotation, flipping, and cropping to increase the diversity of the data, thereby expanding the training set and helping the model learn more features and change patterns. Personalization methods can adjust different aspects of the model according to the needs of specific tasks or data sets, aiming to improve the performance and generalization ability of the model.

[0078] During the training process, the network weights of the pre-trained model can be fine-tuned, and the training is completed to obtain the first diffusion model. The network weights of the pre-trained model include the weights of the self-attention layer and the weights of the cross-attention layer. The self-attention layer is a key component in the model. The main idea of ​​the self-attention layer is to associate elements at different positions in the sequence in order to calculate the representation of the same sequence. The self-attention layer can capture the long-distance dependencies of elements in the sequence, that is, the possible interactions between elements at different positions in the sequence. The self-attention layer enables the model to focus on the important parts of the input sequence and ignore the unimportant parts, thereby improving the processing power of the model. The cross-attention layer is used to establish the association between two different input sequences and calculate the attention weights between them.

[0079] The first diffusion model mentioned above can both learn the unique features of a subject from a small number or even a single input image and generate richer scenes.

[0080] It should be noted that when the pre-trained diffusion model is trained through the personalized method, multi-topic personalization can be achieved with a slight modification of the prompt words. The ability to adapt to multi-topic personalization can use the self-attention layer to capture the information within each position, and use the cross-attention layer to capture the information between different positions, so that the model can handle various types of data more flexibly and improve its performance and generalization ability.

[0081] Step S203: Perform a second training on the first diffusion model using a fractional sampling method to obtain a second diffusion model.

[0082] In this embodiment, the above-mentioned fractional sampling method is a method of sampling from the output of the model so that the sampling result is closer to the target distribution.

[0083] Specifically, fractional sampling can be used to shift the prediction toward the target cue, pushing the noisy prediction of the diffusion model toward the target cue and keeping the model aligned with the target cue.

[0084] The second diffusion model is trained based on the first diffusion model and personalized reference images and prompt words, and uses a fractional sampling method. The second diffusion model has better generalization ability and diversity, which can ensure that the generated images are closely aligned with the prompts required by the user, and can more accurately and faithfully represent the expected visual content based on the provided text description.

[0085] Step S204: Perform personalized image generation processing on the target personalized prompt word through the second diffusion model to obtain a target personalized image.

[0086] In this embodiment, the above-mentioned target personalized prompt words can be understood as prompt words that need to be generated by personalized images, including some text, labels or description information, such as a person's name, occupation, hobbies, etc., or some abstract descriptions, such as "beach under sunset", "winter snow scene", etc.

[0087] The above-mentioned personalized image generation process can be understood as a process of adding a target personalized prompt word during the image generation process to generate an image that meets the personalized requirements.

[0088] This application uses fractional sampling to keep the personalized model aligned with the target prompt, and is able to create images with complex and complicated prompts. By using fractional sampling methods, this application enables the model to learn the unique characteristics of the subject and prevents the model from losing its understanding of the target prompt. This application can not only learn the unique characteristics of the subject from a small number or even a single input image, but also generate richer scenes that are better aligned with the prompts required by the user.

[0089] In this embodiment, the application expands the scope of personalization by allowing different themes to be combined into coherent scenes or using a single artwork as a reference for generating artistic inspiration images, opening up new possibilities for creating diverse and complex visual content. The application can achieve multi-theme personalization with slight modifications and has the ability to adapt to multi-theme personalization.

[0090] The present application obtains personalized reference images and reference prompt words, and according to the personalized reference images and reference prompt words, uses self-attention and cross-attention layers to train a pre-trained diffusion model to obtain a first diffusion model, and uses a fractional sampling method to train the first diffusion model to obtain a second diffusion model. The second diffusion model is used to perform personalized image generation processing on the target personalized prompt words, thereby obtaining a target personalized image, which solves the problem that the existing personalized image generation methods have difficult-to-maintain stylization and other factors that affect the personalization ability or alignment with complex text prompts.

[0091] Continue to refer Figure 3 , shown in Figure 2 Flow chart of a specific implementation of step S202 in FIG. Step S202 specifically includes the following steps:

[0092] Step S2021: preprocess the personalized reference image to obtain a standard reference image.

[0093] In this embodiment, the personalized reference image is a personalized image with personalized characteristics generated according to the reference prompt word. The personalized reference image can be a group or a single image.

[0094] The purpose of the above-mentioned preprocessing operation is to make the personalized reference image closer to or reach the state of the standard reference image. The above-mentioned preprocessing operation can be a denoising process, a grayscale process, a resolution standardization process and the like. The above-mentioned denoising process can be understood as eliminating random noise in the image. The denoising process is to extract the main information in the image to make the image clearer. The above-mentioned grayscale process can be understood as converting the image into black and white or gray. The grayscale process is used to reduce the amount of image data or to highlight certain features in a specific image analysis task. The above-mentioned resolution standardization process can be understood as multiplying all pixel values ​​of the image by a fixed coefficient so that all pixel values ​​are within a specific range, such as 0 to 255. This can standardize the resolution of the image, that is, all pixel values ​​are on the same scale.

[0095] The above-mentioned standard reference image can be understood as a reference image in an ideal state, having consistent visual effects and features.

[0096] Step S2022: input the standard reference image and the reference prompt words into the pre-trained diffusion model.

[0097] In this embodiment, after obtaining the standard reference image and the reference prompt word, the electronic device may input the standard reference image and the reference prompt word into the pre-trained diffusion model.

[0098] The above standard reference image is an ideal reference image with consistent visual effects and features.

[0099] The reference words may be any descriptive language, such as "beach under sunset", "snow scene in winter" and so on.

[0100] Furthermore, during the training process, standard reference images and reference cue words are input into the pre-trained diffusion model to guide the diffusion model to generate images.

[0101] The above pre-trained diffusion model is a tool for generating high-quality images. The diffusion model is a machine learning model used to process the relationship between images and text. The diffusion model can convert text information into image information, and also convert image information into text information.

[0102] Step S2023: Perform a first training on the pre-trained diffusion model using a personalized method. During the training process, fine-tune the network weights of the pre-trained model. After the training is completed, a first diffusion model is obtained.

[0103] In this embodiment, the above-mentioned personalization method adjusts the pre-trained text-to-image model on a set of personal images, while relying heavily on regularization. The personalization method can transform or select the structure and input features of the model according to the characteristics and distribution of the data to improve the fit of the model to the data; the hyperparameters of the model, such as learning rate, regularization coefficient, etc., can be systematically adjusted to find the best parameter combination to improve model performance; the training data can be enhanced, such as rotation, flipping, cropping, etc., to increase the diversity of the data, thereby expanding the training set and helping the model learn more features and change patterns. The personalization method can adjust different aspects of the model according to the needs of a specific task or data set, aiming to improve the performance and generalization ability of the model.

[0104] The network weights of the above pre-trained model include the weights of the self-attention layer and the weights of the cross-attention layer. The above self-attention layer is a key component in the model. The main idea of ​​the self-attention layer is to associate elements at different positions in the sequence in order to calculate the representation of the same sequence. The self-attention layer can capture the long-range dependencies of elements within the sequence, that is, the possible interactions between elements at different positions in the sequence. The self-attention layer enables the model to focus on the important parts of the input sequence and ignore the unimportant parts, thereby improving the processing power of the model. The above cross-attention layer is used to establish the association between two different input sequences and calculate the attention weights between them.

[0105] The present application obtains a standard reference image by preprocessing a personalized reference image, inputs the standard reference image and a reference prompt word into a pre-trained diffusion model, and performs a first training on the pre-trained diffusion model using a personalized method. During the training process, the network weights of the pre-trained model are fine-tuned. After the training is completed, a first diffusion model is obtained. The first diffusion model can learn the unique features of a subject from a small number or even a single input image, and can generate richer scenes.

[0106] Optionally, in the first training process, a loss function is used to constrain the pre-trained model. The formula of the loss function is as follows:

[0107]

[0108] Among them, y is the true value of the image, N is the number of samples, and Υ is the predicted value of the image.

[0109] In this embodiment, the above loss function is used to measure the quantitative index of the difference between the model prediction value and the true value. The above loss function can be a reconstruction loss function, etc. The reconstruction loss function calculates the difference between the original input data and the data generated by the model, and minimizes this difference to enable the model to better fit the data distribution.

[0110] It should be noted that the loss function calculates the "loss" or "cost" between a predicted value and its corresponding true value as a measure of the accuracy of the model. By minimizing the value of the loss function, the optimal solution for the model parameters can be found, thereby improving the prediction accuracy of the model.

[0111] In the first training process, this application uses a loss function and a pre-trained model for constraints. The formula of the loss function is as follows:

[0112]

[0113] Among them, y is the true value of the image, N is the number of samples, and Υ is the predicted value of the image. During the training process, the value of the loss function is minimized, and the model can improve the prediction accuracy of the model.

[0114] Continue to refer Figure 4 , showing that Figure 3 Flow chart of a specific implementation of step S2023 in FIG. Step S2023 specifically includes the following steps:

[0115] Step S20231, determine the first update matrix of the weight matrix of the self-attention layer, perform a first low-rank decomposition on the first update matrix, and obtain a first decomposition matrix.

[0116] In this embodiment, the weight matrix is ​​used to represent the matrix of the relationship between features. Each row of the weight matrix corresponds to a factor or input variable, and each column corresponds to a result or output variable. For example, if the input layer has N elements and the output layer has M elements, then the weight matrix is ​​N rows * M columns.

[0117] The above-mentioned first update matrix can be understood as a matrix used for updating in the weight matrix of the self-attention layer of the pre-trained diffusion model.

[0118] The above low-rank decomposition can be understood as decomposing a matrix into the product of two or more matrices, where the decomposed matrix has a lower rank.

[0119] The first decomposition matrix is ​​a matrix obtained by low-rank decomposition of the first update matrix of the weight matrix of the self-attention layer. Specifically, when the first update matrix of the weight matrix of the self-attention layer of the pre-trained diffusion model is low-rank decomposed, the number of training can be reduced and the training time can be saved.

[0120] Step S20232: determine a second update matrix of the weight matrix of the cross-attention layer, perform a second low-rank decomposition on the second update matrix, and obtain a second decomposition matrix.

[0121] In this embodiment, the second update matrix can be understood as a matrix used for updating in the weight matrix of the cross-attention layer of the pre-trained diffusion model.

[0122] The above-mentioned second decomposition matrix is ​​a matrix obtained by low-rank decomposition of the second update matrix of the weight matrix of the cross attention layer.

[0123] When the first update matrix of the weight matrix of the cross-attention layer of the pre-trained diffusion model is low-rank decomposed, the number of trainings can be reduced and the training time can be saved.

[0124] Step S20233: Perform a first training on the first decomposition matrix and the second decomposition matrix using a standard reference image and reference prompt words to obtain a first diffusion model.

[0125] It should be noted that during the training process, the parameters of the pre-trained diffusion model can be fine-tuned using methods that reduce the complexity of the model.

[0126] The present application determines a first update matrix of the weight matrix of the self-attention layer, performs a first low-rank decomposition on the first update matrix to obtain a first decomposition matrix, determines a second update matrix of the weight matrix of the cross-attention layer, performs a second low-rank decomposition on the second update matrix to obtain a second decomposition matrix, and performs a first training on the first decomposition matrix and the second decomposition matrix through standard reference images and reference prompt words to obtain a first diffusion model, which can help improve the performance of the model on specific tasks while reducing the computational complexity.

[0127] Continue to refer Figure 5 , showing that in Figure 4 Flow chart of a specific implementation of step S20231 in FIG. Step S20231 specifically includes the following steps:

[0128] Step S202311: Determine the rank of the first update matrix using a singular value decomposition method.

[0129] In this embodiment, the singular value decomposition method is a linear algebra method, which is used to decompose a matrix into the product of three matrices, that is, A = UΣV^T, where U, Σ and V are unitary matrices, Σ is a diagonal matrix, and the elements on its diagonal are called singular values. By calculating the singular values ​​and singular vectors of the matrix, the "low rank" representation of the original matrix A can be obtained.

[0130] The rank of the first update matrix above is smaller than the rows and columns of the weight matrix of the self-attention layer.

[0131] The above rank refers to the number of non-zero row vectors or column vectors in a matrix, which can also be understood as the number of linearly independent rows or columns in the matrix. The rank is an adjustable empirical parameter, which is smaller than the rows and columns of the parameter matrix.

[0132] Furthermore, the rank is an adjustable empirical parameter, which is usually much smaller than the rows and columns of the parameter matrix. Therefore, it can reduce the number of parameters involved in training and save training time.

[0133] Step S203212: Perform a first low-rank decomposition on the first update matrix according to the rank of the first update matrix to obtain a first decomposition matrix.

[0134] In this embodiment, the first low-rank decomposition can be understood as decomposing a matrix into the product of two or more matrices, wherein the decomposed matrix has a lower rank.

[0135] The above-mentioned first decomposition matrix can be understood as a matrix obtained by low-rank decomposition of the first update matrix of the weight matrix of the self-attention layer.

[0136] The present application determines the rank of the first update matrix by using a singular value decomposition method, and performs a first low-rank decomposition on the first update matrix according to the rank of the first update matrix to obtain a first decomposition matrix. Through low-rank decomposition, a smaller matrix can be used to approximate the original weight matrix, thereby reducing the computational complexity.

[0137] Continue to refer Figure 6 , showing that Figure 4 Flow chart of a specific implementation of step S20232 in FIG. Step S20232 specifically includes the following steps:

[0138] Step S202321: Use the principal component analysis method to determine the rank of the second update matrix.

[0139] In this embodiment, the principal component analysis method is used to reduce the dimension of the data while retaining the main information in the data as much as possible. The principal component analysis method can identify the main change direction in the data by projecting the original data into a new coordinate system.

[0140] The rank of the second update matrix is ​​smaller than the rows and columns of the weight matrix of the cross attention layer. The rank is an adjustable empirical parameter, which is usually much smaller than the rows and columns of the parameter matrix, so it can reduce the number of parameters involved in training and save training time.

[0141] Step S202322: Perform a second low-rank decomposition on the second update matrix according to the rank of the second update matrix to obtain a second decomposition matrix.

[0142] In this embodiment, the second decomposition matrix can be understood as a matrix obtained by low-rank decomposition of the first update matrix of the weight matrix of the cross-attention layer.

[0143] This application determines the rank of the second update matrix by using a principal component analysis method, and performs a second low-rank decomposition on the second update matrix according to the rank of the second update matrix to obtain a second decomposition matrix. The efficiency and accuracy of the model can be improved by performing a low-rank decomposition on the first update matrix through the rank of the second update matrix.

[0144] Continue to refer Figure 7 , showing Figure 2 Flow chart of a specific implementation manner of step S203 in FIG. Step S203 specifically includes the following steps:

[0145] Step S2031: input the standard reference image and the reference prompt words into the first diffusion model.

[0146] In this embodiment, the above-mentioned standard reference image can be understood as a reference image in an ideal state, with consistent visual effects and features; the above-mentioned reference prompt words can be any descriptive language, such as "beach under sunset", "snow scene in winter", etc.

[0147] The first diffusion model is obtained by inputting a standard reference image and a reference prompt word into a pre-trained diffusion model, and training the pre-trained diffusion model using self-attention and cross-attention layers.

[0148] Step S2032: Perform a second training on the first diffusion model using the fractional sampling method. During the training process, the prediction is pushed toward the target prompt using the fractional sampling method. After the training is completed, a second diffusion model is obtained.

[0149] In this embodiment, the above-mentioned fractional sampling method is a method of sampling from the output of the model so that the sampling result is closer to the target distribution.

[0150] Specifically, fractional sampling can be used to shift the prediction toward the target cue, pushing the noisy prediction of the diffusion model toward the target cue.

[0151] The above-mentioned second diffusion model is trained based on the first diffusion model and standard reference images and prompt words, and uses a fractional sampling method. The second diffusion model has better generalization ability and diversity, which can ensure that the generated images are closely aligned with the prompts required by the user, and can more accurately and faithfully represent the expected visual content based on the provided text description.

[0152] The present application inputs a standard reference image and a reference prompt word into a first diffusion model, and uses a fractional sampling method to perform a second training on the first diffusion model. During the training process, the prediction is pushed toward the target prompt using the fractional sampling method. After the training is completed, a second diffusion model is obtained. The second diffusion model has better generalization ability and diversity, and can ensure that the generated image is closely aligned with the prompt required by the user, and can more accurately and faithfully represent the expected visual content based on the provided text description.

[0153] like Figure 8 As shown, a structural framework diagram of a personalized image generation method based on a diffusion model according to the present application is shown. Specifically, it includes step 1: inputting a personalized reference image; step 2: adding noise to the image and feeding it into a pre-trained diffusion model; step 3: fine-tuning the pre-trained diffusion model; step 4: aligning text prompts using fractional sampling; step 5: generating a target personalized image.

[0154] In this embodiment, the personalized reference image is a personalized image with personalized characteristics generated according to the reference prompt word. The personalized reference image is used as the basis for generating the target personalized image. The personalized reference image can be a group or a single image.

[0155] The above-mentioned image noise addition and feeding into the pre-trained diffusion model can be understood as the personalized reference image being processed by noise addition, that is, adding noise to the image, and then inputting the noisy image into the pre-trained diffusion model. Furthermore, the target subject is reconstructed in the personalized reference image using a personalized method, and an image with complex and complicated prompts is created, so that the model can learn the unique features of the subject and prevent the model from losing its understanding of the target prompt. This can not only learn the unique features of the subject from a small number or even a single input image, but also generate richer scenes and better align with the prompts required by the user.

[0156] The above personalization method adjusts the pre-trained text-to-image model on a set of personal images, while relying heavily on regularization. Personalization methods can transform or select the structure and input features of the model according to the characteristics and distribution of the data to improve the model's fit to the data; they can systematically adjust the model's hyperparameters, such as learning rate, regularization coefficient, etc., to find the best parameter combination and improve model performance; they can enhance the training data by performing operations such as rotation, flipping, and cropping to increase the diversity of the data, thereby expanding the training set and helping the model learn more features and change patterns. Personalization methods can adjust different aspects of the model according to the needs of specific tasks or data sets, aiming to improve the performance and generalization ability of the model.

[0157] The above pre-trained diffusion model is a tool for generating high-quality images. The diffusion model is a machine learning model used to process the relationship between images and text. The diffusion model can convert text information into image information, and can also convert image information into text information.

[0158] The above-mentioned fine-tuning of the pre-trained diffusion model can be understood as fine-tuning the parameters of the pre-trained diffusion model using a method that reduces the complexity of the model to obtain the first diffusion model. Specifically, the network weight parameters of the pre-trained diffusion model are fine-tuned. The network weights of the pre-trained model include the weights of the self-attention layer and the weights of the cross-attention layer. The above-mentioned self-attention layer is a key component in the model. The main idea of ​​the self-attention layer is to associate elements at different positions in the sequence in order to calculate the representation of the same sequence. The self-attention layer can capture the long-distance dependencies of elements in the sequence, that is, the possible interactions between elements at different positions in the sequence. The self-attention layer enables the model to focus on the important parts of the input sequence and ignore the unimportant parts, thereby improving the processing power of the model. The above-mentioned cross-attention layer is used to establish an association between two different input sequences and calculate the attention weights between them.

[0159] Furthermore, this application uses a loss function to constrain the pre-trained diffusion model. The formula of the loss function is as follows:

[0160]

[0161] Among them, y is the true value of the image, N is the number of samples, and Υ is the predicted value of the image.

[0162] The above loss function is used to measure the quantitative index of the difference between the model prediction value and the true value. The above loss function can be a reconstruction loss function, etc. The reconstruction loss function calculates the difference between the original input data and the data generated by the model, and minimizes this difference to enable the model to better fit the data distribution.

[0163] It should be noted that the loss function calculates the "loss" or "cost" between a predicted value and its corresponding true value as a measure of the accuracy of the model. By minimizing the value of the loss function, the optimal solution for the model parameters can be found, thereby improving the prediction accuracy of the model.

[0164] The first diffusion model mentioned above can both learn the unique features of a subject from a small number or even a single input image and generate richer scenes.

[0165] The above-mentioned alignment of text prompts with fractional sampling can be understood as using the fractional sampling method to perform a second training on the first diffusion model, using the fractional sampling method to turn the prediction in the direction of the target prompt, and the training is completed to obtain the second diffusion model. The noise prediction of the first diffusion model is pushed toward the target prompt. Specifically, the fractional sampling method is used to guide the first diffusion model's estimate of the target topic to the distribution density. By using the fractional sampling method, it is possible to generate personalized target topics through the reverse diffusion process while ensuring that the noise prediction is aligned with the text.

[0166] The above-mentioned fractional sampling method is a method of sampling from the output of the model so that the sampling result is closer to the target distribution.

[0167] The second diffusion model is trained based on the first diffusion model and personalized reference images and prompt words, and uses a fractional sampling method. The second diffusion model has better generalization ability and diversity, which can ensure that the generated images are closely aligned with the prompts required by the user, and can more accurately and faithfully represent the expected visual content based on the provided text description.

[0168] The above target personalized image generation can be understood as performing personalized image generation processing on the target personalized prompt word through the second diffusion model to obtain the target personalized image.

[0169] The above-mentioned target personalized prompt words are prompt words that need to be generated by personalized images, including some text, labels or description information, such as a person's name, occupation, hobbies, etc., or some abstract descriptions, such as "beach under sunset", "winter snow scene", etc.

[0170] In this embodiment, the present application uses a fractional sampling method to keep the personalized model aligned with the target prompt, so that the model can learn the unique features of the subject and prevent the model from losing its understanding of the target prompt. It can learn the unique features of the subject from a small number or even a single input image, and can generate richer scenes, which are better consistent with the prompts required by the user. The present application can achieve multi-theme personalization with slight modifications and has the ability to adapt to multi-theme personalization. By allowing different themes to be combined into coherent scenes or using a single artwork as a reference for generating artistic inspiration images, the present application expands the scope of personalization and opens up new possibilities for creating diverse and complex visual content. The present application pushes the boundaries of personalization methods to handle complex prompts containing multiple subjects, even if a subject has only one reference image, and the present application also demonstrates its ability to solve challenges associated with complex prompts and diverse subjects. The present application does not require pre-training on large-scale data, enhances the usability and versatility of the method, and reduces the entry threshold for individuals or organizations seeking to utilize personalized image generation without a lot of pre-training work.

[0171] In this embodiment, the non-Company software tools or components that appear in the embodiments of this application are merely examples and do not represent actual use.

[0172] This embodiment can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0173] The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning. This application can be applied to the fields of artificial intelligence and big data, thereby promoting the construction of smart cities.

[0174] This application belongs to the field of smart cities, and this solution can promote the construction of smart cities.

[0175] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0176] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0177] Further references Fig. 9 , as a response to the above Figure 2 The present application provides an embodiment of a personalized image generation device based on a diffusion model, and the personalized image generation device embodiment based on the diffusion model is Figure 2Corresponding to the embodiment of the personalized image generation method based on the diffusion model shown, the personalized image generation device based on the diffusion model can be specifically applied to various electronic devices.

[0178] like Fig. 9 As shown, the personalized image generation device 900 based on the diffusion model described in this embodiment includes: an acquisition module 901, a first training module 902, a second training module 903 and a generation processing module 904. Among them:

[0179] An acquisition module 901 is used to acquire a personalized reference image and a reference prompt word;

[0180] A first training module 902 is used to perform a first training on the pre-trained diffusion model using a personalized method based on the personalized reference image and the reference prompt word to obtain a first diffusion model;

[0181] The second training module 903 is used to perform a second training on the first diffusion model using a fractional sampling method to obtain a second diffusion model.

[0182] The generation processing module 904 is used to perform personalized image generation processing on the target personalized prompt word through the second diffusion model to obtain a target personalized image.

[0183] In this embodiment, a personalized reference image and a reference prompt word are obtained, and a pre-trained diffusion model is first trained using a personalized method according to the personalized reference image and the reference prompt word to obtain a first diffusion model, and a fractional sampling method is used to perform a second training on the first diffusion model to obtain a second diffusion model. The second diffusion model is used to perform personalized image generation processing on the target personalized prompt word to obtain a target personalized image, thereby solving the problem that the existing personalized image generation method has difficult-to-maintain stylization and other factors that affect the personalization ability or alignment with complex text prompts.

[0184] See also Fig.10 ,for Fig. 9 The first training module 902 is a schematic diagram of a specific implementation scheme, wherein the first training module 902 includes a preprocessing operation submodule 9021, a first input submodule 9022, and a first training submodule 9023.

[0185] A preprocessing operation submodule 9021 is used to perform a preprocessing operation on the personalized reference image to obtain a standard reference image;

[0186] A first input submodule 9022, used to input the standard reference image and the reference prompt word into the pre-trained diffusion model;

[0187] The first training submodule 9023 is used to perform a first training on the pre-trained diffusion model using a personalized method. During the training process, the network weights of the pre-trained model are fine-tuned. After the training is completed, a first diffusion model is obtained.

[0188] In this embodiment, a standard reference image is obtained by preprocessing the personalized reference image, the standard reference image and the reference prompt word are input into a pre-trained diffusion model, and the pre-trained diffusion model is trained using a personalized method. During the training process, the network weights of the pre-trained model are fine-tuned. After the training is completed, a first diffusion model is obtained. The first diffusion model can learn the unique features of the subject from a small number or even a single input image, and can generate richer scenes.

[0189] Optionally, in the first training process, a loss function is used to constrain the pre-trained model. The formula of the loss function is as follows:

[0190]

[0191] Among them, y is the true value of the image, N is the number of samples, and Υ is the predicted value of the image.

[0192] In the first training process, this application uses a loss function and a pre-trained model for constraints. The formula of the loss function is as follows:

[0193]

[0194] Among them, y is the true value of the image, N is the number of samples, and Υ is the predicted value of the image. During the training process, the value of the loss function is minimized, and the model can improve the prediction accuracy of the model.

[0195] See also Fig.11 ,for Fig.10 The first training submodule 9023 is a schematic diagram of a specific implementation scheme, wherein the first training submodule 9023 includes a first low-rank decomposition unit 90231, a second low-rank decomposition unit 90232, and a first training unit 90233.

[0196] A first low-rank decomposition unit 90231 is used to determine a first update matrix of the weight matrix of the self-attention layer, and perform a first low-rank decomposition on the first update matrix to obtain a first decomposition matrix;

[0197] A second low-rank decomposition unit 90232 is used to determine a second update matrix of the weight matrix of the cross attention layer, and perform a second low-rank decomposition on the second update matrix to obtain a second decomposition matrix;

[0198] The first training unit 90233 is used to perform a first training on the first decomposition matrix and the second decomposition matrix through the standard reference image and the reference prompt word to obtain a first diffusion model.

[0199] In this embodiment, by determining a first update matrix of the weight matrix of the self-attention layer, performing a first low-rank decomposition on the first update matrix to obtain a first decomposition matrix, determining a second update matrix of the weight matrix of the cross-attention layer, performing a second low-rank decomposition on the second update matrix to obtain a second decomposition matrix, and performing a first training on the first decomposition matrix and the second decomposition matrix through standard reference images and reference prompt words to obtain a first diffusion model, which can help improve the performance of the model on specific tasks while reducing the computational complexity.

[0200] See also Fig.12 ,for Fig.11 A schematic diagram of a structure of a specific implementation of the first low-rank decomposition unit 90231, wherein the first low-rank decomposition unit 90231 includes a first determination subunit 902311 and a first low-rank decomposition subunit 902312.

[0201] A first determining subunit 902311 is used to determine the rank of a first update matrix using a singular value decomposition method, where the rank of the first update matrix is ​​smaller than the rows and columns of the weight matrix of the self-attention layer;

[0202] The first low-rank decomposition subunit 902312 is used to perform a first low-rank decomposition on the first update matrix according to the rank of the first update matrix to obtain a first decomposition matrix.

[0203] In this embodiment, the rank of the first update matrix is ​​determined by utilizing a singular value decomposition method, and a first low-rank decomposition is performed on the first update matrix according to the rank of the first update matrix to obtain a first decomposition matrix. Through low-rank decomposition, a smaller matrix can be used to approximate the original weight matrix, thereby reducing computational complexity.

[0204] See also Fig.13 ,for Fig.11 A schematic diagram of a structure of a specific implementation of the second low-rank decomposition unit 93222 in FIG. 1 , wherein the second low-rank decomposition unit 90232 includes a second determination subunit 902321 and a second low-rank decomposition subunit 902322. Wherein:

[0205] A second determining subunit 902321 is used to determine the rank of a second update matrix using a principal component analysis method, wherein the rank of the second update matrix is ​​smaller than the rows and columns of the weight matrix of the cross attention layer;

[0206] The second low-rank decomposition subunit 902322 is used to perform a second low-rank decomposition on the second update matrix according to the rank of the second update matrix to obtain a second decomposition matrix.

[0207] In this embodiment, the rank of the second update matrix is ​​determined by using the principal component analysis method, and the second update matrix is ​​subjected to a second low-rank decomposition according to the rank of the second update matrix to obtain a second decomposition matrix. The efficiency and accuracy of the model can be improved by performing a low-rank decomposition of the first update matrix according to the rank of the second update matrix.

[0208] See also Fig.14 ,for Fig. 9 The second training module 903 is a schematic diagram of a specific implementation of the structure of the second training module 903, the second training module 903 includes a second input submodule 9031 and a second training submodule 9032. Wherein:

[0209] A second input submodule 9031 is used to input the standard reference image and the reference prompt word into the first diffusion model;

[0210] The second training submodule 9032 is used to perform a second training on the first diffusion model using a fractional sampling method. During the training process, the prediction is pushed toward the target prompt using the fractional sampling method. After the training is completed, a second diffusion model is obtained.

[0211] In this embodiment, a standard reference image and a reference prompt word are input into a first diffusion model, and a fractional sampling method is used to perform a second training on the first diffusion model. During the training process, the fractional sampling method is used to push the prediction toward the target prompt. After the training is completed, a second diffusion model is obtained. The second diffusion model has better generalization ability and diversity, and can ensure that the generated image is closely aligned with the prompt required by the user, and can more accurately and faithfully represent the expected visual content based on the provided text description.

[0212] To solve the above technical problems, the present application also provides a computer device. Fig.15 , Fig.15 This is a basic structural block diagram of the computer device in this embodiment.

[0213] The computer device 15 includes a memory 151, a processor 152, and a network interface 153 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 15 having components 151-153, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital processors (DSPs), embedded devices, etc.

[0214] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0215] The memory 151 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc.

[0216] In some embodiments, the memory 151 may be an internal storage unit of the computer device 15, such as a hard disk or memory of the computer device 15. In other embodiments, the memory 151 may also be an external storage device of the computer device 15, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 15. Of course, the memory 151 may also include both the internal storage unit of the computer device 15 and its external storage device. In this embodiment, the memory 151 is generally used to store the operating system and various application software installed on the computer device 15, such as computer-readable instructions of a personalized image generation method based on a diffusion model, etc. In addition, the memory 151 may also be used to temporarily store various types of data that have been output or are to be output. In some embodiments, the processor 152 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 152 is generally used to control the overall operation of the computer device 15. In this embodiment, the processor 152 is used to execute computer-readable instructions or process data stored in the memory 151, such as computer-readable instructions for executing the personalized image generation method based on the diffusion model.

[0217] The network interface 153 may include a wireless network interface or a wired network interface. The network interface 153 is generally used to establish a communication connection between the computer device 15 and other electronic devices.

[0218] In this embodiment, the impact data of the system in the knowledge base can be used to generate a personalized image based on the diffusion model that is affected by the target system to be tested and generate it for the user, so that the user can intuitively obtain the impact range of the target system to be tested. According to the impact range of the target system to be tested, the impact of the change can be evaluated more accurately and quickly, irrelevant interference points can be eliminated, and regression use cases can be streamlined to avoid missing impact points in the early stage of the evaluation, resulting in the subsequent development of various systems. When they are jointly debugged, it is found that some associations have not been processed. In the mildest case, it will take up development time to supplement the logic. In the most serious case, the solution design may need to be overturned and redone, thereby improving the efficiency of project development.

[0219] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the personalized image generation method based on the diffusion model as described above.

[0220] In this embodiment, the impact data of the system in the knowledge base can be used to generate a personalized image based on the diffusion model that is affected by the target system to be tested and generate it for the user, so that the user can intuitively obtain the impact range of the target system to be tested. According to the impact range of the target system to be tested, the impact of the change can be evaluated more accurately and quickly, irrelevant interference points can be eliminated, and regression use cases can be streamlined to avoid missing impact points in the early stage of the evaluation, resulting in the subsequent development of various systems. When they are jointly debugged, it is found that some associations have not been processed. In the mildest case, it will take up development time to supplement the logic. In the most serious case, the solution design may need to be overturned and redone, thereby improving the efficiency of project development.

[0221] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0222] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A personalized image generation method based on a diffusion model, characterized in that: The steps include: Get personalized reference images and reference prompts; Based on the personalized reference image and the reference prompt word, a pre-trained diffusion model is first trained using a personalized method to obtain a first diffusion model; Performing a second training on the first diffusion model using a fractional sampling method to obtain a second diffusion model; The personalized image generation process is performed on the target personalized prompt word through the second diffusion model to obtain a target personalized image.

2. The personalized image generation method based on the diffusion model according to claim 1, characterized in that: The first step of training the pre-trained diffusion model based on the personalized reference image and the reference prompt word by using a personalized method to obtain a first diffusion model includes: Performing a preprocessing operation on the personalized reference image to obtain a standard reference image; Inputting the standard reference image and the reference prompt word into a pre-trained diffusion model; The pre-trained diffusion model is first trained using a personalized method. During the training process, the network weights of the pre-trained model are fine-tuned. After the training is completed, a first diffusion model is obtained.

3. The personalized image generation method based on the diffusion model according to claim 2, characterized in that: In the first training process, the pre-trained model is constrained using a loss function, and the formula of the loss function is as follows: Among them, y is the true value of the image, N is the number of samples, and Υ is the predicted value of the image.

4. The personalized image generation method based on the diffusion model according to claim 2, characterized in that: The network weights of the pre-trained model include the weights of the self-attention layer and the weights of the cross-attention layer. The network weights of the pre-trained model are fine-tuned, and the training is completed to obtain a first diffusion model, including: Determine a first update matrix of a weight matrix of a self-attention layer, and perform a first low-rank decomposition on the first update matrix to obtain a first decomposition matrix; Determine a second update matrix for the weight matrix of the cross attention layer, and perform a second low-rank decomposition on the second update matrix to obtain a second decomposition matrix; The first decomposition matrix and the second decomposition matrix are first trained using the standard reference image and the reference prompt words, and after the training is completed, a first diffusion model is obtained.

5. The personalized image generation method based on the diffusion model according to claim 4 is characterized in that: The performing a first low-rank decomposition on the first update matrix to obtain a first decomposition matrix includes: Determine the rank of the first update matrix using a singular value decomposition method, wherein the rank of the first update matrix is ​​smaller than the rows and columns of the weight matrix of the self-attention layer; Perform a first low-rank decomposition on the first update matrix according to the rank of the first update matrix to obtain a first decomposition matrix.

6. The personalized image generation method based on the diffusion model according to claim 4, characterized in that: The performing a second low-rank decomposition on the second update matrix to obtain a second decomposition matrix includes: Determine the rank of the second update matrix using a principal component analysis method, wherein the rank of the second update matrix is ​​less than the rows and columns of the weight matrix of the cross-attention layer; Perform a second low-rank decomposition on the second update matrix according to the rank of the second update matrix to obtain a second decomposition matrix.

7. The personalized image generation method based on the diffusion model according to any one of claims 1 to 6, characterized in that: The step of training the first diffusion model using a fractional sampling method to obtain a second diffusion model includes: Inputting the standard reference image and the reference prompt word into the first diffusion model; The first diffusion model is subjected to a second training using a fractional sampling method. During the training process, the prediction is pushed toward the target prompt using the fractional sampling method. After the training is completed, a second diffusion model is obtained.

8. A personalized image generation device based on a diffusion model, characterized in that: include: An acquisition module, used for acquiring personalized reference images and reference prompt words; A first training module, configured to perform a first training on the pre-trained diffusion model using a personalized method based on the personalized reference image and the reference prompt word to obtain a first diffusion model; A second training module, configured to perform a second training on the first diffusion model using a fractional sampling method to obtain a second diffusion model; The generation processing module is used to perform personalized image generation processing on the target personalized prompt word through the second diffusion model to obtain a target personalized image.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the method for generating personalized images based on a diffusion model as claimed in any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the method for generating a personalized image based on a diffusion model according to any one of claims 1 to 7 are implemented.

Images