Multi-reward reinforcement learning methods and apparatus for customized text-based graph models

Abstract

This application provides a course-based multi-reward reinforcement learning method and apparatus for customized text-to-image (TPE) models, relating to the field of TPE technology, aiming to improve the performance of customized TPE models across multiple evaluation dimensions. The method includes: acquiring images of specific subjects generated by the customized TPE model based on prompt words; evaluating the images based on indicators across multiple evaluation dimensions to obtain evaluation values ​​for multiple indicators, wherein the multiple evaluation dimensions include: prompt word accuracy, subject accuracy, and human preference, with each evaluation dimension including at least one indicator; determining the weight of each indicator based on changes in the evaluation value of each indicator; generating a reward signal based on the evaluation values ​​of the multiple indicators and the weight of each indicator; and updating the model parameters of the customized TPE model based on the reward signal.

Description

Technical Field

[0001] This application relates to the field of text-based graph technology, and in particular to a course-based multi-reward reinforcement learning method and apparatus applied to customized text-based graph models. Background Technology

[0002] Text-based image generation is a task that generates corresponding images based on given prompts. Customized text-based image generation tasks further require that the generated images contain customized subjects from the user. Customized text-based image generation tasks can be accomplished through various customized text-based image generation models.

[0003] However, existing customized text-based image models fail to perform well across multiple evaluation dimensions, particularly in terms of cue word accuracy, subject accuracy, and human preference, often only achieving satisfactory results in one or two of these dimensions. Therefore, improving the performance of customized text-based image models across multiple evaluation dimensions is a pressing technical problem that needs to be solved. Summary of the Invention

[0004] In view of the above problems, embodiments of this application provide a course-based multi-reward reinforcement learning method and apparatus applied to customized text graph models, so as to overcome the above problems or at least partially solve the above problems.

[0005] A first aspect of this application discloses a course-based multi-reward reinforcement learning method applied to a customized text-based graph model, the method comprising:

[0006] Obtain images of specific subjects generated by a customized text-to-image model based on prompts;

[0007] The image is evaluated based on indicators of multiple evaluation dimensions to obtain evaluation values ​​for multiple indicators. The multiple evaluation dimensions include: accuracy of prompt words, accuracy of subject, and human preference. Each evaluation dimension includes at least one indicator.

[0008] The weight of each indicator is determined based on the changes in the evaluation value of each indicator;

[0009] A reward signal is generated based on the evaluation values ​​of multiple indicators and the weight of each indicator;

[0010] The model parameters of the customized text-based image model are updated based on the reward signal.

[0011] Optionally, the weight of each indicator is determined based on the changes in the evaluation value of each indicator, including:

[0012] For indicators whose evaluation values ​​are increasing, a smaller weight is assigned, meaning that the weight of the indicator is reduced compared to the previous update round.

[0013] For indicators whose evaluation values ​​show a decrease, a greater weight is assigned, meaning that the weight of the indicator is increased relative to the previous update round.

[0014] Optionally, the weight of each indicator is determined based on the changes in the evaluation value of each indicator, including:

[0015] Based on the moving average and moving average standard deviation of each indicator, a change score is obtained for each indicator. The magnitude of the change score represents the change in the evaluation value of the indicator.

[0016] The weight of each indicator is determined based on the change score of each indicator, and the weight of the indicator satisfies the following condition: the weight of the indicator with the smaller change score is greater than the weight of the indicator with the larger change score.

[0017] Optionally, the weight of each indicator is determined based on the change score of each indicator, including:

[0018] Based on the change scores of each indicator, a course learning model is constructed. The course learning model includes a first learning item and a second learning item. The first learning item is used to control the sum of the weights of all indicators to be minimized, and the second learning item is used to control the weight of each indicator to change within a preset range.

[0019] By solving the course learning model, the initial weight of each indicator is obtained;

[0020] The weight of each indicator is the ratio of its initial weight to the sum of the initial weights of all indicators.

[0021] Optionally, the moving average and moving average standard deviation of the indicators are calculated as follows:

[0022] Based on the evaluation value of the indicator and the moving average of the previous update round, the moving average of the indicator in the current update round is obtained.

[0023] The moving average standard deviation for the current update round is calculated based on the indicator's evaluation value, the moving average of the indicator in the current update round, and the moving average standard deviation of the indicator in the previous update round.

[0024] Optionally, the image is evaluated based on multiple evaluation dimensions to obtain evaluation values ​​for multiple metrics, including:

[0025] The similarity between the image and the prompt word is evaluated using a prompt word accuracy assessment model to obtain the evaluation value of the prompt word accuracy related indicators;

[0026] The subject accuracy assessment model is used to evaluate the similarity between the subject in the image and the customized subject, and the evaluation values ​​of the subject accuracy related indicators are obtained.

[0027] The human preference evaluation model is used to assess the degree of human satisfaction with the image and obtain the evaluation value of human preference-related indicators. The human preference evaluation model is a reward model trained based on human annotation information.

[0028] Optionally, the customized text-based image model is obtained through the following method:

[0029] Obtain sample images of a specific subject, and obtain prompt text containing the subject's identifier;

[0030] The sample image and the prompt text are input into the text-generated image model for noise addition and denoising to obtain the generated image.

[0031] Based on the noise added by the text image model in the noise addition process and the noise fitted by the text image model in the noise removal process, the loss function value between the sample image and the generated image is calculated.

[0032] The model parameters of the text image model are fine-tuned based on the loss function value to obtain the customized text image model.

[0033] Optionally, based on the noise added by the text-based image model in the denoising process and the noise fitted by the text-based image model in the denoising process, the loss function value between the sample image and the generated image is calculated, including:

[0034] Based on the noise added to a specific subject and the noise added to other subjects in the noise addition process by the text-based image model, and the noise fitted to the specific subject and the noise fitted to other subjects in the denoising process by the text-based image model, the loss function value between the sample image and the generated image is calculated.

[0035] A second aspect of this application discloses a course-based multi-reward reinforcement learning device applied to a customized text-based graph model, the device comprising:

[0036] The acquisition module is used to acquire images of specific subjects generated by the customized text-to-image model based on prompt words;

[0037] The evaluation module is used to evaluate the image based on indicators of multiple evaluation dimensions to obtain evaluation values ​​of multiple indicators. The multiple evaluation dimensions include: accuracy of prompt words, accuracy of subject, and human preference. Each evaluation dimension includes at least one indicator.

[0038] The determination module is used to determine the weight of each indicator based on the changes in the evaluation value of each indicator;

[0039] A generation module is used to generate a reward signal based on the evaluation values ​​of multiple indicators and the weight of each indicator;

[0040] The update module is used to update the model parameters of the customized text-based image model according to the reward signal.

[0041] A third aspect of this application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the course multi-reward reinforcement learning method applied to a customized text-based image model as described in the first aspect of this application.

[0042] A fourth aspect of this application discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the course multi-reward reinforcement learning method applied to a customized text-based graph model as described in the first aspect of this application.

[0043] A fifth aspect of this application discloses a computer program product, including a computer program that, when executed by a processor, implements the steps of the course multi-reward reinforcement learning method applied to a customized text-based graph model as described in the first aspect of this application.

[0044] The embodiments of this application have the following advantages:

[0045] In this embodiment, an image of a specific subject generated by a customized text-to-image (TPE) model based on prompt words is obtained. The image is then evaluated based on multiple evaluation dimensions, yielding evaluation values ​​for each dimension. These dimensions include prompt word accuracy, subject accuracy, and human preference, with each dimension comprising at least one indicator. The weight of each indicator is determined based on its evaluation value, and a reward signal is generated based on the evaluation values ​​and weights of all indicators. Finally, the model parameters of the customized TPE model are updated using the reward signal. By selecting multiple evaluation dimensions as the reward signal for the customized TPE model, optimization across multiple evaluation dimensions is achieved. The indicator weights are determined based on the changes in each indicator's evaluation value, thus resolving reward conflicts during the optimization process and dynamically and adaptively optimizing multiple indicators simultaneously, further improving the performance of the customized TPE model across multiple evaluation dimensions (e.g., prompt word accuracy, subject accuracy, and human preference). Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart illustrating the steps of a course-based multi-reward reinforcement learning method applied to a customized text-based graph model, as provided in an embodiment of this application.

[0048] Figure 2 This is a flowchart illustrating another step of a course-based multi-reward reinforcement learning method applied to a customized text-based graph model, as provided in this application embodiment.

[0049] Figure 3 This is an overall architecture diagram of a course multi-reward reinforcement learning method applied to a customized text-based graph model, provided in an embodiment of this application;

[0050] Figure 4 This is a schematic diagram of the structure of a course multi-reward reinforcement learning device applied to a customized text-based graph model, provided in an embodiment of this application.

[0051] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0052] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0053] Image generation is a task that generates images based on given prompts. Customized image generation tasks further require that the generated images contain a user-defined subject. For example, when a user wants to generate an image of a "puppy running," the "puppy" should be the dog specified by the user, not any other breed. While various customized image generation models can accomplish this task, the images they generate often struggle to perform well across multiple evaluation dimensions, particularly in terms of prompt accuracy, subject accuracy, and human preference. Prompt accuracy refers to whether the generated image matches the prompt. For example, if the prompt is "puppy running," but the generated image depicts a "puppy standing," it fails to meet the prompt accuracy requirement. Subject accuracy refers to whether the subject in the generated image is the specific subject provided by the user. For instance, if the user provides several pictures of their own Corgis, but the generated image shows a Teddy Bear dog, it fails to meet the subject accuracy requirement. Human preference refers to the image quality requirements. If the generated image shows a dog with distorted or illogical body parts, it fails to meet the human preference requirement. Existing customized text-to-image models only perform well on some evaluation dimensions, but poorly on others, and cannot meet all the above requirements at the same time.

[0054] Existing methods for creating customized text-to-image datasets can be categorized into three types based on control conditions: model-based methods, fine-tuning-based methods, and methods that require no training. The most commonly used method is the fine-tuning-based customized text-to-image dataset, which achieves customized generation by fine-tuning some parameters in the text-to-image model and binding the customized subject with special markers. Below is a brief description of these three types of methods:

[0055] The first category involves model-based methods that introduce additional models to encode customized conditions. These additional models are typically encoders, which input specially identifiable prompts and multiple customized subject images as vectors, which are then integrated into the text-to-image (TPI) process. One model-based approach uses subgraph encoders and concept encoders to encode personalized sample images and adds an adapter layer to the model to control customized conditions. These additional models (including the encoder and adapter) are trainable, while the other components of the model remain unchanged. This type of approach consumes more computational space and the pre-training process takes more time; however, once the encoder and adapter are fully trained, they can easily adapt to different customized conditions.

[0056] The second category: tuning-based methods do not require additional models. Instead, they fine-tune certain parts of the original text-based graph model to achieve customization. These tuning-based methods typically employ parameter-efficient fine-tuning techniques instead of traditional fine-tuning, thereby reducing computational resource consumption. While tuning-based methods save computational space by not introducing additional models, each fine-tuning can only adapt to a specific condition.

[0057] The third category: Training-free methods eliminate any training or fine-tuning process, instead utilizing the inherent capabilities of the text-based image model structure (e.g., attention mechanisms) to control generation. One training-free method achieves consistent style generation through minimal "attention sharing" during the diffusion process, where all images share self-attention with the reference image. Training-free methods require no additional model or fine-tuning time, but can only control a limited range of conditions, such as layout or style.

[0058] In summary, existing customized text-to-image generation methods typically fail to perform well across multiple evaluation dimensions, particularly in terms of cue word accuracy, subject accuracy, and human preference, often only achieving satisfactory results in one or two of these dimensions. Previous work has discussed this issue, with the general consensus being that pre-trained generative models tend to overfit to a limited number of sample images, or to information irrelevant to the subject matter within those images. While some methods have attempted to address overfitting through preservation mechanisms and decoupled learning, they have not yet been able to simultaneously improve the quality of generated images across multiple evaluation dimensions. Furthermore, achieving comprehensive optimization is a complex process due to potential conflicts between metrics across multiple evaluation dimensions.

[0059] To overcome the limitations of related technologies, this application provides a course-based multi-reward reinforcement learning method applied to customized text graph models. Specifically, a customized text graph model with customization capabilities is obtained, and multiple evaluation dimensions are used as reward signals. Reinforcement learning is then used to further fine-tune the customized text graph model to maximize rewards. To address the reward conflict problem in the multi-reward (indicator) optimization process, this application proposes a course reweighting mechanism. The weight of each indicator is determined based on the changes in the evaluation value of each indicator to adaptively balance the reward signals, ensuring that each reward signal is fully optimized.

[0060] Reference Figure 1 As shown, Figure 1 This is a flowchart illustrating the steps of a multi-reward reinforcement learning method for a customized text-based graph model, as provided in an embodiment of this application. Figure 1 As shown, the course-based multi-reward reinforcement learning method applied to a customized text-based graph model may include steps S110 to S150:

[0061] Step S110: Obtain an image of a specific subject generated by the customized text-to-image model based on the prompt words.

[0062] In this embodiment, the customized text-to-image model is obtained by fine-tuning a text-to-image model based on a customized subject. The customized text-to-image model refers to the ability to generate images of a specific subject (i.e., a customized subject). Specifically, obtaining the image of the specific subject generated by the customized text-to-image model based on prompts includes: inputting prompts into the customized text-to-image model, and the customized text-to-image model generating the image of the specific subject based on the prompts; wherein, the prompts are descriptive information about the image to be generated.

[0063] Step S120: Evaluate the image based on indicators of multiple evaluation dimensions to obtain evaluation values ​​of multiple indicators. The multiple evaluation dimensions include: accuracy of prompt words, accuracy of subject, and human preference. Each evaluation dimension includes at least one indicator.

[0064] In this embodiment, the evaluation metrics across multiple dimensions include metrics for the accuracy of the prompt words, the accuracy of the subject, and the human preference. Specifically, prompt word accuracy refers to whether the image meets the requirements of the prompt words, i.e., the degree of similarity between the image and the prompt words; subject accuracy refers to whether the subject in the image is a customized subject, i.e., the degree of similarity between the subject in the image and a customized subject; and human preference refers to the requirements for image quality, i.e., the degree to which humans are satisfied with the image.

[0065] The evaluation value of an indicator represents the degree to which an image satisfies that indicator. The greater the degree to which an image satisfies a particular indicator, the higher the evaluation value for that indicator. For example, the greater the similarity between an image and a prompt word, the higher the corresponding evaluation value. Thus, images generated by the customized text-to-image model are evaluated based on different indicators, and this feedback is used to improve the quality of the generated images based on the evaluation values ​​of multiple indicators.

[0066] Step S130: Determine the weight of each indicator based on the changes in the evaluation value of each indicator.

[0067] In this embodiment, after using the evaluation value of the indicator as the reward signal for reinforcement learning of the customized text graph model, the influence of different indicators on the customized text graph model may vary. If the evaluation values ​​of all indicators are directly added together as the final reward signal, conflicts may arise between different indicators. Consequently, the optimization method for the customized text graph model may focus only on one or more indicators, rather than striving to optimize all indicators simultaneously. Therefore, this embodiment employs a course-based multi-reward reinforcement learning method to determine the weight of each indicator based on the changes in its evaluation value.

[0068] Specifically, the weight of each indicator is determined based on the change in the evaluation value of each indicator, including: assigning a smaller weight to indicators whose evaluation value is increasing, whereby the smaller weight means that the weight of the indicator is reduced relative to the previous update round; and assigning a larger weight to indicators whose evaluation value is decreasing, whereby the larger weight means that the weight of the indicator is increased relative to the previous update round.

[0069] In this way, multiple metrics are balanced for the customized text-to-image (TPE) model. When the TPE model performs poorly on one or more metrics, the weights of those metrics are increased, thereby dynamically and adaptively optimizing multiple reward signals (metrics) simultaneously. For example, after a certain update, the TPE model's generated images show significant improvement in cue word accuracy and human preference, but a decline in subject accuracy. In this case, the weight of the metric corresponding to subject accuracy is increased, making the TPE model focus more on the subject accuracy of the generated images, thus comprehensively improving the quality of images generated by the TPE model.

[0070] Step S140: Generate a reward signal based on the evaluation values ​​of the multiple indicators and the weight of each indicator.

[0071] In this embodiment of the application, after obtaining the weight of each indicator, a reward signal is generated based on the evaluation values ​​of multiple indicators and the weight of each indicator. The generated reward signal can be expressed as follows:

[0072] ,

[0073] in, It is the number of indicators. It's a reward signal. It is the evaluation value of the m-th indicator. The weight of the m-th indicator.

[0074] Step S150: Update the model parameters of the customized text image model according to the reward signal.

[0075] In this embodiment, reinforcement learning is used to maximize all reward signals and update the model parameters of the customized text-to-image model according to the reward signals. This allows the customized text-to-image model to be optimized on multiple metrics simultaneously, resulting in higher quality customized image generation. After the update ends, a customized text-to-image model with good performance across multiple evaluation dimensions is obtained.

[0076] Through the above process, multiple evaluation dimensions are selected as reward signals for the customized text-to-graph model, enabling optimization across multiple evaluation dimensions. The weights of the indicators are determined based on the changes in the evaluation value of each indicator, thus resolving reward conflicts during the optimization process of multiple indicators. This allows for dynamic and adaptive simultaneous optimization of multiple indicators, further improving the performance of the customized text-to-graph model across multiple evaluation dimensions (e.g., cue word accuracy, subject accuracy, and human preference).

[0077] In conjunction with the above embodiments, in one implementation, this application also provides a course-based multi-reward reinforcement learning method applied to customized text-based image models. In this method, step S120, "evaluating the image based on multiple evaluation dimensions to obtain evaluation values ​​for multiple metrics," specifically includes steps S120-1 to S120-3:

[0078] Step S120-1: Using the prompt word accuracy evaluation model, assess the similarity between the image and the prompt word to obtain the evaluation value of the prompt word accuracy related index.

[0079] Step S120-2: Using the subject accuracy assessment model, evaluate the similarity between the subject in the image and the customized subject, and obtain the assessment value of the subject accuracy related indicators.

[0080] Step S120-3: Evaluate the degree of human satisfaction with the image through a human preference evaluation model to obtain the evaluation value of human preference-related indicators. The human preference evaluation model is a reward model trained based on human annotation information.

[0081] In this embodiment, a reward model with multiple evaluation dimensions (i.e., a cue word accuracy evaluation model, a subject accuracy evaluation model, and a human preference evaluation model) is used to evaluate the image. Among them, the cue word accuracy can adopt the CLIP-T index, that is, the cue word accuracy evaluation model is CLIP-T, and then the CLIP-T reward model is used to evaluate the similarity between the image and the cue word.

[0082] Subject accuracy can be assessed using two metrics: CLIP-I and DINO. That is, the subject accuracy assessment model can be CLIP-I and DINO, respectively, using the CLIP and DINO reward models to evaluate the similarity between the subject in the image and the customized subject.

[0083] Human preferences can be assessed using three metrics: Aesthetic, PickScore, and ImageReward. These three reward models evaluate human satisfaction with an image, and are trained based on human annotation information. Therefore, they can reflect human liking for an image to a certain extent.

[0084] In the above process, different reward models are used to evaluate the images generated by the customized text-to-image model, thereby quickly obtaining the evaluation values ​​of multiple indicators of the image on different evaluation dimensions, so that the model parameters of the customized text-to-image model can be updated based on the evaluation values ​​of multiple indicators.

[0085] In conjunction with the above embodiments, in one implementation, this application also provides a course-based multi-reward reinforcement learning method applied to customized text-based graph models. In this method, step S130, "determining the weight of each indicator based on the changes in the evaluation value of each indicator," specifically includes steps S130-1 to S130-2:

[0086] Step S130-1: Based on the moving average and moving average standard deviation of each indicator, obtain the change score for each indicator. The magnitude of the change score represents the change in the evaluation value of the indicator.

[0087] In this embodiment, to balance the impact of different indicators on the customized text graph model, when the customized text graph model performs poorly on one or more indicators, the weights of these indicators are increased, thereby dynamically and adaptively optimizing multiple indicators simultaneously. To achieve this, it is first necessary to determine the change in the evaluation value of each indicator, that is, whether the evaluation value of each indicator increases or decreases in the global process. This embodiment reflects this through the Exponential Moving Average (EMA) and the Exponential Moving Standard Deviation (EMSD) of each indicator. Specifically, by calculating the change score of each indicator based on its EMA and EMSD, the magnitude of the change score characterizes the change in the evaluation value of the indicator.

[0088] The moving average and moving average standard deviation of the indicators are calculated as follows:

[0089] The moving average of the indicator in the current update round is obtained based on the indicator's evaluation value and the moving average of the previous update round; the moving average standard deviation of the current update round is calculated based on the indicator's evaluation value, the moving average of the indicator in the current update round, and the moving average standard deviation of the previous update round.

[0090] For example, the moving average and standard deviation of the moving average for the current update round are expressed as follows:

[0091] ,

[0092] ,

[0093] in, It is the moving average of the m-th indicator in the i-th update round. The moving average of the m-th indicator in the (i-1)-th update round. It is the moving average standard deviation of the m-th indicator in the i-th update round. The moving average standard deviation of the m-th indicator in the (i-1)-th update round It is the attenuation coefficient of the m-th index. It is the evaluation value of the m-th indicator in the (i-1)-th update round.

[0094] The change score for each indicator is calculated based on its moving average and moving average standard deviation. This includes subtracting the moving average from the current update round's evaluation value and using the ratio of the moving average to the moving average standard deviation as the change score. For example, the change score can be expressed as:

[0095] ,

[0096] in, It is the change score of the m-th indicator in the i-th update round.

[0097] Since the moving average of an indicator reflects its average performance over past update cycles, a positive result (subtracting the moving average from the current evaluation value) indicates an increase in the indicator's evaluation value, while a negative result indicates a decrease. Furthermore, the change score is calculated by subtracting the moving average from the current update value and then subtracting the moving average from the moving average standard deviation. The magnitude of the moving average standard deviation reflects the degree of increase or decrease in the indicator's evaluation value. Therefore, the magnitude of the change score characterizes the change in the indicator's evaluation value.

[0098] Step S130-2: Determine the weight of each indicator based on the change score of each indicator. The weight of the indicator satisfies the following condition: the weight of the indicator with the smaller change score is greater than the weight of the indicator with the larger change score.

[0099] In this embodiment, each indicator is assigned an appropriate weight based on its change score (i.e., the change in the evaluation value of the indicator). Based on the course learning method, indicators that decrease more are given higher weights, allowing the customized text-based image model to be optimized first. Indicators that increase more are given lower weights. Therefore, the weights of the indicators satisfy the condition that the weights of indicators with smaller change scores are greater than the weights of indicators with larger change scores.

[0100] For example, the weights of indicators that satisfy the condition that the weight of an indicator with a smaller change score is greater than the weight of an indicator with a larger change score can be expressed as:

[0101] ,

[0102] In other words, if the change score of the m-th indicator The change score less than or equal to the nth indicator Then the weight of the nth indicator The weight of the m-th indicator is less than or equal to .

[0103] Specifically, the weight of each indicator is determined based on the change score of each indicator, including:

[0104] Based on the change scores of each indicator, a course learning model is constructed. The course learning model includes a first learning item and a second learning item. The first learning item is used to control the sum of the weights of all indicators to be minimized, and the second learning item is used to control the weight of each indicator to change within a preset range.

[0105] By solving the course learning model, the initial weight of each indicator is obtained;

[0106] The weight of each indicator is the ratio of its initial weight to the sum of the initial weights of all indicators.

[0107] For example, the course learning model is represented as follows:

[0108] ,

[0109] in, It is the first learning item; The second learning term is a regularization term used to control the weight of each indicator within a preset range, that is, to prevent the weight of the indicator from being too small or too large. When the weight is 1, it degenerates into a normal weighted reward sum. It is the regularization coefficient of the m-th index.

[0110] By solving the course learning model, the initial weight of each indicator is obtained. The initial weight of the indicator can be expressed as:

[0111] ,

[0112] Finally, an L1 regularization is applied to all weights to provide a holistic view of all indicators. The initial weight of each indicator is then calculated as the ratio of its initial weight to the sum of the initial weights of all indicators. For example, the weight of an indicator can be expressed as:

[0113] ,

[0114] The final reward signal It can be represented as:

[0115] ,

[0116] in, It is the number of indicators. It is the evaluation value of the m-th indicator. The weight of the m-th indicator.

[0117] In this embodiment of the application, after obtaining the evaluation value and weight of each indicator, the model parameters of the customized text-to-image model are updated by reinforcement learning, so that the customized text-to-image model is optimized on multiple indicators at the same time, resulting in a customized text-to-image model with good performance on multiple evaluation dimensions.

[0118] For example, updating the model parameters of the customized text-based image model based on the reward signal can be expressed as follows:

[0119] ,

[0120] Where t represents the diffusion stage of the customized text-based graph model. It is a latent variable in the image at the t-th diffusion stage. The latent variable in the image during the (t-1)th diffusion stage. It's a reward signal. y is the initial latent variable of the image, and y is the given condition, which in text-based images generally refers to the text prompt.

[0121] In conjunction with the above embodiments, in one implementation, this application also provides a course-based multi-reward reinforcement learning method applied to a customized text-based graph model. In this method, the customized text-based graph model is obtained through the following steps, specifically including steps A1 to A4:

[0122] Step A1: Obtain a sample image of a specific subject, and obtain the prompt text containing the subject's identifier.

[0123] Step A2: Input the sample image and the prompt text into the text-to-image model for noise addition and denoising processing to obtain the generated image.

[0124] Step A3: Calculate the loss function value between the sample image and the generated image based on the noise added by the text-based image model in the noise addition process and the noise fitted by the text-based image model in the noise removal process.

[0125] Step A4: Fine-tune the model parameters of the text image model based on the loss function value to obtain the customized text image model.

[0126] In this embodiment, a customized text-based image model is fine-tuned based on a fine-tuning method to obtain a customized text-based image model. The text-based image model is a Latent Diffusion Model (LDM), typically composed of a U-Net as the main backbone network, a Variational Auto-Encoder (VAE) that represents images as latent variables, and a text encoder for encoding cue words. This is achieved by using K sample images of a specific subject... and including the main identifier Prompt text Input the text image model that needs fine-tuning to achieve customization.

[0127] The sample image and the prompt text are input into the text-to-image model for noise addition and denoising to obtain the generated image. To simplify the calculation, the difference between the sample image and the generated image is converted into the difference between the noise added during noise addition and the noise fitted during denoising.

[0128] For example, the loss function value between the sample image and the generated image can be expressed as:

[0129] ,

[0130] in, It is a latent variable in the image at the t-th diffusion stage. It is the diffusion stage. It is a text encoder. It refers to adding noise in noise processing. It refers to the noise fitted during the denoising process, specifically the noise in the Wensheng graph model's U-shaped network through parameters. Fit the corresponding noise.

[0131] Further, based on the noise added by the text-based image model in the noise addition process and the noise fitted by the text-based image model in the denoising process, the loss function value between the sample image and the generated image is calculated, including:

[0132] Based on the noise added to a specific subject and the noise added to other subjects in the noise addition process by the text-based image model, and the noise fitted to the specific subject and the noise fitted to other subjects in the denoising process by the text-based image model, the loss function value between the sample image and the generated image is calculated.

[0133] In this embodiment of the application, in order to prevent the generated image content from having a large offset, a regularization term for the subject category is added to this loss function to ensure that the category of the generated customized subject is correct.

[0134] For example, the loss function value between the sample image and the generated image can also be expressed as:

[0135] ,

[0136] in, It refers to adding noise to other subjects during noise reduction processing. It is a descriptive text for prompts targeting other subjects. It is a diffusion stage targeting other entities. It is in the Latent variables of images in each diffusion stage, It is the coefficient of the regularization term.

[0137] Finally, the model parameters of the text-based image model are fine-tuned based on the loss function value to obtain the customized text-based image model. Thus, by fine-tuning a part of the hidden diffusion model (e.g., a U-shaped network or a text encoder), the latent variables of the main image are bound to the latent variables of the special identifier, achieving customized generation. After this step, given a main image and prompt words, the customized text-based image model is capable of generating images that simultaneously meet the main requirements and prompt words. At this point, the customized text-based image model still cannot perform well across multiple evaluation dimensions. Therefore, the model parameters of the customized text-based image model are further updated through steps S110 to S150 above, thereby obtaining a customized text-based image model with good performance across multiple evaluation dimensions.

[0138] Reference Figure 2 As shown, Figure 2 This is a flowchart illustrating another course-based multi-reward reinforcement learning method applied to customized text-based graph models, provided in an embodiment of this application. The method includes steps S210 to S260:

[0139] Step S210: Obtain the customized text image model.

[0140] Specifically, a sample image of a specific subject is obtained, and a prompt text containing the subject's identifier is obtained; the sample image and the prompt text are input into a text-based image model for noise addition and denoising processing to obtain a generated image; based on the noise added by the text-based image model in the noise addition processing and the noise fitted by the text-based image model in the denoising processing, a loss function value between the sample image and the generated image is calculated; the model parameters of the text-based image model are fine-tuned based on the loss function value to obtain the customized text-based image model.

[0141] Step S220: Obtain an image of a specific subject generated by the customized text-to-image model based on the prompt words.

[0142] Step S230: Evaluate the image based on indicators of multiple evaluation dimensions to obtain evaluation values ​​of multiple indicators. The multiple evaluation dimensions include: accuracy of prompt words, accuracy of subject, and human preference. Each evaluation dimension includes at least one indicator.

[0143] Specifically, the image is evaluated based on multiple evaluation dimensions to obtain evaluation values ​​for multiple indicators, including: using a prompt word accuracy evaluation model to assess the similarity between the image and the prompt word, obtaining evaluation values ​​for prompt word accuracy-related indicators; using a subject accuracy evaluation model to assess the similarity between the subject in the image and the customized subject, obtaining evaluation values ​​for subject accuracy-related indicators; and using a human preference evaluation model to assess the degree of human satisfaction with the image, obtaining evaluation values ​​for human preference-related indicators. The human preference evaluation model is a reward model trained based on human annotation information.

[0144] Step S240: Based on the moving average and moving average standard deviation of each indicator, obtain the change score of each indicator, the magnitude of the change score characterizes the change in the evaluation value of the indicator; based on the change score of each indicator, determine the weight of each indicator, the weight of the indicator satisfies the following: the weight of the indicator with the smaller change score is greater than the weight of the indicator with the larger change score.

[0145] Step S250: Generate a reward signal based on the evaluation values ​​of the multiple indicators and the weight of each indicator.

[0146] Step S260: Update the model parameters of the customized text image model according to the reward signal.

[0147] In this embodiment, multiple evaluation dimensions are selected as reward signals for the customized text-to-graph model, enabling optimization across multiple evaluation dimensions. The weights of the indicators are determined based on the changes in the evaluation value of each indicator, thereby resolving reward conflicts during the optimization process of multiple indicators. This allows for dynamic and adaptive simultaneous optimization of multiple indicators, further improving the performance of the customized text-to-graph model across multiple evaluation dimensions (e.g., cue word accuracy, subject accuracy, and human preference).

[0148] Reference Figure 3 As shown, Figure 3 This is an overall architecture diagram of a course-based multi-reward reinforcement learning method applied to a customized text-based graph model, provided in an embodiment of this application. The method consists of two stages. In the first stage, customized text-based graph fine-tuning is performed: using multiple sample images containing customized subjects and prompt text containing subject identifiers, the text-based graph model is fine-tuned to give the generated graph model customization capabilities, resulting in a customized text-based graph model.

[0149] The second stage employs multi-reward reinforcement learning: the images are evaluated based on multiple evaluation dimensions, specifically using a multi-dimensional reward model (i.e., a cue word accuracy evaluation model, a subject accuracy evaluation model, and a human preference evaluation model) to obtain evaluation values ​​for multiple metrics. These evaluation values ​​are then used as reward signals, and reinforcement learning is used to further fine-tune the model to maximize rewards. Furthermore, to address reward conflict challenges during multi-reward optimization, a course reweighting mechanism is adopted: metrics showing an increase in evaluation values ​​are assigned lower weights, while metrics showing a decrease in evaluation values ​​are assigned higher weights. This dynamically and adaptively optimizes multiple metrics simultaneously, further improving the performance of the customized text-based image model across multiple evaluation dimensions.

[0150] This application also provides a course-based multi-reward reinforcement learning device applied to customized text-based graph models, referring to... Figure 4 As shown, Figure 4 This is a schematic diagram of a course-based multi-reward reinforcement learning device applied to a customized text-based graph model, provided in an embodiment of this application. The device includes:

[0151] The acquisition module 410 is used to acquire images of specific subjects generated by the customized text-to-image model based on prompt words;

[0152] The evaluation module 420 is used to evaluate the image based on indicators of multiple evaluation dimensions to obtain evaluation values ​​of multiple indicators. The multiple evaluation dimensions include: accuracy of prompt words, accuracy of subject, and human preference. Each evaluation dimension includes at least one indicator.

[0153] Module 430 is used to determine the weight of each indicator based on the changes in the evaluation value of each indicator;

[0154] The generation module 440 is used to generate a reward signal based on the evaluation values ​​of the multiple indicators and the weight of each indicator;

[0155] The update module 450 is used to update the model parameters of the customized text-based image model according to the reward signal.

[0156] In an optional embodiment, the determining module is further configured to assign a smaller weight to indicators whose evaluation values ​​are increasing, wherein assigning a smaller weight means that the weight of the indicator is reduced relative to the previous update round; and to assign a larger weight to indicators whose evaluation values ​​are decreasing, wherein assigning a larger weight means that the weight of the indicator is increased relative to the previous update round.

[0157] In one optional embodiment, the determining module includes:

[0158] The change score module is used to obtain the change score of each indicator based on the moving average and moving average standard deviation of each indicator. The magnitude of the change score represents the change in the evaluation value of the indicator.

[0159] The weight determination module is used to determine the weight of each indicator based on the change score of each indicator. The weight of the indicator satisfies the following condition: the weight of the indicator with the smaller change score is greater than the weight of the indicator with the larger change score.

[0160] In an optional embodiment, the weight determination module is further configured to construct a course learning model based on the change scores of each indicator. The course learning model includes a first learning term and a second learning term. The first learning term is used to control the sum of the weights of all indicators to be minimized, and the second learning term is used to control the weight of each indicator to change within a preset range. The initial weight of each indicator is obtained by solving the course learning model. The ratio of the initial weight of each indicator to the sum of the initial weights of all indicators is used as the weight of each indicator.

[0161] In an optional embodiment, the change score module is further configured to obtain the moving average of the indicator in the current update round based on the indicator's evaluation value and the moving average of the previous update round; and to calculate the moving average standard deviation of the current update round based on the indicator's evaluation value, the moving average of the indicator in the current update round, and the moving average standard deviation of the previous update round.

[0162] In an optional embodiment, the first evaluation module includes:

[0163] The first evaluation submodule is used to evaluate the similarity between the image and the prompt word through a prompt word accuracy evaluation model, and obtain the evaluation value of the prompt word accuracy related index;

[0164] The second evaluation submodule is used to evaluate the similarity between the subject in the image and the customized subject through the subject accuracy evaluation model, and obtain the evaluation value of the subject accuracy related indicators.

[0165] The third evaluation submodule is used to evaluate the degree of human satisfaction with the image through a human preference evaluation model, and obtain the evaluation value of human preference-related indicators. The human preference evaluation model is a reward model trained based on human annotation information.

[0166] In an optional embodiment, the apparatus further includes a model acquisition module for acquiring the customized text image model, the model acquisition module comprising:

[0167] The acquisition module is used to acquire sample images of a specific subject, and to acquire prompt text containing the subject's identifier;

[0168] The input module is used to input the sample image and the prompt text into the text-to-image model for noise addition and denoising processing to obtain the generated image;

[0169] The calculation module is used to calculate the loss function value between the sample image and the generated image based on the noise added by the text image model in the noise addition process and the noise fitted by the text image model in the noise removal process.

[0170] The fine-tuning module is used to fine-tune the model parameters of the text-based graph model according to the loss function value, so as to obtain the customized text-based graph model.

[0171] In an optional embodiment, the calculation module is further configured to calculate the loss function value between the sample image and the generated image based on the noise added to a specific subject and the noise added to other subjects by the text-based image model in the noise addition process, and the noise fitted to the specific subject and the noise fitted to other subjects by the text-based image model in the denoising process.

[0172] This application also provides an electronic device, see embodiments thereof. Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 5 As shown, the electronic device 500 includes a memory 510 and a processor 520. The memory 510 and the processor 520 are connected via a bus for communication. The memory 510 stores a computer program that can run on the processor 520 to implement the steps of the course multi-reward reinforcement learning method applied to a customized text-based image model as described in the embodiments of this application.

[0173] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the course multi-reward reinforcement learning method applied to a customized text-based graph model as described in this application.

[0174] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the course multi-reward reinforcement learning method applied to a customized text-based graph model as described in this application.

[0175] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0176] This application describes embodiments of methods and apparatus according to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0177] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0178] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0179] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0180] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0181] The above provides a detailed description of a multi-reward reinforcement learning method and apparatus for customized text-based graph models. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A course-based multi-reward reinforcement learning method applied to customized text-based graph models, characterized in that, The method includes: Obtain images of specific subjects generated by a customized text-to-image model based on prompts; The image is evaluated based on multiple evaluation dimensions to obtain evaluation values ​​for multiple indicators, including: using a prompt word accuracy evaluation model to assess the similarity between the image and the prompt word, obtaining evaluation values ​​for prompt word accuracy-related indicators; using a subject accuracy evaluation model to assess the similarity between the subject in the image and the customized subject, obtaining evaluation values ​​for subject accuracy-related indicators; and using a human preference evaluation model to assess the degree of human satisfaction with the image, obtaining evaluation values ​​for human preference-related indicators. The human preference evaluation model is a reward model trained based on human annotation information. Based on the changes in the evaluation value of each indicator, the weight of each indicator is determined, including: assigning a smaller weight to indicators whose evaluation value is increasing, whereby the smaller weight means that the weight of the indicator is reduced relative to the previous update round; and assigning a larger weight to indicators whose evaluation value is decreasing, whereby the larger weight means that the weight of the indicator is increased relative to the previous update round. A reward signal is generated based on the evaluation values ​​of multiple indicators and the weight of each indicator; The model parameters of the customized text-based image model are updated based on the reward signal.

2. The method according to claim 1, characterized in that, Based on the changes in the evaluation value of each indicator, determine the weight of each indicator, including: Based on the moving average and moving average standard deviation of each indicator, a change score is obtained for each indicator. The magnitude of the change score represents the change in the evaluation value of the indicator. The weight of each indicator is determined based on the change score of each indicator, and the weight of the indicator satisfies the following condition: the weight of the indicator with the smaller change score is greater than the weight of the indicator with the larger change score.

3. The method according to claim 2, characterized in that, Based on the change score of each indicator, determine the weight of each indicator, including: Based on the change scores of each indicator, a course learning model is constructed. The course learning model includes a first learning item and a second learning item. The first learning item is used to control the sum of the weights of all indicators to be minimized, and the second learning item is used to control the weight of each indicator to change within a preset range. By solving the course learning model, the initial weight of each indicator is obtained; The weight of each indicator is the ratio of its initial weight to the sum of the initial weights of all indicators.

4. The method according to claim 2, characterized in that, The moving average and standard deviation of the indicators are calculated as follows: Based on the evaluation value of the indicator and the moving average of the previous update round, the moving average of the indicator in the current update round is obtained. The moving average standard deviation for the current update round is calculated based on the indicator's evaluation value, the moving average of the indicator in the current update round, and the moving average standard deviation of the indicator in the previous update round.

5. The method according to any one of claims 1-4, characterized in that, The customized text image model was obtained through the following method: Obtain sample images of a specific subject, and obtain prompt text containing the subject's identifier; The sample image and the prompt text are input into the text-generated image model for noise addition and denoising to obtain the generated image. Based on the noise added by the text image model in the noise addition process and the noise fitted by the text image model in the noise removal process, the loss function value between the sample image and the generated image is calculated. The model parameters of the text image model are fine-tuned based on the loss function value to obtain the customized text image model.

6. The method according to claim 5, characterized in that, Based on the noise added by the text-based image model in the noise addition process, and the noise fitted by the text-based image model in the denoising process, the loss function value between the sample image and the generated image is calculated, including: Based on the noise added to a specific subject and the noise added to other subjects in the noise addition process by the text-based image model, and the noise fitted to the specific subject and the noise fitted to other subjects in the denoising process by the text-based image model, the loss function value between the sample image and the generated image is calculated.

7. A course-based multi-reward reinforcement learning device applied to customized text-based graph models, characterized in that, The device comprises: The acquisition module is used to acquire images of specific subjects generated by the customized text-to-image model based on prompt words; The evaluation module is used to evaluate the image based on multiple evaluation dimensions and obtain evaluation values ​​for multiple indicators, including: evaluating the similarity between the image and the prompt words using a prompt word accuracy evaluation model to obtain evaluation values ​​for prompt word accuracy-related indicators; evaluating the similarity between the subject in the image and the customized subject using a subject accuracy evaluation model to obtain evaluation values ​​for subject accuracy-related indicators; and evaluating the degree of human satisfaction with the image using a human preference evaluation model, wherein the human preference evaluation model is a reward model trained based on human annotation information. The determination module is used to determine the weight of each indicator based on the change in the evaluation value of each indicator, including: assigning a smaller weight to indicators whose evaluation value is increasing, whereby the smaller weight means that the weight of the indicator is reduced relative to the previous update round; and assigning a larger weight to indicators whose evaluation value is decreasing, whereby the larger weight means that the weight of the indicator is increased relative to the previous update round. A generation module is used to generate a reward signal based on the evaluation values ​​of multiple indicators and the weight of each indicator; The update module is used to update the model parameters of the customized text-based image model according to the reward signal.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the course multi-reward reinforcement learning method for customized text-based graph models as described in any one of claims 1-6.

Images