Course multi-reward reinforcement learning method and device applied to customized text graph model

Abstract

The invention provides a course multi-reward reinforcement learning method and device applied to a customized text graph model, relates to the technical field of text graphs, and aims to improve the performance of the customized text graph model in multiple evaluation dimensions. The method comprises the following steps: acquiring a picture of a specific subject generated by a customized text graph model according to a cue word; the picture is evaluated based on indexes of multiple evaluation dimensions, evaluation values of the multiple indexes are obtained, the multiple evaluation dimensions comprise cue word accuracy, subject accuracy and human preference, and each evaluation dimension comprises at least one index; determining the weight of each index according to the change condition of the evaluation value of each index; generating a reward signal according to the evaluation values of the plurality of indexes and the weight of each index; and updating model parameters of the customized text graph model according to the reward signal.

Description

Technical Field

[0001] The present application relates to the field of text graph technology, and in particular to a course multi-reward reinforcement learning method and device applied to a customized text graph model. Background Art

[0002] A text map is a task used to generate corresponding images based on given prompts, while a customized text map task further requires that the generated image contain the user's customized subject. The customized text map task can be completed through various customized text map models.

[0003] However, the images generated by the existing customized text-generated graph model cannot perform well in multiple evaluation dimensions, especially in terms of prompt word accuracy, subject accuracy, and human preference, which are often only satisfactory in one or two of the three dimensions. Therefore, how to improve the performance of the customized text-generated graph model in multiple evaluation dimensions is a technical problem that needs to be solved urgently. Summary of the invention

[0004] In view of the above problems, an embodiment of the present application provides a course multi-reward reinforcement learning method and device applied to a customized literary graph model, so as to overcome the above problems or at least partially solve the above problems.

[0005] In a first aspect of an embodiment of the present application, a course multi-reward reinforcement learning method applied to a customized cultural graph model is disclosed, the method comprising: Get the picture of the specific subject generated by the customized text image model according to the prompt word; Evaluate the image based on indicators of multiple evaluation dimensions to obtain evaluation values ​​of multiple indicators, wherein the multiple evaluation dimensions include: prompt word accuracy, subject accuracy, and human preference, and each evaluation dimension includes at least one indicator; Determine the weight of each indicator based on the changes in the evaluation value of each indicator; generating a reward signal according to the evaluation values ​​of the plurality of indicators and the weight of each indicator; Model parameters of the customized text graph model are updated according to the reward signal.

[0006] Optionally, the weight of each indicator is determined according to the change of the evaluation value of each indicator, including: For the indicator whose evaluation value changes upward, a smaller weight is assigned, wherein the smaller weight means that the weight of the indicator is reduced relative to the previous update round; For the indicator whose evaluation value changes downward, a greater weight is assigned, and the assigning of a greater weight means that the weight of the indicator is increased relative to the previous update round.

[0007] Optionally, the weight of each indicator is determined according to the change of the evaluation value of each indicator, including: According to the moving average and moving average standard deviation of each indicator, a change score of each indicator is obtained, and the size of the change score represents the change of the evaluation value of the indicator; The weight of each indicator is determined according to the change score of each indicator, and the weight of the indicator satisfies: the weight of the indicator with a smaller change score is greater than the weight of the indicator with a larger change score.

[0008] Optionally, the weight of each indicator is determined according to the change score of each indicator, including: According to the change score of each indicator, a course learning model is constructed, wherein the course learning model includes a first learning item and a second learning item, wherein the first learning item is used to control the sum of weights of all indicators to be minimum, and the second learning item is used to control the weight of each indicator to change within a preset range; By solving the course learning model, the initial weight of each indicator is obtained; The ratio of the initial weight of each indicator to the sum of the initial weights of all indicators is used as the weight of each indicator.

[0009] Optionally, the indicator's moving average and moving average standard deviation are calculated as follows: According to the evaluation value of the indicator and the moving average value of the previous update round, the moving average value of the indicator of the current update round is obtained; The moving average standard deviation of the current update round is calculated based on the evaluation value of the indicator, the moving average of the indicator in the current update round, and the moving average standard deviation of the previous update round.

[0010] Optionally, the image is evaluated based on indicators of multiple evaluation dimensions to obtain evaluation values ​​of multiple indicators, including: Using a prompt word accuracy evaluation model, the similarity between the image and the prompt word is evaluated to obtain an evaluation value of an indicator related to the prompt word accuracy; Using a subject accuracy evaluation model, the similarity between the subject in the image and the customized subject is evaluated to obtain an evaluation value of a subject accuracy-related indicator; The degree of satisfaction of humans with the picture is evaluated by a human preference evaluation model to obtain an evaluation value of a human preference-related index. The human preference evaluation model is a reward model trained based on human annotation information.

[0011] Optionally, the customized text graph model is obtained by: Obtain sample images of a specific subject, and obtain prompt word text containing the subject identification; Inputting the sample image and the prompt word text into the text graph model for noise addition and denoising to obtain a generated image; Calculating a loss function value between the sample image and the generated image according to the added noise of the Vincent graph model in the noise adding process and the noise fitted by the Vincent graph model in the denoising process; The model parameters of the document graph model are fine-tuned according to the loss function value to obtain the customized document graph model.

[0012] Optionally, calculating the loss function value between the sample image and the generated image according to the added noise of the Vincent graph model in the noise adding process and the noise fitted by the Vincent graph model in the denoising process includes: The loss function value between the sample image and the generated image is calculated based on the added noise for the specific subject and the added noise for other subjects of the Vincent graph model in the noise addition process, and the noise fitted to the specific subject and the noise fitted to other subjects by the Vincent graph model in the denoising process.

[0013] In a second aspect of the embodiment of the present application, a course multi-reward reinforcement learning device applied to a customized cultural graph model is disclosed, the device comprising: An acquisition module is used to acquire a picture of a specific subject generated by a customized text image model according to a prompt word; An evaluation module, used to evaluate the image based on indicators of multiple evaluation dimensions to obtain evaluation values ​​of multiple indicators, wherein the multiple evaluation dimensions include: prompt word accuracy, subject accuracy, and human preference, and each evaluation dimension includes at least one indicator; A determination module is used to determine the weight of each indicator according to the change of the evaluation value of each indicator; A generating module, configured to generate a reward signal according to the evaluation values ​​of the plurality of indicators and the weight of each indicator; An updating module is used to update the model parameters of the customized text graph model according to the reward signal.

[0014] According to a third aspect of an embodiment of the present application, an electronic device is disclosed, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the course multi-reward reinforcement learning method applied to a customized literary graph model described in the first aspect of the embodiment of the present application are implemented.

[0015] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is disclosed, on which a computer program is stored. When the computer program is executed by a processor, the steps of the course multi-reward reinforcement learning method applied to a customized literary graph model described in the first aspect of the embodiment of the present application are implemented.

[0016] In a fifth aspect of an embodiment of the present application, a computer program product is disclosed, including a computer program, which, when executed by a processor, implements the steps of the course multi-reward reinforcement learning method applied to a customized literary graph model described in the first aspect of the embodiment of the present application.

[0017] The embodiments of the present application include the following advantages: In an embodiment of the present application, a picture of a specific subject generated by a customized text graph model according to a prompt word is obtained; and the picture is evaluated based on indicators of multiple evaluation dimensions to obtain evaluation values ​​of multiple indicators, wherein the multiple evaluation dimensions include: prompt word accuracy, subject accuracy, and human preference, and each evaluation dimension includes at least one indicator; and then the weight of each indicator is determined according to the change of the evaluation value of each indicator, and a reward signal is generated according to the evaluation values ​​of the multiple indicators and the weight of each indicator; finally, the model parameters of the customized text graph model are updated according to the reward signal. In this way, indicators of multiple evaluation dimensions are selected as reward signals of the customized text graph model, and the customized text graph model can be optimized on multiple evaluation dimensions. The weight of the indicator is determined according to the change of the evaluation value of each indicator, thereby solving the reward conflict in the optimization process of multiple indicators, dynamically and adaptively realizing the simultaneous optimization of multiple indicators, and further improving the performance of the customized text graph model on multiple evaluation dimensions (for example, prompt word accuracy, subject accuracy, and human preference). BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 It is a flowchart of the steps of a course multi-reward reinforcement learning method applied to a customized cultural graph model provided in an embodiment of the present application; Figure 2 It is a flowchart of the steps of another method for multi-reward reinforcement learning of courses applied to a customized cultural graph model provided in an embodiment of the present application; Figure 3It is an overall architecture diagram of a course multi-reward reinforcement learning method applied to a customized cultural graph model provided in an embodiment of the present application; Figure 4 It is a structural schematic diagram of a course multi-reward reinforcement learning device applied to a customized cultural graph model provided in an embodiment of the present application; Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] A vincent graph is a task for generating corresponding images based on given prompts, while a customized vincent graph task further requires that the generated image contain a customized subject of the user. For example, when a user wants to generate a picture of a "puppy running", the "puppy" here should be a dog given by the user, not a dog of any other breed. Although various customized vincent graph models can currently complete this task, it is difficult for the images they generate to perform well on multiple evaluation dimensions at the same time, especially for the following three aspects: prompt word accuracy, subject accuracy, and human preference. Among them, prompt word accuracy refers to whether the generated image meets the requirements of the prompt word. For example, the prompt word is "puppy running", but the generated image depicts the appearance of a "puppy standing", which does not meet the prompt word accuracy requirement. Subject accuracy refers to whether the subject in the generated image is a special subject provided by the user. For example, the user provides several pictures of his own corgi, but the generated picture is a teddy bear, which does not meet the subject accuracy requirement. Human preference is a requirement for image quality. If the puppy in the generated picture has some distorted and unreasonable body parts, it does not meet the human preference requirement. The existing customized text graph model only performs well in some evaluation dimensions, but performs poorly in other evaluation dimensions and cannot meet all the above requirements at the same time.

[0022] Existing customized vinyasa graph methods are divided into three categories according to the control conditions: model-based methods, fine-tuning-based methods, and methods without any training. The most commonly used method is the customized vinyasa graph method based on fine-tuning, which binds the customized subject to a special marker by fine-tuning some parameters in the vinyasa graph model, thereby achieving customized generation. The following is a brief description of these three methods: The first category: model-based methods introduce additional models to encode customized conditions. The additional model here is generally an encoder. The prompt words with special identifiers and the given multiple customized subject pictures are input into the encoder to represent them as vectors, and then integrated into the text map process. A model-based method encodes personalized sample images by using sub-graph encoders and concept encoders, and adds an adapter layer to the model to achieve customized condition control. These additional models (including encoders and adapters) are trainable, while other components of the model remain unchanged. This type of method introduces additional models that consume more computing space, and the pre-training process consumes more computing time, but once the encoder and adapter are fully trained, they can easily adapt to different customized conditions.

[0023] The second category: tuning-based methods do not require additional models, but fine-tune some parts of the original text graph model to achieve customization. In these tuning-based methods, parameter-efficient fine-tuning techniques are usually used to replace traditional fine-tuning to reduce the consumption of computing resources. Tuning-based methods save computing space by not introducing additional models, but each fine-tuning can only adapt to a specific condition.

[0024] The third category: untrained methods omit any training or fine-tuning process, and instead use the intrinsic capabilities of the structure of the image model (e.g., attention mechanism) to control the generation. A untrained method achieves consistent style generation by performing minimal "attention sharing" in the diffusion process, where all images share self-attention with the reference image. Untrained methods do not require additional models or fine-tuning time, but can only control a limited range of conditions, such as layout or style.

[0025] In summary, the images generated by existing customized text-based image methods usually cannot perform well in multiple evaluation dimensions, especially for the three aspects of prompt word accuracy, subject accuracy, and human preference, which are often only satisfactory in one or two dimensions. Some previous work has discussed this issue. It is generally believed that pre-trained generative models tend to overfit on limited sample images or overfit on information in these images that is not related to the subject. Although some methods have tried to solve the overfitting problem through techniques such as retention mechanisms and decoupled learning, they have not been able to improve the quality of generated images in multiple evaluation dimensions at the same time. In addition, since there may be conflicts between the indicators of multiple evaluation dimensions, achieving comprehensive optimization is a complex process.

[0026] In order to overcome the limitations of related technologies, the embodiment of the present application provides a course multi-reward reinforcement learning method applied to a customized Wensheng graph model. Specifically, a customized Wensheng graph model with customization capabilities is obtained, and indicators of multiple evaluation dimensions are used as reward signals. The customized Wensheng graph model is further fine-tuned through reinforcement learning to maximize the reward; in order to deal with the reward conflict problem in the multi-reward (indicator) optimization process, the embodiment of the present application proposes a course re-weighting mechanism, which determines the weight of each indicator according to the change in the evaluation value of each indicator, so as to adaptively balance the reward signal to ensure that each reward signal is fully optimized.

[0027] Reference Figure 1 As shown, Figure 1 1 is a flowchart of the steps of a course multi-reward reinforcement learning method applied to a customized cultural graph model provided in an embodiment of the present application. Figure 1 As shown, the course multi-reward reinforcement learning method applied to the customized literary graph model may include steps S110 to S150: Step S110: obtaining a picture of a specific subject generated by the customized text-based image model according to the prompt word.

[0028] In the embodiment of the present application, the customized text graph model is obtained by fine-tuning the text graph model based on the customized subject, and the customized text graph model refers to the ability to generate a picture of a specific subject (i.e., a customized subject). Specifically, obtaining a picture of a specific subject generated by the customized text graph model according to a prompt word includes: inputting the prompt word into the customized text graph model, and the customized text graph model generates a picture of the specific subject according to the prompt word; wherein the prompt word is the description information of the desired generated picture.

[0029] Step S120: Evaluate the image based on indicators of multiple evaluation dimensions to obtain evaluation values ​​of multiple indicators, wherein the multiple evaluation dimensions include: prompt word accuracy, subject accuracy, and human preference, and each evaluation dimension includes at least one indicator.

[0030] In the embodiment of the present application, the indicators of multiple evaluation dimensions include indicators on prompt word accuracy, indicators on subject accuracy, and indicators on human preference. Among them, prompt word accuracy refers to whether the image meets the requirements of the prompt word, that is, the degree of similarity between the image and the prompt word; subject accuracy refers to whether the subject in the image is a customized subject, that is, the degree of similarity between the subject in the image and the customized subject; human preference is the requirement for image quality, that is, the degree of satisfaction of humans with the image.

[0031] The evaluation value of an indicator represents the degree to which the image satisfies the indicator. The greater the degree to which the image satisfies a certain indicator, the higher the evaluation value of the indicator. For example, the greater the similarity between the image and the prompt word, the greater the corresponding evaluation value. In this way, the image generated by the customized text graph model is evaluated based on different indicators, and then fed back to the customized text graph model, so that the customized text graph model improves the quality of the generated image based on the evaluation values ​​of multiple indicators.

[0032] Step S130: Determine the weight of each indicator according to the change of the evaluation value of each indicator.

[0033] In the embodiment of the present application, after the evaluation value of the indicator is used as the reward signal for the reinforcement learning of the customized Wensheng graph model, different indicators may have different degrees of influence on the customized Wensheng graph model. If the evaluation values ​​of all indicators are directly added as the final reward signal, conflicts will arise between different indicators, and the optimization method of the customized Wensheng graph model may only focus on one or more indicators, rather than moving towards optimizing all indicators at the same time. Therefore, the embodiment of the present application adopts a course multi-reward reinforcement learning method to determine the weight of each indicator according to the change in the evaluation value of each indicator.

[0034] Specifically, the weight of each indicator is determined according to the change in the evaluation value of each indicator, including: for the indicator whose evaluation value changes as an increase, a smaller weight is assigned, and assigning a smaller weight means that the weight of the indicator is reduced relative to the previous update round; for the indicator whose evaluation value changes as a decrease, a larger weight is assigned, and assigning a larger weight means that the weight of the indicator is increased relative to the previous update round.

[0035] In this way, the customized text-generated graph model is balanced for multiple indicators. When the customized text-generated graph model performs poorly on one or more of the indicators, the weights of these indicators are increased, thereby dynamically and adaptively optimizing multiple reward signals (indicators) at the same time. For example, after a certain update, the indicators of prompt word accuracy and human preference of the pictures generated by the customized text-generated graph model have improved a lot, but the indicators of subject accuracy have decreased. In this way, the weights of the indicators corresponding to subject accuracy are increased, so that the customized text-generated graph model pays more attention to the subject accuracy of the generated pictures, thereby comprehensively improving the quality of the pictures generated by the customized text-generated graph model.

[0036] Step S140: Generate a reward signal according to the evaluation values ​​of the multiple indicators and the weight of each indicator.

[0037] In the embodiment of the present application, after obtaining the weight of each indicator, a reward signal is generated according to the evaluation values ​​of multiple indicators and the weight of each indicator. The generated reward signal can be expressed as: , in, is the number of indicators, is a reward signal, is the evaluation value of the mth indicator, The weight of the mth indicator.

[0038] Step S150: updating the model parameters of the customized text graph model according to the reward signal.

[0039] In the embodiment of the present application, in order to maximize all reward signals, a reinforcement learning method is adopted to update the model parameters of the customized text graph model according to the reward signal, so that the customized text graph model is optimized on multiple indicators at the same time, resulting in higher quality customized image generation; and after the conditions for ending the update are met, a customized text graph model with good performance in multiple evaluation dimensions is obtained.

[0040] Through the above process, indicators of multiple evaluation dimensions are selected as reward signals for the customized text graph model, and the customized text graph model can be optimized in multiple evaluation dimensions. The weight of the indicator is determined according to the change of the evaluation value of each indicator, thereby resolving the reward conflict in the optimization process of multiple indicators, dynamically and adaptively realizing the simultaneous optimization of multiple indicators, and further improving the performance of the customized text graph model in multiple evaluation dimensions (for example, prompt word accuracy, subject accuracy, and human preference).

[0041] In combination with the above embodiments, in one implementation, the present application embodiment also provides a course multi-reward reinforcement learning method applied to a customized text image model. In this method, the "evaluating the image based on indicators of multiple evaluation dimensions to obtain evaluation values ​​of multiple indicators" in the above step S120 may specifically include steps S120-1 to S120-3: Step S120 - 1 : using a prompt word accuracy evaluation model, evaluating the similarity between the image and the prompt word, and obtaining an evaluation value of an indicator related to the prompt word accuracy.

[0042] Step S120 - 2 : Using a subject accuracy evaluation model, the similarity between the subject in the image and the customized subject is evaluated to obtain evaluation values ​​of subject accuracy related indicators.

[0043] Step S120 - 3 : Evaluate the degree of satisfaction of humans with the image through a human preference evaluation model to obtain evaluation values ​​of human preference related indicators. The human preference evaluation model is a reward model trained based on human annotation information.

[0044] In an embodiment of the present application, a reward model with multiple evaluation dimensions (i.e., a prompt word accuracy evaluation model, a subject accuracy evaluation model, and a human preference evaluation model) is used to evaluate the image; wherein, the prompt word accuracy can adopt the CLIP-T indicator, that is, the prompt word accuracy evaluation model is CLIP-T, and then the CLIP-T reward model is used to evaluate the similarity between the image and the prompt word.

[0045] The subject accuracy can be measured using two indicators, CLIP-I and DINO. That is, the subject accuracy evaluation model can be CLIP-I and DINO. The similarity between the subject in the image and the customized subject is evaluated using two reward models, CLIP and DINO, respectively.

[0046] Human preference can be evaluated using three indicators: Aesthetic, PickScore and ImageReward. That is, the human preference evaluation models are Aesthetic, PickScore and ImageReward. The three reward models of Aesthetic, PickScore and ImageReward are used to evaluate the degree of human satisfaction with the image. These reward models are reward models trained based on human annotation information, and thus can reflect the degree of human preference for a picture to a certain extent.

[0047] In the above process, different reward models are used to evaluate the images generated by the customized text-generated graph model, so that the evaluation values ​​of multiple indicators of the images in different evaluation dimensions can be quickly obtained, so that the model parameters of the customized text-generated graph model can be updated based on the evaluation values ​​of multiple indicators.

[0048] In combination with the above embodiments, in one implementation, the present application embodiment also provides a course multi-reward reinforcement learning method applied to a customized cultural graph model. In this method, the "determine the weight of each indicator according to the change in the evaluation value of each indicator" in the above step S130 can specifically include steps S130-1 to S130-2: Step S130 - 1 : Obtain a change score for each indicator based on the moving average and moving average standard deviation of each indicator, wherein the size of the change score represents the change in the evaluation value of the indicator.

[0049] In the embodiment of the present application, in order to balance the impact of different indicators on the customized text graph model, when the customized text graph model performs poorly on one or more of the indicators, the weights of these indicators are increased, thereby dynamically and adaptively optimizing multiple indicators at the same time. In order to achieve this, it is first necessary to determine the change in the evaluation value of each indicator, that is, whether the evaluation value of each indicator rises or falls in the global process. The embodiment of the present application reflects this through the moving average (Exponential Moving Average, EMA) and moving average standard deviation (Exponential Moving Standard Deviation, EMSD) of each indicator. Specifically, the change score of each indicator is calculated based on the moving average and moving average standard deviation of each indicator. The size of the change score represents the change in the evaluation value of the indicator.

[0050] Among them, the moving average and moving average standard deviation of the indicator are calculated as follows: The moving average value of the indicator in the current update round is obtained based on the evaluation value of the indicator and the moving average value of the previous update round; the moving average standard deviation of the indicator in the current update round is calculated based on the evaluation value of the indicator, the moving average value of the indicator in the current update round and the moving average standard deviation of the previous update round.

[0051] For example, the moving average of the indicator of the current update round and the moving average standard deviation of the current update round are expressed as: , , in, is the moving average of the mth indicator in the i-th update round, The moving average of the mth indicator in the i-1th update round, is the moving average standard deviation of the mth indicator in the i-th update round, The moving average standard deviation of the mth indicator in the i-1th update round, is the attenuation coefficient of the mth indicator, is the evaluation value of the mth indicator in the i-1th update round.

[0052] According to the moving average and moving average standard deviation of each indicator, the change score of each indicator is obtained, including: taking the evaluation value of the current update round minus the ratio of the moving average to the moving average standard deviation as the change score; for example, the change score can be expressed as: , in, is the change score of the mth indicator in the i-th update round.

[0053] Since the moving average of an indicator can reflect the average situation of the indicator in the past update rounds, if the indicator's evaluation value minus the moving average is positive, it means that the indicator's evaluation value is rising, and if the indicator's evaluation value minus the moving average is negative, it means that the indicator's evaluation value is falling; and since the ratio of the evaluation value of the current update round minus the moving average to the moving average standard deviation is used as the change score, the size of the moving average standard deviation can reflect the degree of increase or decrease in the indicator's evaluation value. Therefore, the size of the change score can characterize the change in the indicator's evaluation value.

[0054] Step S130 - 2 : Determine the weight of each indicator according to the change score of each indicator, and the weight of the indicator satisfies: the weight of the indicator with a smaller change score is greater than the weight of the indicator with a larger change score.

[0055] In the embodiment of the present application, a suitable weight is assigned to each indicator according to the change score of each indicator (i.e., the change in the evaluation value of the indicator). Based on the course learning method, higher weights are assigned to indicators that have decreased more, so that the customized text graph model is optimized first, and lower weights are assigned to indicators that have increased more. Therefore, the weights of the indicators satisfy the requirement that the weight of the indicator with a small change score is greater than the weight of the indicator with a large change score.

[0056] For example, the weight of the indicator satisfies that the weight of the indicator with a small change score is greater than the weight of the indicator with a large change score, which can be expressed as: , That is, if the change score of the mth indicator Less than or equal to the change score of the nth indicator , then the weight of the nth indicator Less than or equal to the weight of the mth indicator .

[0057] Specifically, the weight of each indicator is determined according to the change score of each indicator, including: According to the change score of each indicator, a course learning model is constructed, wherein the course learning model includes a first learning item and a second learning item, wherein the first learning item is used to control the sum of weights of all indicators to be minimum, and the second learning item is used to control the weight of each indicator to change within a preset range; By solving the course learning model, the initial weight of each indicator is obtained; The ratio of the initial weight of each indicator to the sum of the initial weights of all indicators is used as the weight of each indicator.

[0058] For example, the curriculum learning model is represented as: , in, It is the first learning item; It is the second learning term. The second learning term is a regular term used to control the weight of each indicator to change within a preset range, that is, to prevent the weight of the indicator from being too small or too large. When the weight is 1, it degenerates into a normal weighted reward sum. is the regularization coefficient of the mth index.

[0059] By solving the course learning model, the initial weight of each indicator is obtained. The initial weight of the indicator can be expressed as: , Finally, all weights are subjected to an L1 regularization to consider all indicators as a whole, that is, the ratio of the initial weight of each indicator to the sum of the initial weights of all indicators is used as the weight of each indicator. For example, the weight of an indicator can be expressed as: , The final reward signal It can be expressed as: , in, is the number of indicators, is the evaluation value of the mth indicator, The weight of the mth indicator.

[0060] In an embodiment of the present application, after obtaining the evaluation value and weight of each indicator, the model parameters of the customized text graph model are updated by reinforcement learning, so that the customized text graph model is optimized on multiple indicators at the same time, and a customized text graph model with good performance in multiple evaluation dimensions is obtained.

[0061] For example, updating the model parameters of the customized text graph model according to the reward signal can be expressed as: , Among them, t is the diffusion stage of the customized cultural graph model, is the hidden variable of the image in the tth diffusion stage, is the hidden variable of the image in the t-1th diffusion stage, is a reward signal, is the initial image hidden variable, y is the given condition, which generally refers to the text prompt word in the text graph.

[0062] In combination with the above embodiments, in one implementation, the present application embodiment also provides a course multi-reward reinforcement learning method applied to a customized text graph model. In this method, the customized text graph model is obtained by the following steps, specifically, including steps A1 to A4: Step A1: Obtain a sample image of a specific subject, and obtain a prompt word text containing a subject identifier.

[0063] Step A2: Input the sample image and the prompt word text into the text-generated graph model for noise addition and denoising to obtain a generated image.

[0064] Step A3: Calculate the loss function value between the sample image and the generated image according to the added noise of the Wensheng graph model in the noise adding process and the noise fitted by the Wensheng graph model in the denoising process.

[0065] Step A4: fine-tuning the model parameters of the document graph model according to the loss function value to obtain the customized document graph model.

[0066] In the embodiment of the present application, the customized text graph model is fine-tuned based on the fine-tuning customized text graph method to obtain a customized text graph model. The text graph model is a latent diffusion model (LDM), which is usually composed of a U-net as the main backbone network, a variational auto-encoder (VAE) that represents the image as a latent variable, and a text encoder for encoding the prompt word. , and contains the subject identifier Prompt word text , input the text graph model that needs to be fine-tuned to make it customized.

[0067] The sample image and the prompt word text are input into the text graph model for noise addition and denoising to obtain a generated image. In order to simplify the calculation, the difference between the sample image and the generated image is converted into the difference between the added noise in the noise addition process and the fitted noise in the denoising process.

[0068] For example, the loss function value between the sample image and the generated image can be expressed as: , in, is the hidden variable of the image in the tth diffusion stage, is the diffusion stage, is a text encoder, is the added noise in the noise adding process, is the noise fitted in the denoising process, that is, the U-shaped network in the Wensheng graph model is Fit the corresponding noise.

[0069] Further, according to the added noise of the Wensheng graph model in the noise adding process and the noise fitted by the Wensheng graph model in the denoising process, calculating the loss function value between the sample image and the generated image includes: The loss function value between the sample image and the generated image is calculated based on the added noise for the specific subject and the added noise for other subjects of the Vincent graph model in the noise addition process, and the noise fitted to the specific subject and the noise fitted to other subjects by the Vincent graph model in the denoising process.

[0070] In the embodiment of the present application, in order to prevent a large deviation in the generated image content, a regularization term of the subject category is added to the loss function to ensure that the category of the generated customized subject is correct.

[0071] For example, the loss function value between the sample image and the generated image can also be expressed as: , in, It is the added noise for other subjects in the noise adding process. It is a description text of the prompt words for other subjects. It is the diffusion stage for other entities. It is in The latent variables of the image in the diffusion stage, is the regularization term coefficient.

[0072] Finally, the model parameters of the text graph model are fine-tuned according to the loss function value to obtain the customized text graph model. In this way, by fine-tuning a certain part of the latent diffusion model (for example, the U-shaped network or the text encoder), the latent variables of the subject image are bound to the latent variables of the special identifier to achieve customized generation. After this step, given the subject image and the prompt words, the customized text graph model is already capable of generating pictures that meet both the subject requirements and the prompt words. At this point, the customized text graph model cannot perform well in multiple evaluation dimensions, so the model parameters of the customized text graph model are further updated through the above-mentioned steps S110 to S150, thereby obtaining a customized text graph model with good performance in multiple evaluation dimensions at the same time.

[0073] Reference Figure 2 As shown, Figure 2It is a flowchart of another method for multi-reward reinforcement learning of courses applied to a customized cultural graph model provided in an embodiment of the present application, the method comprising steps S210 to S260: Step S210: Obtain a customized text graph model.

[0074] Specifically, a sample image of a specific subject is obtained, and a prompt word text containing a subject identifier is obtained; the sample image and the prompt word text are input into a text graph model for noise addition and denoising to obtain a generated image; a loss function value between the sample image and the generated image is calculated based on the added noise of the text graph model in the noise addition process and the noise fitted by the text graph model in the denoising process; and the model parameters of the text graph model are fine-tuned based on the loss function value to obtain the customized text graph model.

[0075] Step S220: obtaining a picture of a specific subject generated by the customized text-based image model according to the prompt word.

[0076] Step S230: Evaluate the image based on indicators of multiple evaluation dimensions to obtain evaluation values ​​of multiple indicators, wherein the multiple evaluation dimensions include: prompt word accuracy, subject accuracy, and human preference, and each evaluation dimension includes at least one indicator.

[0077] Specifically, the image is evaluated based on indicators of multiple evaluation dimensions to obtain evaluation values ​​of multiple indicators, including: evaluating the similarity between the image and the prompt word through a prompt word accuracy evaluation model to obtain evaluation values ​​of indicators related to prompt word accuracy; evaluating the similarity between the subject in the image and the customized subject through a subject accuracy evaluation model to obtain evaluation values ​​of indicators related to subject accuracy; evaluating the degree of satisfaction of humans with the image through a human preference evaluation model to obtain evaluation values ​​of indicators related to human preference, wherein the human preference evaluation model is a reward model trained based on human annotation information.

[0078] Step S240: According to the moving average and moving average standard deviation of each indicator, obtain the change score of each indicator, the size of the change score represents the change of the evaluation value of the indicator; according to the change score of each indicator, determine the weight of each indicator, the weight of the indicator satisfies: the weight of the indicator with a smaller change score is greater than the weight of the indicator with a larger change score.

[0079] Step S250: Generate a reward signal according to the evaluation values ​​of the multiple indicators and the weight of each indicator.

[0080] Step S260: updating the model parameters of the customized text graph model according to the reward signal.

[0081] In the embodiment of the present application, multiple evaluation dimension indicators are selected as the reward signal of the customized text graph model, and the customized text graph model can be optimized in multiple evaluation dimensions. The weight of the indicator is determined according to the change of the evaluation value of each indicator, thereby resolving the reward conflict in the optimization process of multiple indicators, thereby dynamically and adaptively realizing the simultaneous optimization of multiple indicators, and further improving the performance of the customized text graph model in multiple evaluation dimensions (for example, prompt word accuracy, subject accuracy, and human preference).

[0082] Reference Figure 3 As shown, Figure 3 This is an overall architecture diagram of a course multi-reward reinforcement learning method applied to a customized text graph model provided in an embodiment of the present application. The method is divided into two stages. In the first stage, a customized text graph is fine-tuned: using multiple sample images containing customized subjects and prompt word text containing subject identification, the text graph model is customized and fine-tuned, so that the generated graph model has customization capabilities, and a customized text graph model is obtained.

[0083] The second stage is to conduct course multi-reward reinforcement learning: the picture is evaluated based on indicators of multiple evaluation dimensions, that is, the reward model of multiple evaluation dimensions (i.e., prompt word accuracy evaluation model, subject accuracy evaluation model, human preference evaluation model) is used to evaluate the picture, and the evaluation values ​​of multiple indicators are obtained. Then, the evaluation values ​​of multiple indicators are used as reward signals, and the model is further fine-tuned through reinforcement learning to maximize the reward. In addition, in order to cope with the challenge of reward conflict in the multi-reward optimization process, a course re-weighting mechanism is adopted, and a smaller weight is given to the indicator whose evaluation value changes as an increase; a larger weight is given to the indicator whose evaluation value changes as a decrease. In this way, the simultaneous optimization of multiple indicators is realized dynamically and adaptively, and the performance of the customized Wenshengtu model in multiple evaluation dimensions is further improved.

[0084] The present application embodiment also provides a course multi-reward reinforcement learning device applied to a customized cultural graph model, referring to Figure 4 As shown, Figure 4 : is a structural diagram of a course multi-reward reinforcement learning device applied to a customized cultural graph model provided in an embodiment of the present application, the device comprising: An acquisition module 410 is used to acquire a picture of a specific subject generated by a customized text image model according to a prompt word; An evaluation module 420 is used to evaluate the image based on indicators of multiple evaluation dimensions to obtain evaluation values ​​of multiple indicators, wherein the multiple evaluation dimensions include: prompt word accuracy, subject accuracy, and human preference, and each evaluation dimension includes at least one indicator; A determination module 430 is used to determine the weight of each indicator according to the change of the evaluation value of each indicator; A generating module 440, configured to generate a reward signal according to the evaluation values ​​of the plurality of indicators and the weight of each indicator; The updating module 450 is used to update the model parameters of the customized text graph model according to the reward signal.

[0085] In an optional embodiment, the determination module is also used to assign a smaller weight to an indicator whose evaluation value changes upward, wherein assigning a smaller weight means that the weight of the indicator is reduced relative to the last update round; and to assign a larger weight to an indicator whose evaluation value changes downward, wherein assigning a larger weight means that the weight of the indicator is increased relative to the last update round.

[0086] In an optional embodiment, the determining module includes: A change score module is used to obtain a change score of each indicator according to the moving average and moving average standard deviation of each indicator, wherein the size of the change score represents the change of the evaluation value of the indicator; The weight determination module is used to determine the weight of each indicator according to the change score of each indicator, and the weight of the indicator satisfies: the weight of the indicator with a small change score is greater than the weight of the indicator with a large change score.

[0087] In an optional embodiment, the weight determination module is also used to construct a course learning model according to the change score of each indicator, and the course learning model includes a first learning item and a second learning item, the first learning item is used to control the sum of the weights of all indicators to meet the minimum, and the second learning item is used to control the weight of each indicator to change within a preset range; by solving the course learning model, the initial weight of each indicator is obtained; the ratio of the initial weight of each indicator to the sum of the initial weights of all indicators is used as the weight of each indicator.

[0088] In an optional embodiment, the change score module is also used to obtain the moving average value of the indicator in the current update round based on the evaluation value of the indicator and the moving average value of the previous update round; and calculate the moving average standard deviation of the current update round based on the evaluation value of the indicator, the moving average value of the indicator in the current update round and the moving average standard deviation of the previous update round.

[0089] In an optional embodiment, the first evaluation module includes: A first evaluation submodule is used to evaluate the similarity between the image and the prompt word through a prompt word accuracy evaluation model to obtain an evaluation value of an indicator related to the prompt word accuracy; A second evaluation submodule is used to evaluate the similarity between the subject in the image and the customized subject through a subject accuracy evaluation model, and obtain an evaluation value of a subject accuracy-related index; The third evaluation submodule is used to evaluate the satisfaction degree of humans for the picture through a human preference evaluation model to obtain an evaluation value of human preference related indicators. The human preference evaluation model is a reward model trained based on human annotation information.

[0090] In an optional embodiment, the device further includes a model acquisition module, which is used to acquire the customized text graph model, and the model acquisition module includes: An acquisition module, used to acquire sample images of a specific subject, and acquire prompt word text containing a subject identifier; An input module, used for inputting the sample image and the prompt word text into the text graph model for noise addition and denoising to obtain a generated image; A calculation module, used for calculating the loss function value between the sample image and the generated image according to the added noise of the Wensheng graph model in the noise adding process and the noise fitted by the Wensheng graph model in the denoising process; A fine-tuning module is used to fine-tune the model parameters of the document graph model according to the loss function value to obtain the customized document graph model.

[0091] In an optional embodiment, the calculation module is further used to calculate the loss function value between the sample image and the generated image based on the added noise for the specific subject and the added noise for other subjects in the noise addition process of the Vincent graph model, and the noise fitted to the specific subject and the noise fitted to other subjects in the denoising process of the Vincent graph model.

[0092] The present application also provides an electronic device, referring to Figure 5 , Figure 5 Schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown, the electronic device 500 includes: a memory 510 and a processor 520. The memory 510 and the processor 520 are connected via a bus communication. A computer program is stored in the memory 510. The computer program can be executed on the processor 520 to implement the steps of the course multi-reward reinforcement learning method applied to the customized literary graph model described in the embodiment of the present application.

[0093] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the course multi-reward reinforcement learning method applied to the customized literary graph model described in the embodiment of the present application are implemented.

[0094] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the course multi-reward reinforcement learning method applied to the customized literary graph model described in the embodiment of the present application.

[0095] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0096] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods and devices according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0097] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0098] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0099] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0100] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0101] The above is a detailed introduction to a course multi-reward reinforcement learning method and device for a customized cultural graph model provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A course multi-reward reinforcement learning method applied to a customized cultural graph model, characterized in that: The method comprises: Get the picture of the specific subject generated by the customized text image model according to the prompt word; Evaluate the image based on indicators of multiple evaluation dimensions to obtain evaluation values ​​of multiple indicators, wherein the multiple evaluation dimensions include: prompt word accuracy, subject accuracy, and human preference, and each evaluation dimension includes at least one indicator; Determine the weight of each indicator based on the changes in the evaluation value of each indicator; Generate a reward signal according to the evaluation values ​​of the plurality of indicators and the weight of each indicator; Model parameters of the customized text graph model are updated according to the reward signal.

2. The method according to claim 1, characterized in that According to the changes in the evaluation value of each indicator, the weight of each indicator is determined, including: For the indicator whose evaluation value changes upward, a smaller weight is assigned, wherein the smaller weight means that the weight of the indicator is reduced relative to the previous update round; For the indicator whose evaluation value changes downward, a greater weight is assigned, and the assigning of a greater weight means that the weight of the indicator is increased relative to the previous update round.

3. The method according to claim 1, characterized in that According to the changes in the evaluation value of each indicator, the weight of each indicator is determined, including: According to the moving average and moving average standard deviation of each indicator, a change score of each indicator is obtained, and the size of the change score represents the change of the evaluation value of the indicator; The weight of each indicator is determined according to the change score of each indicator, and the weight of the indicator satisfies: the weight of the indicator with a smaller change score is greater than the weight of the indicator with a larger change score.

4. The method according to claim 3, characterized in that According to the change score of each indicator, the weight of each indicator is determined, including: According to the change score of each indicator, a course learning model is constructed, wherein the course learning model includes a first learning item and a second learning item, wherein the first learning item is used to control the sum of weights of all indicators to be minimum, and the second learning item is used to control the weight of each indicator to change within a preset range; By solving the course learning model, the initial weight of each indicator is obtained; The ratio of the initial weight of each indicator to the sum of the initial weights of all indicators is used as the weight of each indicator.

5. The method according to claim 3, characterized in that: The moving average and moving average standard deviation of the indicator are calculated as follows: According to the evaluation value of the indicator and the moving average value of the previous update round, the moving average value of the indicator of the current update round is obtained; The moving average standard deviation of the current update round is calculated based on the evaluation value of the indicator, the moving average of the indicator in the current update round, and the moving average standard deviation of the previous update round.

6. The method according to claim 1, characterized in that The image is evaluated based on indicators of multiple evaluation dimensions to obtain evaluation values ​​of multiple indicators, including: Using a prompt word accuracy evaluation model, the similarity between the image and the prompt word is evaluated to obtain an evaluation value of an indicator related to the prompt word accuracy; Using a subject accuracy evaluation model, the similarity between the subject in the image and the customized subject is evaluated to obtain an evaluation value of a subject accuracy-related indicator; The degree of satisfaction of humans with the picture is evaluated by a human preference evaluation model to obtain an evaluation value of a human preference-related index. The human preference evaluation model is a reward model trained based on human annotation information.

7. The method according to any one of claims 1 to 6, characterized in that: The customized cultural graph model is obtained in the following way: Obtain sample images of a specific subject, and obtain prompt word text containing the subject identification; Inputting the sample image and the prompt word text into the text graph model for noise addition and denoising to obtain a generated image; Calculating a loss function value between the sample image and the generated image according to the added noise of the Vincent graph model in the noise adding process and the noise fitted by the Vincent graph model in the denoising process; The model parameters of the document graph model are fine-tuned according to the loss function value to obtain the customized document graph model.

8. The method according to claim 7, characterized in that Calculating the loss function value between the sample image and the generated image according to the added noise of the Wensheng graph model in the noise adding process and the noise fitted by the Wensheng graph model in the denoising process, including: The loss function value between the sample image and the generated image is calculated based on the added noise for the specific subject and the added noise for other subjects of the Vincent graph model in the noise addition process, and the noise fitted to the specific subject and the noise fitted to other subjects by the Vincent graph model in the denoising process.

9. A course multi-reward reinforcement learning device applied to a customized text graph model, characterized in that: The device comprises: An acquisition module is used to acquire a picture of a specific subject generated by a customized text image model according to a prompt word; An evaluation module, used to evaluate the image based on indicators of multiple evaluation dimensions to obtain evaluation values ​​of multiple indicators, wherein the multiple evaluation dimensions include: prompt word accuracy, subject accuracy, and human preference, and each evaluation dimension includes at least one indicator; A determination module is used to determine the weight of each indicator according to the change of the evaluation value of each indicator; A generating module, configured to generate a reward signal according to the evaluation values ​​of the plurality of indicators and the weight of each indicator; An updating module is used to update the model parameters of the customized text graph model according to the reward signal.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the course multi-reward reinforcement learning method applied to the customized literary graph model described in any one of claims 1-8 are implemented.

Images