Model training method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a model training method and device, electronic equipment, computer storage medium and computer program product. The method comprises: obtaining a training data set; inputting the training data set into a multi-modal reward model; analyzing and processing each positive and negative sample image pair in the training data set through the multi-modal reward model to obtain an evaluation result of each positive and negative sample image pair; performing loss calculation based on the evaluation result and a preference reference label, and outputting a loss result of the multi-modal reward model; when the loss result of the multi-modal reward model meets a preset model training stop condition, obtaining a trained multi-modal reward model; and the multi-modal reward model is used for feedback incentive of image data evaluation and / or generation model. The present method improves the evaluation efficiency and flexibility, avoids the generation of reward utilization behavior in the reinforcement learning process, and thus improves the model generalization ability and accuracy of the generation model.

Description

Technical Field

[0001] This disclosure relates to the field of visual processing technology, and in particular to a model training method, apparatus, electronic device, computer storage medium, and computer program product. Background Technology

[0002] With the continuous development of large-scale post-training techniques, reinforcement learning from human feedback (RLHF) has gradually become a core technology for performance optimization in generative tasks such as image and video generation. This technology models human subjective preferences as a learnable neural network reward model and iteratively optimizes the generative model parameters through gradient backpropagation, achieving precise alignment between the generated results and human aesthetics and needs, thus significantly improving the practical value of the generative model.

[0003] In traditional technologies, reward model construction schemes typically follow two core logics: First, they use a text-image consistency evaluation algorithm as the core evaluation criterion, outputting reward scores by quantifying the degree of matching between generated content and text instructions; second, they design fixed instruction templates based on a certain evaluation dimension and generate instruction prompts, using the same instructions to guide the reward model in evaluation during both the training and inference phases.

[0004] However, in traditional technologies, reward models often focus on a single evaluation dimension, which cannot cover the multi-dimensional evaluation needs in generation tasks. This makes it difficult for generation models to achieve comprehensive performance optimization, ultimately resulting in poor generalization of generation models and unsatisfactory practical application effects. Summary of the Invention

[0005] This disclosure provides a model training method, apparatus, electronic device, computer storage medium, and computer program product. The technical solution of this disclosure is as follows:

[0006] According to a first aspect of the present disclosure, a model training method is provided, comprising:

[0007] Obtain a training dataset; the training dataset contains pairs of positive and negative sample images, and the pairs of positive and negative sample images are associated with instruction prompts and preference reference labels; the instruction prompts are randomly sampled from copies of instruction prompts in the target dimension, and are used to define the output evaluation benchmark of the target dimension for the multimodal reward model;

[0008] The training dataset is input into the multimodal reward model, and the multimodal reward model analyzes and processes each pair of positive and negative sample images in the training dataset to obtain the evaluation results of each pair of positive and negative sample images.

[0009] Based on the evaluation results and the preference reference labels, loss calculation is performed, and the loss result of the multimodal reward model is output.

[0010] When the loss result of the multimodal reward model meets the preset model training stopping condition, the trained multimodal reward model is obtained; the multimodal reward model is used for image data evaluation and / or feedback incentives for generating models.

[0011] In one exemplary embodiment, the method further includes:

[0012] A multi-dimensional evaluation system is constructed based on multiple pre-defined evaluation dimensions;

[0013] Based on the multi-dimensional evaluation system, a preset large language model is called to expand the text of each evaluation dimension, generating multiple instruction prompt word copies corresponding to each evaluation dimension.

[0014] Based on the multi-dimensional evaluation system and the multiple instruction prompt word copies corresponding to the multi-dimensional evaluation system, an instruction prompt word library is constructed. The instruction prompt word library contains instruction prompt word copies of each dimension in the multi-dimensional evaluation system, which are used to randomly sample instruction prompt words of the target dimension.

[0015] In one embodiment, obtaining the training dataset includes:

[0016] Obtain pairs of positive and negative sample images pre-labeled with preference reference tags;

[0017] For each pair of positive and negative sample images, at least one target evaluation dimension is matched, and based on the target evaluation dimension, the corresponding instruction prompt word is sampled from the instruction prompt word library;

[0018] A training dataset is constructed based on the positive and negative sample image pairs and the instruction prompts.

[0019] In an exemplary embodiment, the step of analyzing and processing each of the positive and negative sample image pairs in the training dataset using the multimodal reward model to obtain the evaluation results of each of the positive and negative sample image pairs includes:

[0020] The multimodal reward model is used to fuse and analyze the input positive and negative sample image pairs and the instruction prompt words, and outputs the probability prediction values ​​of the positive and negative sample image pairs for different discrimination results.

[0021] Based on the predicted probability values, the reward scores of the positive and negative sample image pairs under the corresponding evaluation dimensions are calculated using an activation function, and the reward scores are used as the evaluation results of the positive and negative sample image pairs.

[0022] In an exemplary embodiment, the step of calculating the loss based on the evaluation result and the preference reference label, and outputting the loss result of the multimodal reward model, includes:

[0023] Based on the evaluation results of each positive and negative sample image pair output by the multimodal reward model, the probability prediction value of each positive and negative sample image pair is extracted.

[0024] The cross-entropy loss calculation method is used to compare the extracted probability prediction value with the target probability distribution corresponding to the preference reference label to obtain and output the loss result of the multimodal reward model.

[0025] In an exemplary embodiment, obtaining the trained multimodal reward model when the loss result of the multimodal reward model meets a preset model training stopping condition includes:

[0026] Obtain the loss result of the multimodal reward model in the current training round;

[0027] Retrieve the preset model training stopping condition, and compare the loss result with the model training stopping condition to determine whether the loss result meets the model training stopping condition;

[0028] If the loss result does not meet the model training stopping condition, the training process of the multimodal reward model continues.

[0029] If the loss result satisfies the model training stopping condition, then the training process of the multimodal reward model is stopped, and the trained multimodal reward model is output.

[0030] In one exemplary embodiment, the method further includes:

[0031] During the training of the generative model, the target image pairs output by the generative model are obtained, and target instruction prompt words are randomly selected from the instruction prompt word library;

[0032] Based on the trained multimodal reward model and the target instruction prompt, the target image pair is evaluated to obtain an evaluation score for the target image pair.

[0033] Based on the evaluation score, a reward signal is generated, which is used for parameter optimization during the training process of the generative model.

[0034] According to a second aspect of the present disclosure, a model training apparatus is provided, comprising:

[0035] The acquisition unit is configured to acquire a training dataset; the training dataset contains pairs of positive and negative sample images, and the pairs of positive and negative sample images are associated with instruction prompts and preference reference labels; the instruction prompts are randomly sampled from copies of instruction prompts in the target dimension, and are used to define the output evaluation benchmark of the target dimension for the multimodal reward model;

[0036] The analysis unit is configured to input the training dataset into the multimodal reward model, and analyze and process each of the positive and negative sample image pairs in the training dataset through the multimodal reward model to obtain the evaluation results of each of the positive and negative sample image pairs;

[0037] The loss calculation unit is configured to perform loss calculation based on the evaluation results and the preference reference labels, and output the loss result of the multimodal reward model;

[0038] The output unit is configured to obtain a trained multimodal reward model when the loss result of the multimodal reward model meets a preset model training stopping condition; the multimodal reward model is used for image data evaluation and / or feedback incentives for generating models.

[0039] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0040] processor;

[0041] Memory used to store the processor's executable instructions;

[0042] The processor is configured to execute the instructions to implement the model training method described in the first aspect above.

[0043] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the model training method described in the first aspect.

[0044] According to a fifth aspect of the present disclosure, a computer program product is provided, the computer program product including instructions that, when executed by a processor of an electronic device, enable the electronic device to perform the model training method described in the first aspect.

[0045] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0046] By constructing a training dataset containing positive and negative sample image pairs, multi-dimensional instruction prompts, and preference reference labels, and combining it with a multimodal reward model, a comprehensive evaluation of multi-dimensional image features is achieved. Simultaneously, by leveraging a dynamic instruction prompt sampling mechanism and loss optimization process, the trained multimodal reward model can accurately align with human multi-dimensional preferences, completing a comprehensive evaluation of multiple aspects of the image. This significantly improves evaluation efficiency and flexibility; it avoids reward exploitation during reinforcement learning, ensuring the effectiveness of generative model parameter optimization. Furthermore, it provides more robust and reliable technical support for image data evaluation and reinforcement learning feedback for generative models, improving the generalization ability and accuracy of the generative model.

[0047] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0048] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0049] Figure 1 This is a flowchart illustrating a model training method according to an exemplary embodiment.

[0050] Figure 2 This is a flowchart illustrating a method for constructing instruction prompts according to an exemplary embodiment.

[0051] Figure 3 This is a flowchart illustrating a step of constructing a training dataset according to an exemplary embodiment.

[0052] Figure 4 This is a flowchart illustrating an evaluation process for obtaining positive and negative sample image pairs according to an exemplary embodiment.

[0053] Figure 5 This is a flowchart illustrating a step for calculating the loss result of a model according to an exemplary embodiment.

[0054] Figure 6 This is a flowchart illustrating the steps of a multimodal reward model that has completed training, according to an exemplary embodiment.

[0055] Figure 7 This is a flowchart illustrating an evaluation and feedback method for a generative model based on a multimodal reward model, according to an exemplary embodiment.

[0056] Figure 8 This is a schematic diagram illustrating the basic framework of a multimodal reward model based on multidimensional dynamic instructions, according to an exemplary embodiment.

[0057] Figure 9 This is a block diagram illustrating a model training apparatus according to an exemplary embodiment.

[0058] Figure 10 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0059] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0060] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0061] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0062] Figure 1 This is a flowchart illustrating a model training method according to an exemplary embodiment, such as... Figure 1 As shown, this model training method is used in a terminal and includes the following steps:

[0063] In step S110, the training dataset is obtained.

[0064] The training dataset includes pairs of positive and negative sample images, each associated with multi-dimensional instruction prompts and preference reference labels. The instruction prompts are randomly sampled from copies of instruction prompts in the target dimension and are used to define the output evaluation benchmark for the target dimension of the multimodal reward model.

[0065] In implementation, during the training of the multimodal reward model, the terminal first acquires a training dataset. This training dataset is constructed with human preference evaluation as its core and incorporates multi-dimensional dynamic instruction features. Specifically, the training dataset contains pairs of positive and negative sample images that have been evaluated and labeled by humans (i.e., carrying preset preference reference labels). Each pair of positive and negative sample images is explicitly associated with an instruction prompt, thus all positive and negative sample image pairs in the entire training dataset are associated with multi-dimensional instruction prompts.

[0066] Optionally, the multi-dimensional instruction prompts can be derived from a pre-built offline instruction prompt library. The construction process of this library is as follows: First, N image evaluation dimensions are determined (including image-text consistency, aesthetics, lighting, image quality, etc.). For each evaluation dimension, initial prompts are constructed. These initial prompts are then input into a pre-trained language model, which expands and amplifies them without altering their semantics, generating multiple copies of the instruction prompts. Finally, these copies are integrated to form an offline instruction prompt library containing each evaluation dimension and its corresponding copy. This ensures that each positive and negative sample image pair can be matched with diverse instruction prompts for the corresponding evaluation dimension, providing sufficient and compliant material support for dynamic sampling.

[0067] In step S120, the training dataset is input into the multimodal reward model, and the multimodal reward model analyzes and processes each positive and negative sample image pair in the training dataset to obtain the evaluation results of each positive and negative sample image pair.

[0068] In practice, the terminal inputs the completed training dataset into the pre-deployed multimodal reward model, which then performs comprehensive analysis and processing on each positive and negative sample image pair to obtain the evaluation results.

[0069] Specifically, for each positive and negative sample image pair, the terminal first randomly selects one or more corresponding dimension instruction prompts from its associated offline instruction prompt word library. This positive and negative sample image pair is used as visual input, and randomly selected instruction prompts matching the positive and negative sample image pair are used as text input. Both are then fed into the multimodal reward model. After receiving the visual and text inputs, the multimodal reward model performs deep fusion analysis on the two types of information, outputting the probability prediction values ​​for the positive judgment result (e.g., "Yes") and negative judgment result (e.g., "No") corresponding to the positive and negative sample image pair. Subsequently, based on these probability prediction values, the terminal calculates the reward score for the corresponding evaluation dimension using an activation function. This reward score is the final evaluation result for the positive and negative sample image pair.

[0070] In step S130, loss is calculated based on the evaluation results and preference reference labels, and the loss result of the multimodal reward model is output.

[0071] In practice, the terminal calculates the loss based on the obtained evaluation results and the preference reference labels in the training dataset, and then outputs the loss result of the multimodal reward model.

[0072] Specifically, the terminal determines the target probability distribution based on the preference reference label: if the positive sample image in the positive-negative sample image pair meets the requirements of the corresponding evaluation dimension, the target probability distribution is 0 for the positive discrimination result and 1 for the negative discrimination result; if the positive sample image does not meet the requirements of the corresponding evaluation dimension, the target probability distribution is 0 for the positive discrimination result and 1 for the negative discrimination result. Subsequently, the terminal extracts the probability prediction values ​​of the positive and negative discrimination results output by the multimodal reward model from the evaluation results, uses the cross-entropy loss calculation method, compares the extracted probability prediction values ​​with the determined target probability distribution, quantifies the degree of deviation between the model prediction results and the true preference label, and finally obtains and outputs the loss result of the multimodal reward model in the current training state.

[0073] In step S140, when the loss result of the multimodal reward model meets the preset model training stopping condition, the trained multimodal reward model is obtained.

[0074] Among them, the multimodal reward model is used for image data evaluation and / or feedback incentives for generative models.

[0075] In practice, the terminal continuously monitors the loss result calculated during the training iteration of the multimodal reward model. When the loss result meets the preset model training stopping condition, the training stops and the trained multimodal reward model is obtained.

[0076] Specifically, the preset model training stopping conditions include, but are not limited to: the loss result being lower than a set threshold, the loss result remaining stable without a decreasing trend for a preset number of consecutive training rounds, and the training rounds reaching a preset maximum value. This embodiment does not limit the model training stopping conditions for the multimodal reward model. This embodiment uses the model training stopping condition of the loss result being lower than a set threshold as an example to describe the model training process: The terminal compares the current loss result with the preset loss threshold. If it is determined that the preset loss condition is met, for example, the loss result is less than or equal to the preset loss threshold, the terminal immediately stops the model training process, saves all parameters of the current model, and outputs the trained multimodal reward model; if it is determined that the preset loss condition is not met, the model parameter update process is triggered. After adjusting the model parameters through gradient backpropagation, the training process is resumed until the loss result meets the stopping condition. The trained multimodal reward model can be flexibly applied to image data evaluation scenarios (improving evaluation stability through multiple forward propagation averaging) and reinforcement learning feedback incentive scenarios for generated models (providing reward signals through dynamic instruction sampling), achieving multi-dimensional and highly robust evaluation and feedback support.

[0077] In the aforementioned model training method, a training dataset containing positive and negative sample image pairs, multi-dimensional instruction prompts, and preference reference labels is constructed. This dataset is then combined with a multi-modal reward model to achieve a comprehensive evaluation of the multi-dimensional features of images. Furthermore, by leveraging a dynamic instruction prompt sampling mechanism and loss optimization process, the trained multi-modal reward model can accurately align with human multi-dimensional preferences, completing a comprehensive evaluation of multiple aspects of the image. This significantly improves evaluation efficiency and flexibility. It also avoids reward exploitation during reinforcement learning, ensuring the effectiveness of parameter optimization in the generative model. Consequently, it provides more robust and reliable technical support for image data evaluation and reinforcement learning feedback for the generative model, improving the model's generalization ability and accuracy.

[0078] In one exemplary embodiment, such as Figure 2 As shown, the method also includes:

[0079] In step 201, a multi-dimensional evaluation system is constructed based on multiple preset evaluation dimensions.

[0080] In implementation, the terminal constructs a multi-dimensional evaluation system based on core evaluation indicators of human interest in the image generation task. Specifically, the terminal can first define N key evaluation dimensions, which may include, but are not limited to: image-text consistency (i.e., the degree of matching between the generated image and the text instruction), aesthetics (i.e., the visual aesthetic value of the image), lighting effects (i.e., the light distribution and tonal gradation of the image), and image quality (i.e., the image's sharpness, noise level, etc.). Then, the terminal forms a dimension set d∈{ , …, Each evaluation dimension corresponds to a clear evaluation standard, ensuring that subsequent instruction prompt generation and model evaluation have a clear basis, while meeting the needs of image generation tasks for performance improvement in multiple aspects.

[0081] In step 202, based on the multi-dimensional evaluation system, a preset large language model is called to expand the text of each evaluation dimension, generating multiple instruction prompt word copies corresponding to each evaluation dimension.

[0082] In implementation, the terminal, based on the established multi-dimensional evaluation system, calls a pre-trained large language model to expand the text of each evaluation dimension, generating multiple instruction prompt word copies for each evaluation dimension. These instruction prompt word copies serve as candidate instructions for the multimodal reward model, used to clarify the dimensional evaluation rules of the multimodal reward model. Specifically, for each evaluation dimension, the terminal uses the pre-trained large language model to generate initial prompt words to guide the multimodal model in evaluation, based on the construction prompt word requirements for each evaluation dimension. Subsequently, the terminal further controls the pre-trained large language model to rewrite the initial prompt words in various ways without changing their semantics, ultimately generating k semantically consistent but differently expressed instruction prompt word copies for each evaluation dimension. This enriches the single-dimensional instructions, providing ample material for subsequent dynamic sampling.

[0083] In step 203, an instruction prompt word library is constructed based on the multi-dimensional evaluation system and multiple instruction prompt word copies corresponding to the multi-dimensional evaluation system.

[0084] The instruction prompt word library contains copies of instruction prompt words for each dimension in the multi-dimensional evaluation system, which are used to randomly sample instruction prompt words for the target dimension.

[0085] In implementation, the terminal constructs a structured command prompt word library based on a multi-dimensional evaluation system and multiple command prompt word copies corresponding to each evaluation dimension. Specifically, the terminal categorizes and organizes the generated command prompt word copies according to the evaluation dimensions, and associates each evaluation dimension with its corresponding k command prompt word copies, forming an offline-stored command prompt word library. ,in, Representative dimension The Each instruction prompt word is a copy; this instruction prompt word library has the characteristics of being dimensionally distinguishable and having copies that can be randomly called.

[0086] In this embodiment, a multi-dimensional evaluation system is constructed, and multiple instruction prompt word copies are generated for each evaluation dimension using a preset large language model. These copies are then integrated to form a structured instruction prompt word library. This not only enables the instruction prompt words to have multi-dimensional coverage capabilities, meeting the needs of multi-faceted performance evaluation of the generated results in image generation tasks and avoiding the limitations of single-dimensional evaluation, but also provides sufficient material for dynamic random sampling in the subsequent model training and evaluation process through the diversification and systematic management of instruction prompt words. This effectively improves the adaptability and generalization of the multimodal reward model to different expression instructions. At the same time, the offline-built instruction prompt word library can optimize the execution efficiency of subsequent processes.

[0087] In one exemplary embodiment, such as Figure 3 As shown, in step S110, obtaining the training dataset can be achieved through the following steps:

[0088] In step S211, positive and negative sample image pairs with pre-labeled preference reference tags are obtained.

[0089] In implementation, the terminal collects pre-evaluated and screened image pairs. Each image pair contains two images with differences in quality or performance. One image is a positive sample image identified in the pre-evaluation (i.e., an image that better meets the corresponding evaluation requirements), and the other is a negative sample image (i.e., an image that relatively does not meet the evaluation requirements), thus forming a positive and negative sample image pair. Meanwhile, based on the pre-evaluation results, a preset preference reference label is labeled for each positive and negative sample image pair, which clearly indicates the evaluation tendency that the positive sample image in the image pair should correspond to. This provides a real and reliable label reference for loss calculation in the subsequent model training process, ensuring that the model can learn evaluation logic that fits human preferences.

[0090] In step S212, at least one target evaluation dimension is matched for each positive and negative sample image pair, and the corresponding instruction prompt word is sampled from the instruction prompt word library based on the target evaluation dimension.

[0091] In practice, the terminal matches at least one target evaluation dimension to each positive and negative sample image pair, and samples the corresponding command prompt word from a pre-built command prompt word library based on the target evaluation dimension.

[0092] Specifically, based on the core evaluation requirements of the image generation task, the terminal selects an appropriate target evaluation dimension for each positive and negative sample image pair from a multi-dimensional evaluation system (covering dimensions such as image-text consistency, aesthetics, lighting effects, and image quality), ensuring that the target evaluation dimension matches the core differences between the positive and negative sample image pairs. The instruction prompt word library is an offline-built structured database containing multiple semantically consistent but differently expressed instruction prompt word copies corresponding to each evaluation dimension. For a matched target evaluation dimension, the terminal randomly samples one or more instruction prompt words from the set of instruction prompt word copies corresponding to that target evaluation dimension, achieving dynamic selection of instruction prompt words and avoiding the model overfitting problem caused by fixed instructions.

[0093] In step S213, a training dataset is constructed based on positive and negative sample image pairs and instruction prompts.

[0094] In implementation, the terminal integrates and constructs a training dataset based on positive and negative sample image pairs, matched target evaluation dimensions, and sampled instruction prompts. Specifically, the terminal associates and binds each positive and negative sample image pair with its corresponding preference reference label, matched target evaluation dimension, and randomly sampled instruction prompts to form a structured training data unit. Subsequently, all training data units are aggregated and integrated to form a complete training dataset. This training dataset includes both visual input (positive and negative sample image pairs) for model learning and text input (dynamically sampled instruction prompts) for guiding model evaluation, along with preference reference labels for loss calculation. It can comprehensively support the training process of multimodal reward models and help models learn multi-dimensional and dynamic evaluation capabilities.

[0095] In this embodiment, positive and negative sample image pairs with pre-labeled preference reference tags are obtained. At least one target evaluation dimension is matched for each positive and negative sample image pair, and instruction prompts corresponding to the dimension are sampled from the instruction prompt word library. This achieves accurate matching between the evaluation dimension and the image samples, allowing the model to focus on different core evaluation indicators for training. At the same time, the overfitting problem caused by fixed instructions is avoided by using dynamically sampled instruction prompts. Finally, by integrating positive and negative sample image pairs with corresponding instruction prompts to construct a training dataset, the training dataset simultaneously possesses visual samples that conform to human preferences, multi-dimensional evaluation guidance, and dynamic text input features, effectively improving the model's adaptability and generalization to multi-dimensional scenarios.

[0096] In one exemplary embodiment, such as Figure 4 As shown, in step S120, the evaluation results of each positive and negative sample image pair in the training dataset are obtained by analyzing and processing them using a multimodal reward model. This can be achieved through the following steps:

[0097] In step 401, the input positive and negative sample image pairs and instruction prompts are fused and analyzed using a multimodal reward model to output the probability prediction values ​​of different discrimination results for the positive and negative sample image pairs.

[0098] In implementation, the terminal uses a multimodal reward model to perform cross-modal fusion analysis on each input positive and negative sample image pair and its corresponding instruction prompts, outputting the probability prediction value of the positive and negative sample image pair for different discrimination results. Specifically, the multimodal reward model, as a model capable of processing both visual and linguistic data, first extracts features from the input visual information (positive and negative sample image pairs, including positive and negative sample images) and textual information (instruction prompts of corresponding dimensions randomly sampled from an offline instruction prompt library). Then, through the cross-modal fusion mechanism within the model, it deeply integrates the two types of features, fully exploring the correlation between image features and the evaluation dimension requirements implied by the instruction prompts. Finally, based on the fused features, it outputs the probability prediction values ​​corresponding to the two types of discrimination results, namely, the probability prediction value of affirmative discrimination results (such as "Yes") and the probability prediction value of negative discrimination results (such as "No"), providing core data support for the subsequent calculation of reward scores. The specific calculation formula for this probability prediction value is as follows:

[0099]

[0100] in For a multimodal reward model, The positive and negative sample image pairs are used as the visual content input to the multimodal reward model; These are instruction prompts, serving as the text content input to the multimodal reward model.

[0101] In step 402, based on the probability prediction value, the reward score of the positive and negative sample image pairs under the corresponding evaluation dimension is calculated by the activation function, and the reward score is used as the evaluation result of the positive and negative sample image pairs.

[0102] In implementation, the terminal calculates the reward score for the positive and negative image pairs in the corresponding evaluation dimension based on the probability prediction value output by the multimodal reward model and uses the activation function. This reward score is then used as the evaluation result for the positive and negative image pairs. Specifically, the terminal uses the softmax activation function, commonly used in generative multimodal models, and substitutes the probability prediction values ​​of affirmative and negative discrimination results into the calculation to obtain the normalized probabilities of the two types of discrimination results. This normalized probability accurately quantifies the degree to which the positive and negative image pairs meet the requirements of the corresponding evaluation dimension. The normalized probability corresponding to the discrimination result that matches the evaluation dimension requirements is the reward score of the positive and negative image pairs in the current evaluation dimension. This reward score directly reflects the multimodal reward model's evaluation conclusion for the positive and negative image pairs in the target dimension.

[0103] Specifically, the probability calculation for affirmative and negative judgments using the softmax activation function is as follows:

[0104]

[0105]

[0106] in, This is the final probability prediction value of the affirmative judgment result after normalization, and its value ranges from (0,1). This is the final probability prediction value of the negative discrimination result after normalization, and its value ranges from (0,1). and These are the probability predictions for positive and negative judgments made by the multimodal reward model.

[0107] In this embodiment, a multimodal reward model is used to perform cross-modal fusion analysis on positive and negative sample image pairs and corresponding instruction prompts. This fully explores the deep correlation between visual features and text evaluation requirements, outputting probability prediction values ​​for different discrimination results, ensuring that the evaluation logic is highly consistent with the evaluation criteria of the target dimension. Then, the probability prediction values ​​are normalized using an activation function to obtain the reward score. This not only realizes the quantitative presentation of the evaluation results, giving the model a clear basis for judging the performance of the image in the corresponding dimension, but also ensures the rationality and discriminativeness of the reward score, effectively reflecting the degree to which positive and negative sample image pairs conform to human preferences.

[0108] In one exemplary embodiment, such as Figure 5 As shown, the specific processing steps for calculating the loss based on the evaluation results and preference reference labels in step 130, and outputting the loss result of the multimodal reward model, include:

[0109] In step 501, based on the evaluation results of each positive and negative sample image pair output by the multimodal reward model, the probability prediction value of each positive and negative sample image pair is extracted.

[0110] In implementation, the terminal extracts the probability prediction value corresponding to each positive and negative sample image pair based on the evaluation results of each positive and negative sample image pair output by the multimodal reward model. Specifically, the evaluation results of the positive and negative sample image pairs output by the multimodal reward model are presented in the form of reward scores under the corresponding evaluation dimensions. These reward scores are generated by the multimodal reward model after fusing and analyzing visual and textual information. The terminal selectively extracts the probability prediction values ​​of affirmative judgments (such as "Yes") and negative judgments (such as "No") from the evaluation results. These two types of probability prediction values ​​directly reflect the model's judgment tendency on whether the positive and negative sample image pairs meet the requirements of the target evaluation dimension, providing core data support for subsequent comparison calculations with preference reference labels.

[0111] In step 502, the cross-entropy loss calculation method is used to compare the extracted probability prediction values ​​with the target probability distribution corresponding to the preference reference labels to obtain and output the loss result of the multimodal reward model.

[0112] In implementation, the terminal uses the cross-entropy loss calculation method to compare the extracted probability prediction value with the target probability distribution corresponding to the preference reference label, and then calculates and outputs the loss result of the multimodal reward model.

[0113] Specifically, the terminal first determines the target probability distribution based on the preference reference label: if the human-annotated positive sample image in the positive-negative sample image pair meets the corresponding evaluation dimension requirements, the softmax function result should be [1,0], then the target probability distribution is a positive judgment probability of 1 and a negative judgment probability of 0; if it does not meet the requirements, the target probability distribution is a positive judgment probability of 0 and a negative judgment probability of 1. Subsequently, the terminal substitutes the predicted probability values ​​of the positive or negative judgment results output by the multimodal reward model, along with the determined target probability distribution, into the cross-entropy loss calculation formula. By quantifying the deviation between the model's predicted results and the true human preference label, the loss result of the multimodal reward model in the current training round is obtained. This loss result will serve as the basis for optimizing model parameters and guide parameter adjustments during subsequent gradient backpropagation. The cross-entropy loss calculation formula is shown below:

[0114]

[0115] in, This represents the cross-entropy loss value. This indicates the probability of a positive judgment in the preference reference label. This indicates the probability of a negative judgment in the preference reference label. and These are the probability predictions for positive and negative judgments made by the multimodal reward model.

[0116] In this embodiment, by acquiring the loss result of the current training round of the multimodal reward model in real time and comparing it with the preset training stopping conditions, it is possible to dynamically monitor the training progress and fitting effect of the model, accurately control the training rhythm, and avoid poor model performance due to insufficient training or overfitting due to overtraining. It also ensures that the model can stop in time when it reaches the optimal training state. This not only ensures that the trained model has the evaluation ability to fit human multidimensional preferences and good robustness, but also avoids unnecessary consumption of computing power and wasted time, thus helping to efficiently produce multimodal reward models that meet the needs of practical applications.

[0117] In one exemplary embodiment, such as Figure 6 As shown, the specific processing steps of step 140 include:

[0118] In step 601, the loss result of the current training round of the multimodal reward model is obtained.

[0119] In implementation, the terminal obtains the loss result output by the multimodal reward model in the current training round. This loss result is obtained by the terminal based on the evaluation results of the positive and negative sample image pairs output by the multimodal reward model, combined with the target probability distribution corresponding to the preference reference label, and calculated using the cross-entropy loss method. Its value directly reflects the degree of deviation between the model's prediction result and the actual human preference in the current training round, providing a core quantitative basis for judging the subsequent training stopping conditions.

[0120] In step 602, the preset model training stopping condition is retrieved, and the loss result is compared with the model training stopping condition to determine whether the loss result meets the model training stopping condition.

[0121] In implementation, the terminal retrieves preset model training stopping conditions and compares the loss result of the current training epoch with each of these conditions to determine whether the loss result meets the stopping requirements. Specifically, the preset model training stopping conditions include multi-dimensional judgment criteria, including the loss result being lower than a preset threshold, the loss result remaining stable without a decreasing trend for a preset number of consecutive epochs, and the total number of model training epochs reaching a preset maximum value. The terminal matches and verifies the current loss result against the above criteria to comprehensively determine whether the model has reached the optimal training state.

[0122] In step 603, if the loss result does not meet the model training stopping condition, the training process of the multimodal reward model continues.

[0123] In practice, if the terminal determines that the current loss result does not meet the preset model training stopping condition after comparison, it continues to execute the model training process to update and optimize the model parameters of the multimodal reward model. Specifically, the terminal adjusts the network parameters of the multimodal reward model based on the current loss result using the gradient backpropagation algorithm, and then returns to the training dataset input stage to repeat the positive and negative sample image pair analysis and processing, loss calculation, and other processes to enter the next round of model training until the loss result meets the stopping condition.

[0124] In step 604, if the loss result meets the model training stopping condition, the training process of the multimodal reward model is stopped, and the trained multimodal reward model is output.

[0125] In practice, if the terminal determines that the current loss result meets the preset model training stopping condition after comparison, the training process of the multimodal reward model is immediately terminated. Subsequently, the terminal saves all network parameters of the model in the current training state, generates and outputs the trained multimodal reward model. This trained multimodal reward model can be directly applied to multidimensional evaluation scenarios of image data, or provide accurate feedback incentive signals for the reinforcement learning process of the generated model.

[0126] In this embodiment, by acquiring the loss results of the current training round of the multimodal reward model in real time and making targeted comparisons and judgments based on preset training stopping conditions, the training progress and fitting effect of the model can be dynamically monitored, the training rhythm can be accurately controlled, and the poor model performance due to insufficient training or the overfitting problem due to overtraining can be avoided. Furthermore, through the closed-loop logic of "continuing training if the stopping condition is not met, and terminating training if the stopping condition is met", it is ensured that the model can stop in time when it reaches the optimal training state. This not only ensures that the trained model has the evaluation ability to conform to human multidimensional preferences and good robustness, but also avoids unnecessary consumption of computing power and wasted time. At the same time, it provides a clear judgment basis for the stability and reliability of training results, and helps to efficiently produce multimodal reward models that meet the needs of practical applications.

[0127] In one exemplary embodiment, such as Figure 7 As shown, the method also includes:

[0128] In step 701, during the training of the generative model, the target image pairs output by the generative model are obtained, and target instruction prompts are randomly selected from the instruction prompt word library.

[0129] In implementation, during the training of the generative model, the terminal acquires the target image pairs output by the model and randomly selects target instruction prompts from a pre-built instruction prompt lexicon. The target image pairs are a set of images with performance differences generated by the generative model based on the input text instructions, including high-quality images that conform to human preferences and images to be optimized by the generative model. The instruction prompt lexicon is a structured offline lexicon covering multiple dimensions such as image-text consistency, aesthetics, lighting, and image quality. For each target image pair generation scenario, the terminal randomly samples an instruction prompt from the corresponding set of instruction prompt copy sets for each evaluation dimension as the target instruction prompt. This dynamic selection avoids reward exploitation issues caused by fixed instructions, ensuring the objectivity and generalization of subsequent evaluations.

[0130] In step 702, based on the trained multimodal reward model and the target instruction prompt, the target image pair is evaluated to obtain the evaluation score of the target image pair.

[0131] In implementation, the terminal evaluates the acquired target image pairs based on the trained multimodal reward model and the selected target instruction prompts, obtaining an evaluation score for each target image pair. Specifically, the terminal inputs the target image pairs as visual input and the target instruction prompts as text input into the trained multimodal reward model. This multimodal reward model, through a cross-modal feature fusion mechanism, deeply mines the correlation between the visual features of the target image pairs and the evaluation requirements of the target instruction prompts, outputting probability prediction values ​​for different discrimination results for the target image pairs. Then, the probability prediction values ​​are normalized using an activation function to obtain a quantitative evaluation score for the target image pairs in the corresponding evaluation dimension. This evaluation score directly reflects the degree to which the target image pairs meet the evaluation requirements of the target instruction prompts.

[0132] In step 703, a reward signal is generated based on the evaluation score. The reward signal is used to generate parameter optimization during the model training process.

[0133] In implementation, the terminal generates a reward signal based on the obtained evaluation score. This reward signal is used for parameter optimization during the training process of the generation model. Specifically, the terminal maps the evaluation score to a reward signal that meets the requirements of reinforcement learning training. If the evaluation score of the high-quality image in the target image pair is significantly higher than that of the image to be optimized, a positive reward signal is generated; otherwise, a negative reward signal is generated. After receiving the reward signal, the generation model uses it as an optimization guide to adjust its own network parameters through the gradient backpropagation algorithm, gradually optimizing the performance of image generation. This makes the generated images more in line with human subjective preferences in multiple dimensions, realizing the collaborative training and iterative optimization of the generation model and the multimodal reward model.

[0134] In this embodiment, image evaluation is performed using a trained multimodal reward model. This not only leverages the cross-modal fusion capability and multi-dimensional evaluation characteristics of the multimodal model to achieve accurate quantitative evaluation of how well the target image aligns with human preferences, but also effectively avoids reward exploitation through random sampling of instruction prompts, thus improving the objectivity and generalization of the evaluation. Furthermore, the reward signal generated based on the evaluation score provides clear guidance for the parameter optimization of the generative model, allowing it to continuously iterate towards directions that conform to human multi-dimensional preferences during reinforcement learning, significantly improving the quality and effect of the generated images, and ultimately achieving collaborative optimization between the multimodal reward model and the generative model.

[0135] In an exemplary embodiment, taking an application system composed of a trained multimodal reward model and a generative model as an example in a real image processing scenario, the processing procedure is as follows: First, the input data includes a text instruction provided by the user (e.g., "Generate an image of children's clothing with a star and moon pattern") and a set of images to be evaluated (containing at least two generated images with subtle differences) output by the generative model based on the text instruction. The terminal first inputs the text instruction into the instruction prompt word library, and randomly selects a target instruction prompt word from the instruction copy set corresponding to the evaluation dimension (e.g., "Evaluate the image-text relevance of these two images, which one better fits the description of 'children's clothing with a star and moon pattern'?"). Then, the set of images to be evaluated is used as visual input, and the target instruction prompt word is used as text input. The input is fed into the trained multimodal reward model. This model first extracts the visual features of the image group and the text features of the instruction prompts, and then deeply correlates the two types of features through a cross-modal fusion module. It then outputs the original probability scores of the image group to be evaluated as "meets the instruction requirements" and "does not meet the instruction requirements". The scores are then normalized by the softmax activation function to obtain the quantified evaluation scores. The final output data is the ranking of the evaluation scores of the image group under the corresponding evaluation dimension (e.g., the image-text consistency score of image A is 0.85 and that of image B is 0.62). At the same time, the generation model receives the reward signal corresponding to the evaluation score and automatically adjusts its own parameters to optimize the subsequent image generation effect, realizing a complete processing closed loop from input text, image generation to evaluation feedback and model iteration.

[0136] In an exemplary embodiment, a basic framework for a multimodal reward model based on multidimensional dynamic instructions is given. Within this basic framework, such as... Figure 8As shown: Based on preset evaluation dimensions such as "text-image consistency" and "image aesthetics," the terminal calls a large language model to expand and amplify the initial instructions for each dimension, generating a corresponding instruction prompt vocabulary (e.g., the text-image consistency instruction vocabulary contains multiple sets of semantically similar evaluation instructions). Next, the terminal randomly samples an instruction prompt word from the preset evaluation dimensions of the multi-dimensional instruction prompt vocabulary (e.g., "Evaluate the text-image relevance of two images; is image 1 better than image 2?"), and simultaneously acquires the two images to be evaluated. Subsequently, the terminal inputs the sampled instruction and the two images into a trained multimodal model, driving the multimodal model to perform fusion analysis on the two images and the instruction prompt word. Finally, the terminal receives the vocabulary probability output by the multimodal model, completing the evaluation task of the set of images under the corresponding evaluation dimension.

[0137] It should be understood that, although Figures 1-7 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figures 1-7 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0138] It is understood that the same / similar parts between the various embodiments of the methods described above in this specification can be referred to each other. Each embodiment focuses on the differences from other embodiments, and relevant parts can be referred to the description of other method embodiments.

[0139] Figure 9 This is a block diagram illustrating a model training apparatus according to an exemplary embodiment. (Refer to...) Figure 9 The device includes an acquisition unit 901, an analysis unit 902, a loss calculation unit 903, and an output unit 904.

[0140] The acquisition unit 901 is configured to acquire a training dataset; the training dataset contains pairs of positive and negative sample images, and the pairs of positive and negative sample images are associated with instruction prompts and preference reference labels; the instruction prompts are randomly sampled from copies of instruction prompts in the target dimension, and are used to define the output evaluation benchmark of the target dimension for the multimodal reward model;

[0141] Analysis unit 902 is configured to input the training dataset into a multimodal reward model, analyze and process each positive and negative sample image pair in the training dataset through the multimodal reward model, and obtain the evaluation results of each positive and negative sample image pair.

[0142] The loss calculation unit 903 is configured to perform loss calculation based on the evaluation results and preference reference labels, and output the loss result of the multimodal reward model;

[0143] Output unit 904 is configured to obtain a trained multimodal reward model when the loss result of the multimodal reward model meets the preset model training stopping condition; the multimodal reward model is used for image data evaluation and / or feedback incentives for generating models.

[0144] In one exemplary embodiment, the device 900 further includes:

[0145] The first building unit is configured to execute multiple preset evaluation dimensions to build a multi-dimensional evaluation system;

[0146] The calling unit is configured to execute a multi-dimensional evaluation system, call a preset large language model to expand the text of each evaluation dimension, and generate multiple instruction prompt word copies corresponding to each evaluation dimension.

[0147] The second building unit is configured to execute an instruction prompt word library based on a multi-dimensional evaluation system and multiple instruction prompt word copies corresponding to the multi-dimensional evaluation system. The instruction prompt word library contains instruction prompt word copies of each dimension in the multi-dimensional evaluation system, which are used to randomly sample instruction prompt words of the target dimension.

[0148] In one exemplary embodiment, the acquisition unit 901 includes:

[0149] The first acquisition subunit is configured to acquire pairs of positive and negative sample images pre-labeled with preference reference tags;

[0150] The sampling subunit is configured to perform matching at least one target evaluation dimension for each positive and negative sample image pair, and to sample the instruction prompt word corresponding to the target evaluation dimension from the instruction prompt word library based on the target evaluation dimension;

[0151] The first construction subunit is configured to perform the construction of a training dataset based on positive and negative sample image pairs and instruction prompts.

[0152] In one exemplary embodiment, the analysis unit 902 includes:

[0153] The output subunit is configured to perform a fusion analysis of the input positive and negative sample image pairs and instruction prompts using a multimodal reward model, and output the probability prediction values ​​of the positive and negative sample image pairs for different discrimination results.

[0154] The first computational subunit is configured to perform a probability-based prediction, calculate the reward score of the positive and negative sample image pairs in the corresponding evaluation dimension through an activation function, and use the reward score as the evaluation result of the positive and negative sample image pairs.

[0155] In one exemplary embodiment, the loss calculation unit 903 includes:

[0156] The extraction subunit is configured to perform the evaluation results of each positive and negative sample image pair based on the output of the multimodal reward model, and extract the probability prediction value of each positive and negative sample image pair;

[0157] The second calculation subunit is configured to perform a cross-entropy loss calculation method, which compares the extracted probability prediction values ​​with the target probability distribution corresponding to the preference reference labels, and then calculates and outputs the loss result of the multimodal reward model.

[0158] In one exemplary embodiment, the output unit 904 includes:

[0159] The second acquisition subunit is configured to acquire the loss result of the current training round of the multimodal reward model;

[0160] The judgment subunit is configured to retrieve the preset model training stopping condition and compare the loss result with the model training stopping condition to determine whether the loss result meets the model training stopping condition.

[0161] The first processing subunit is configured to execute the training process of the multimodal reward model if the loss result does not meet the model training stopping condition.

[0162] The second processing subunit is configured to execute the training process of the multimodal reward model if the loss result meets the model training stopping condition, and output the trained multimodal reward model.

[0163] In one exemplary embodiment, the device 900 includes:

[0164] The selection module is used to obtain the target image pairs output by the generative model during the training process of the generative model, and to randomly select target instruction prompts from the instruction prompt word library;

[0165] The evaluation module is used to evaluate the target image pairs based on the trained multimodal reward model and target instruction prompts, and obtain the evaluation score of the target image pairs.

[0166] The generation module is used to generate reward signals based on the evaluation scores. These reward signals are used to optimize parameters during the model training process.

[0167] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0168] Figure 10 This is a block diagram illustrating an electronic device 1000 for model training according to an exemplary embodiment. For example, the electronic device 1000 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0169] Reference Figure 10 The electronic device 1000 may include one or more of the following components: processing component 1002, memory 1004, power supply component 1006, multimedia component 1008, audio component 1010, input / output (I / O) interface 1012, sensor component 1014, and communication component 1016.

[0170] Processing component 1002 typically controls the overall operation of electronic device 1000, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 1002 may include one or more processors 1020 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 1002 may include one or more modules to facilitate interaction between processing component 1002 and other components. For example, processing component 1002 may include a multimedia module to facilitate interaction between multimedia component 1008 and processing component 1002.

[0171] Memory 1004 is configured to store various types of data to support the operation of electronic device 1000. Examples of such data include instructions for any application or method operating on electronic device 1000, contact data, phonebook data, messages, pictures, videos, etc. Memory 1004 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, optical disk, or graphene storage.

[0172] Power supply component 1006 provides power to various components of electronic device 1000. Power supply component 1006 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 1000.

[0173] Multimedia component 1008 includes a screen that provides an output interface between the electronic device 1000 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 1008 includes a front-facing camera and / or a rear-facing camera. When the electronic device 1000 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0174] Audio component 1010 is configured to output and / or input audio signals. For example, audio component 1010 includes a microphone (MIC) configured to receive external audio signals when electronic device 1000 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 1004 or transmitted via communication component 1016. In some embodiments, audio component 1010 also includes a speaker for outputting audio signals.

[0175] I / O interface 1012 provides an interface between processing component 1002 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0176] Sensor assembly 1014 includes one or more sensors for providing state assessments of various aspects of electronic device 1000. For example, sensor assembly 1014 can detect the on / off state of electronic device 1000, the relative positioning of components such as the display and keypad of electronic device 1000, changes in position of electronic device 1000 or its components, the presence or absence of user contact with electronic device 1000, orientation or acceleration / deceleration of device 1000, and temperature changes of electronic device 1000. Sensor assembly 1014 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1014 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 1014 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0177] Communication component 1016 is configured to facilitate wired or wireless communication between electronic device 1000 and other devices. Electronic device 1000 can access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof. In one exemplary embodiment, communication component 1016 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 1016 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0178] In an exemplary embodiment, the electronic device 1000 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0179] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 1004 including instructions, which can be executed by a processor 1020 of an electronic device 1000 to perform the above-described method. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0180] In an exemplary embodiment, a computer program product is also provided, the computer program product including instructions that can be executed by the processor 1020 of the electronic device 1000 to perform the above method.

[0181] It should be noted that the above-mentioned apparatus, electronic equipment, computer-readable storage medium, computer program product, etc., may also include other implementation methods according to the description of the method embodiments. For specific implementation methods, please refer to the description of the relevant method embodiments, which will not be elaborated here.

[0182] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0183] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A model training method, characterized in that, include: Obtain a training dataset; the training dataset contains pairs of positive and negative sample images, and the pairs of positive and negative sample images are associated with instruction prompts and preference reference labels; The instruction prompt words are randomly sampled from the instruction prompt word copies in the target dimension and are used to define the output evaluation benchmark of the target dimension for the multimodal reward model; The training dataset is input into the multimodal reward model, and the multimodal reward model analyzes and processes each pair of positive and negative sample images in the training dataset to obtain the evaluation results of each pair of positive and negative sample images. Based on the evaluation results and the preference reference labels, loss calculation is performed, and the loss result of the multimodal reward model is output. When the loss result of the multimodal reward model meets the preset model training stopping condition, the trained multimodal reward model is obtained; the multimodal reward model is used for image data evaluation and / or feedback incentives for generating models.

2. The model training method according to claim 1, characterized in that, The method further includes: A multi-dimensional evaluation system is constructed based on multiple pre-defined evaluation dimensions; Based on the multi-dimensional evaluation system, a preset large language model is called to expand the text of each evaluation dimension, generating multiple instruction prompt word copies corresponding to each evaluation dimension. Based on the multi-dimensional evaluation system and the multiple instruction prompt word copies corresponding to the multi-dimensional evaluation system, an instruction prompt word library is constructed. The instruction prompt word library contains instruction prompt word copies of each dimension in the multi-dimensional evaluation system, which are used to randomly sample instruction prompt words of the target dimension.

3. The model training method according to claim 2, characterized in that, The acquisition of the training dataset includes: Obtain pairs of positive and negative sample images pre-labeled with preference reference tags; For each pair of positive and negative sample images, at least one target evaluation dimension is matched, and based on the target evaluation dimension, the corresponding instruction prompt word is sampled from the instruction prompt word library; A training dataset is constructed based on the positive and negative sample image pairs and the instruction prompts.

4. The model training method according to claim 1, characterized in that, The step of analyzing and processing each pair of positive and negative sample images in the training dataset using the multimodal reward model to obtain the evaluation result of each pair of positive and negative sample images includes: The multimodal reward model is used to fuse and analyze the input positive and negative sample image pairs and the instruction prompt words, and outputs the probability prediction values ​​of the positive and negative sample image pairs for different discrimination results. Based on the predicted probability values, the reward scores of the positive and negative sample image pairs under the corresponding evaluation dimensions are calculated using an activation function, and the reward scores are used as the evaluation results of the positive and negative sample image pairs.

5. The model training method according to claim 1, characterized in that, The step of calculating the loss based on the evaluation results and the preference reference labels, and outputting the loss result of the multimodal reward model, includes: Based on the evaluation results of each positive and negative sample image pair output by the multimodal reward model, the probability prediction value of each positive and negative sample image pair is extracted. The cross-entropy loss calculation method is used to compare the extracted probability prediction value with the target probability distribution corresponding to the preference reference label to obtain and output the loss result of the multimodal reward model.

6. The model training method according to claim 1, characterized in that, The step of obtaining a trained multimodal reward model when the loss result of the multimodal reward model meets a preset model training stopping condition includes: Obtain the loss result of the multimodal reward model in the current training round; Retrieve the preset model training stopping condition, and compare the loss result with the model training stopping condition to determine whether the loss result meets the model training stopping condition; If the loss result does not meet the model training stopping condition, the training process of the multimodal reward model continues. If the loss result satisfies the model training stopping condition, then the training process of the multimodal reward model is stopped, and the trained multimodal reward model is output.

7. The model training method according to any one of claims 1 to 5, characterized in that, The method further includes: During the training of the generative model, the target image pairs output by the generative model are obtained, and target instruction prompt words are randomly selected from the instruction prompt word library; Based on the trained multimodal reward model and the target instruction prompt, the target image pair is evaluated to obtain an evaluation score for the target image pair. Based on the evaluation score, a reward signal is generated, which is used for parameter optimization during the training process of the generative model.

8. A model training device, characterized in that, include: The acquisition unit is configured to acquire a training dataset; the training dataset contains pairs of positive and negative sample images, and the pairs of positive and negative sample images are associated with instruction prompts and preference reference labels; The instruction prompt words are randomly sampled from the instruction prompt word copies in the target dimension and are used to define the output evaluation benchmark of the target dimension for the multimodal reward model; The analysis unit is configured to input the training dataset into the multimodal reward model, and analyze and process each of the positive and negative sample image pairs in the training dataset through the multimodal reward model to obtain the evaluation results of each of the positive and negative sample image pairs; The loss calculation unit is configured to perform loss calculation based on the evaluation results and the preference reference labels, and output the loss result of the multimodal reward model; The output unit is configured to execute the following: when the loss result of the multimodal reward model meets the preset model training stopping condition, the trained multimodal reward model is obtained. The multimodal reward model is used for feedback incentives in image data evaluation and / or generation models.

9. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the model training method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the model training method as described in any one of claims 1 to 7.

11. A computer program product, the computer program product comprising instructions, characterized in that, When the instruction is executed by the processor of the electronic device, the electronic device is able to perform the model training method as described in any one of claims 1 to 7.

Images