Multi-dimensional fine-grained reward method and device for image and video generation

Abstract

The invention relates to the technical field of computer vision and multi-modal generation, in particular to a multi-dimensional fine-grained reward method and device for image and video generation, and the method comprises the steps: constructing an evaluation system of each target image or video, extracting a binary judgment result of each image or video from the evaluation system of each image or video; training a pre-constructed visual language base model by using each image or video, the multi-dimensional multi-choice question of each image or video and the binary judgment result of each image or video to generate a multi-dimensional fine-grained reward model; inputting the to-be-detected image or video into the multi-dimensional fine-grained reward model to generate an actual binary judgment result; and performing linear weighting on the actual binary judgment result to predict a visual preference result of the to-be-detected image or video. Therefore, the problem that an optimized result is not ideal due to the fact that an existing human feedback reinforcement learning method excessively optimizes or weakens some factors is solved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and multimodal generation technology, in particular to the field of human feedback alignment technology for multimodal generation, and in particular to a multi-dimensional fine-grained reward method and device for image and video generation. Background Art

[0002] Visual generative models, including text-to-image and text-to-video generation, have developed rapidly recently. Based on text descriptions, these visual generative models can generate high-quality images or videos. Inspired by reinforcement learning with human feedback (RLHF) in large language models (LLMs), researchers found that aligning text to images according to human preferences is crucial due to various challenges such as alignment, fidelity, aesthetics, and security. Some works such as ImageReward developed diffusion model optimization methods based on these reward models or preference datasets. With the development of text-to-video models, similar evaluation and optimization methods have also been tried and improved.

[0003] Despite the progress, current RLHF methods for text-to-vision models still face huge challenges. The first is that the reward model is biased and uninterpretable. The current reward model learns from human preferences, which contains many trade-offs between different factors, resulting in preference bias. The second is that it is difficult to evaluate the dynamic quality of the video. The third is that existing RLHF methods tend to over-optimize or weaken certain factors, resulting in unsatisfactory results after optimization. Summary of the invention

[0004] The present invention provides a multi-dimensional fine-grained reward method and device for image and video generation to solve the problems of over-optimization or weakening of certain factors in existing human feedback reinforcement learning methods, resulting in unsatisfactory optimized results.

[0005] The first aspect of the present invention provides a multi-dimensional fine-grained reward method for image and video generation, comprising the following steps: constructing an evaluation system for each target image or video, and extracting the binary judgment results of each image or video in the evaluation system of each image or video; training a pre-constructed visual language base model using each image or video, the multi-dimensional multiple-choice questions of each image or video, and the binary judgment results of each image or video to generate a multi-dimensional fine-grained reward model; inputting the image or video to be tested into the multi-dimensional fine-grained reward model to generate an actual binary judgment result; linearly weighting the actual binary judgment result to predict the visual preference result of the image or video to be tested.

[0006] Optionally, constructing an evaluation system for each target image or video and extracting a binary judgment result from the evaluation system for each image or video includes:

[0007] Sending each target image or video and its corresponding multi-dimensional multiple-choice question to a target annotator, and instructing the target annotator to annotate each target image or video according to a pre-designed multi-dimensional multiple-choice question;

[0008] After the target annotator completes the annotation, decomposing the options of the multi-dimensional multiple-choice question into multiple judgment questions to form an evaluation system for each image or video;

[0009] A binary judgment result of each image or video is extracted in the evaluation system of each image or video.

[0010] Optionally, the training of a pre-built visual language base model using each image or video, the multi-dimensional multiple-choice question of each image or video, and the binary judgment result of each image or video to generate a multi-dimensional fine-grained reward model includes:

[0011] Performing balanced sampling on each image or video, the multi-dimensional multiple-choice question of each image or video, and the binary judgment result of each image or video to obtain a balanced sampled training data set;

[0012] The visual language base model is fine-tuned using the balanced sampled training dataset to generate the multi-dimensional fine-grained reward model.

[0013] Optionally, the linearly weighting the actual binary judgment result to predict the visual preference result of the image or video to be tested includes:

[0014] Mapping the actual binary judgment result into a preset feature value, and linearly weighting the preset feature value to obtain a weighted sum result;

[0015] classifying the weighted sum results into preset category labels according to a pre-constructed human preference dataset;

[0016] The weighted sum result and the preset category label are used to perform linear regression to predict the visual preference result of the image or video to be tested.

[0017] The second aspect of the present invention provides a multi-dimensional fine-grained reward device for image and video generation, including: building an extraction module, used to build an evaluation system for each target image or video, and extracting the binary judgment results of each image or video in the evaluation system of each image or video; a training module, used to train a pre-built visual language base model using each image or video, the multi-dimensional multiple-choice questions of each image or video, and the binary judgment results of each image or video to generate a multi-dimensional fine-grained reward model; a generation module, used to input the image or video to be tested into the multi-dimensional fine-grained reward model to generate an actual binary judgment result; a prediction module, used to linearly weight the actual binary judgment result to predict the visual preference result of the image or video to be tested.

[0018] Optionally, the construction extraction module includes:

[0019] A labeling unit, used to send each target image or video and its corresponding multi-dimensional multiple-choice question to a target annotator, and instruct the target annotator to label each target image or video according to a pre-designed multi-dimensional multiple-choice question;

[0020] A decomposition unit, configured to decompose the options of the multi-dimensional multiple-choice question into a plurality of judgment questions after the target annotator has completed the annotation, so as to form an evaluation system for each image or video;

[0021] The extraction unit is used to extract the binary judgment result of each image or video in the evaluation system of each image or video.

[0022] Optionally, the training module includes:

[0023] A sampling unit, used for performing balanced sampling on each image or video, the multi-dimensional multiple-choice question of each image or video, and the binary judgment result of each image or video, so as to obtain a training data set after balanced sampling;

[0024] A fine-tuning unit is used to fine-tune the visual language base model using the balanced sampled training data set to generate the multi-dimensional fine-grained reward model.

[0025] Optionally, the prediction module includes:

[0026] A weighting unit, used for mapping the actual binary judgment result into a preset characteristic value, and performing linear weighting on the preset characteristic value to obtain a weighted sum result;

[0027] A classification unit, configured to classify the weighted sum result into a preset category label according to a pre-constructed human preference dataset;

[0028] A prediction unit is used to perform linear regression using the weighted sum result and the preset category label to predict the visual preference result of the image or video to be tested.

[0029] The third aspect of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-dimensional fine-grained reward method for image and video generation as described in the above embodiments.

[0030] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the multi-dimensional fine-grained reward method for image and video generation as described above.

[0031] The multi-dimensional fine-grained reward method and device for image and video generation proposed in the embodiments of the present invention construct a multi-objective fine-grained multimodal evaluation system and reward model, so as to provide accurate evaluation results, and develop a multi-objective optimization algorithm based on the accurate evaluation results to stably adjust the visual generation model, avoid over-optimization or lack of optimization of certain factors, and significantly improve the multimodal generation effect.

[0032] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0034] Figure 1 A flowchart of a multi-dimensional fine-grained reward method for image and video generation provided by an embodiment of the present invention;

[0035] Figure 2 A flowchart of a multi-dimensional fine-grained reward method for image and video generation provided by an embodiment of the present invention;

[0036] Figure 3 A block diagram of a multi-dimensional fine-grained reward device for image and video generation provided by an embodiment of the present invention;

[0037] Figure 4 A block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0038] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0039] The following describes a multi-dimensional fine-grained reward method and device for image and video generation according to an embodiment of the present invention with reference to the accompanying drawings.

[0040] Figure 1 A flowchart of a multi-dimensional fine-grained reward method for image and video generation provided by an embodiment of the present invention.

[0041] like Figure 1 As shown, the multi-dimensional fine-grained reward method for image and video generation includes the following steps:

[0042] In step S101, an evaluation system for each target image or video is constructed, and a binary judgment result of each image or video is extracted from the evaluation system for each image or video.

[0043] In some embodiments, constructing an evaluation system for each target image or video, and extracting a binary judgment result from the evaluation system for each image or video, includes:

[0044] Send each target image or video and its corresponding multi-dimensional multiple-choice question to a target annotator, and instruct the target annotator to annotate each target image or video according to the pre-designed multi-dimensional multiple-choice question;

[0045] After the target annotator completes the annotation, the options of the multi-dimensional multiple-choice question are decomposed into multiple judgment questions to form an evaluation system for each image or video;

[0046] A binary judgment result of each image or video is extracted in the evaluation system of each image or video.

[0047] In the actual implementation process, Figure 2 As shown, when evaluating images or videos, human preferences are often the result of the interaction of multiple factors, so it is necessary to strike a balance between various considerations. In order to systematically deconstruct human preferences, an embodiment of the present invention develops a fine-grained, multi-dimensional preference decomposition framework.

[0048] Design multiple-choice questions of each dimension for each target image or video in advance, send each target image or video and its corresponding multi-dimensional multiple-choice question to the target annotator, and instruct the target annotator to annotate each target image or video according to the pre-designed multi-dimensional multiple-choice question, where the annotation can include 3 million questions for 48k images and 2 million questions for 33k videos.

[0049] Furthermore, based on a fine-grained, multi-dimensional preference decomposition framework, during the annotation process, the annotators are asked to choose the option that best suits the image. After the annotators complete the annotation, the options of the multi-dimensional multiple-choice questions are decomposed into multiple judgment (yes / no) questions, ultimately forming an evaluation system for evaluating each image or video, which can be used for subsequent evaluation and reward model training.

[0050] It should be noted that in order to address the challenges of video evaluation, extensive observations of the dynamic content of the video, such as motion stability or motion quality, can be included in the multiple-choice questions.

[0051] In step S102, the pre-built visual language base model is trained using each image or video, the multi-dimensional multiple-choice question of each image or video, and the binary judgment result of each image or video to generate a multi-dimensional fine-grained reward model.

[0052] In some embodiments, a pre-built visual language base model is trained using each image or video, a multi-dimensional multiple-choice question for each image or video, and a binary judgment result for each image or video to generate a multi-dimensional fine-grained reward model, including:

[0053] Perform balanced sampling on each image or video, the multi-dimensional multiple-choice questions of each image or video, and the binary judgment results of each image or video to obtain a balanced sampling training data set;

[0054] The visual-language base model is fine-tuned using the balanced sampled training dataset to generate a multi-dimensional fine-grained reward model.

[0055] In the actual implementation process, Figure 2 As shown, in the evaluation system of each image or video, the binary judgment result of each image or video is extracted to obtain millions of annotated binary judgment results; then, each image or video, the multi-dimensional multiple-choice question of each image or video, and the binary judgment result of each image or video are balanced and sampled to solve the imbalance problem between positive and negative instances, ensure that the number of positive and negative instances related to each judgment question is roughly equal, and obtain the balanced instructions composed of judgment questions (i.e., the training data set after balanced sampling); finally, the pre-built visual language base model is fine-tuned using the balanced instructions composed of judgment questions to generate the multi-dimensional fine-grained reward model. Among them, CogVLM2 is used as the visual language base model for image understanding, and CogVLM2-Video is used as the visual language base model for video understanding.

[0056] In step S103, the image or video to be tested is input into the multi-dimensional fine-grained reward model to generate an actual binary judgment result.

[0057] In step S104, the actual binary judgment results are linearly weighted to predict the visual preference result of the image or video to be tested.

[0058] In some embodiments, linear weighting is performed on the actual binary judgment results to predict the visual preference results of the image or video to be tested, including:

[0059] Mapping the actual binary judgment result to a preset feature value, and linearly weighting the preset feature value to obtain a weighted sum result;

[0060] Classify weighted sum results into preset category labels based on a pre-built human preference dataset;

[0061] Linear regression is performed using the weighted sum results and preset category labels to predict the visual preference results of the image or video to be tested.

[0062] In the actual implementation process, Figure 2 As shown, the image or video to be tested is input into the multi-dimensional fine-grained reward model to generate a series of binary (yes or no) answers (i.e., the actual binary judgment results), and the series of binary (yes or no) answers are mapped to preset feature values, such as mapping "yes" and "no" to "1" and "-1" respectively, and constructing a feature vector, and linearly weighting the feature vector using a set of linear weights to predict human preferences, which is specifically expressed as follows:

[0063] score=x T w=x1w1+x2w2+......+x n w n

[0064] Among them, x1...x n is the eigenvector, w1...w n is a set of linear weights.

[0065] In order to learn these parameters, the embodiment of the present invention collects human preference data sets in advance, calculates the feature difference of each pair of data, and assigns corresponding labels according to the human preference data sets. Then, a logistic regression model is used for regression to predict the visual preference results of the image or video to be tested. The visual preference result (probability) is given by the following formula:

[0066]

[0067] The objective function to be minimized during the prediction process is defined as:

[0068] Loss(w)=-Ε[ylog(σ(Δx T w))+(1-y)log(1-σ(Δx Tw))]

[0069] Among them, Ε is the expected value, y is the preference label, and σ is the sigmoid calculation function.

[0070] In addition, if Figure 2 As shown, to ensure that all dimensions are reasonably promoted, the problem of multi-dimensional fine-grained reward methods for image and video generation is merged into these dimensions and formulated as rewards. Given two images, a dominance pair is defined to ensure that each dimension is not weakened in preference.

[0071]

[0072] Among them, R() is the reward model, is the evaluation dimension, and k is the dimension number.

[0073] In summary, the multi-dimensional fine-grained reward method for image and video generation proposed in the embodiment of the present invention has the following features:

[0074] Beneficial effects:

[0075] (1) We decompose the factors that influence human preferences by designing a unified annotation system for image and video generation;

[0076] 2) It has high accuracy and high interpretability in predicting human preferences, which is much better than existing preference prediction methods, especially in video evaluation, which is 17.2% higher than VideoScore;

[0077] 3) The visual preference results can be introduced into the multi-objective preference optimization (MPO) algorithm to stably adjust the visual generation model and avoid over-optimization or lack of optimization for certain factors. In addition, the model using MPO outperforms the model directly adjusted using manual annotation or other reward models.

[0078] Next, the multi-dimensional fine-grained reward device for image and video generation proposed in accordance with an embodiment of the present invention will be described with reference to the accompanying drawings.

[0079] Figure 3 It is a block diagram of a multi-dimensional fine-grained reward device for image and video generation according to an embodiment of the present invention.

[0080] like Figure 3 As shown, the multi-dimensional fine-grained reward device 30 for image and video generation includes: a construction and extraction module 301, a training module 302, a generation module 303 and a prediction module 304.

[0081] Among them, the construction and extraction module 301 is used to construct an evaluation system for each target image or video, and extract the binary judgment result of each image or video in the evaluation system of each image or video. The training module 302 is used to train the pre-constructed visual language base model using each image or video, the multi-dimensional multiple-choice questions of each image or video, and the binary judgment result of each image or video to generate a multi-dimensional fine-grained reward model. The generation module 303 is used to input the image or video to be tested into the multi-dimensional fine-grained reward model to generate the actual binary judgment result. The prediction module 304 is used to linearly weight the actual binary judgment result to predict the visual preference result of the image or video to be tested.

[0082] In some embodiments, constructing the extraction module 301 includes:

[0083] A labeling unit, used to send each target image or video and its corresponding multi-dimensional multiple-choice question to a target annotator, and instruct the target annotator to label each target image or video according to a pre-designed multi-dimensional multiple-choice question;

[0084] A decomposition unit is used to decompose the options of the multi-dimensional multiple-choice question into multiple judgment questions after the target annotator completes the annotation, so as to form an evaluation system for each image or video;

[0085] The extraction unit is used to extract the binary judgment result of each image or video in the evaluation system of each image or video.

[0086] In some embodiments, the training module 302 includes:

[0087] A sampling unit, used for performing balanced sampling on each image or video, the multi-dimensional multiple-choice question of each image or video, and the binary judgment result of each image or video, so as to obtain a training data set after balanced sampling;

[0088] A fine-tuning unit is used to fine-tune the visual-language base model using the balanced sampled training dataset to generate a multi-dimensional fine-grained reward model.

[0089] In some embodiments, the prediction module 304 includes:

[0090] A weighting unit, used to map the actual binary judgment result into a preset feature value, and linearly weight the preset feature value to obtain a weighted sum result;

[0091] A classification unit, which is used to classify the weighted sum results into preset category labels based on a pre-constructed human preference dataset;

[0092] The prediction unit is used to perform linear regression using the weighted sum result and the preset category label to predict the visual preference result of the image or video to be tested.

[0093] It should be noted that the above explanation of the embodiment of the multi-dimensional fine-grained reward method for image and video generation is also applicable to the multi-dimensional fine-grained reward device for image and video generation of this embodiment, and will not be repeated here.

[0094] According to the multi-dimensional fine-grained reward device for image and video generation proposed in the embodiment of the present invention, a multi-objective fine-grained multimodal evaluation system and reward model are constructed, so that accurate evaluation results can be given. A multi-objective optimization algorithm can be developed based on the accurate evaluation results to stably adjust the visual generation model, avoid over-optimization or lack of optimization of certain factors, and significantly improve the multimodal generation effect.

[0095] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device may include:

[0096] Memory 401 , processor 402 , and a computer program stored in the memory 401 and executable on the processor 402 .

[0097] When the processor 402 executes the program, the multi-dimensional fine-grained reward method for image and video generation provided in the above embodiment is implemented.

[0098] Furthermore, the electronic device further comprises:

[0099] The communication interface 403 is used for communication between the memory 401 and the processor 402 .

[0100] The memory 401 is used to store computer programs that can be executed on the processor 402 .

[0101] The memory 401 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0102] If the memory 401, the processor 402 and the communication interface 403 are implemented independently, the communication interface 403, the memory 401 and the processor 402 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0103] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can communicate with each other through an internal interface.

[0104] The processor 402 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0105] An embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-dimensional fine-grained reward method for image and video generation as described above.

[0106] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0107] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "N" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0108] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

[0109] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0110] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0111] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0112] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0113] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A multi-dimensional fine-grained reward method for image and video generation, characterized in that: The following steps are involved: Constructing an evaluation system for each target image or video, and extracting a binary judgment result of each image or video in the evaluation system for each image or video; Training a pre-built visual language base model using each image or video, the multi-dimensional multiple-choice question of each image or video, and the binary judgment result of each image or video to generate a multi-dimensional fine-grained reward model; Inputting the image or video to be tested into the multi-dimensional fine-grained reward model to generate an actual binary judgment result; The actual binary judgment results are linearly weighted to predict the visual preference results of the image or video to be tested.

2. The multi-dimensional fine-grained reward method for image and video generation according to claim 1, characterized in that: The step of constructing an evaluation system for each target image or video and extracting a binary judgment result from the evaluation system for each image or video includes: Sending each target image or video and its corresponding multi-dimensional multiple-choice question to a target annotator, and instructing the target annotator to annotate each target image or video according to a pre-designed multi-dimensional multiple-choice question; After the target annotator completes the annotation, decomposing the options of the multi-dimensional multiple-choice question into multiple judgment questions to form an evaluation system for each image or video; A binary judgment result of each image or video is extracted in the evaluation system of each image or video.

3. The multi-dimensional fine-grained reward method for image and video generation according to claim 1, characterized in that: The method of training a pre-built visual language base model using each image or video, the multi-dimensional multiple-choice question of each image or video, and the binary judgment result of each image or video to generate a multi-dimensional fine-grained reward model includes: Performing balanced sampling on each image or video, the multi-dimensional multiple-choice question of each image or video, and the binary judgment result of each image or video to obtain a balanced sampled training data set; The visual language base model is fine-tuned using the balanced sampled training dataset to generate the multi-dimensional fine-grained reward model.

4. The multi-dimensional fine-grained reward method for image and video generation according to claim 1, characterized in that: The linear weighting of the actual binary judgment result to predict the visual preference result of the image or video to be tested includes: Mapping the actual binary judgment result into a preset feature value, and linearly weighting the preset feature value to obtain a weighted sum result; classifying the weighted sum results into preset category labels according to a pre-constructed human preference dataset; The weighted sum result and the preset category label are used to perform linear regression to predict the visual preference result of the image or video to be tested.

5. A multi-dimensional fine-grained reward device for image and video generation, characterized in that: include: Constructing an extraction module, which is used to construct an evaluation system for each target image or video, and extracting a binary judgment result of each image or video in the evaluation system of each image or video; A training module, for training a pre-built visual language base model using each image or video, the multi-dimensional multiple-choice question of each image or video, and the binary judgment result of each image or video to generate a multi-dimensional fine-grained reward model; A generation module, used for inputting the image or video to be tested into the multi-dimensional fine-grained reward model to generate an actual binary judgment result; The prediction module is used to perform linear weighting on the actual binary judgment result to predict the visual preference result of the image or video to be tested.

6. The multi-dimensional fine-grained reward device for image and video generation according to claim 5, characterized in that: The construction extraction module comprises: A labeling unit, used to send each target image or video and its corresponding multi-dimensional multiple-choice question to a target annotator, and instruct the target annotator to label each target image or video according to a pre-designed multi-dimensional multiple-choice question; A decomposition unit, configured to decompose the options of the multi-dimensional multiple-choice question into a plurality of judgment questions after the target annotator has completed the annotation, so as to form an evaluation system for each image or video; The extraction unit is used to extract the binary judgment result of each image or video in the evaluation system of each image or video.

7. The multi-dimensional fine-grained reward device for image and video generation according to claim 5, characterized in that: The training module includes: A sampling unit, used for performing balanced sampling on each image or video, the multi-dimensional multiple-choice question of each image or video, and the binary judgment result of each image or video, so as to obtain a training data set after balanced sampling; A fine-tuning unit is used to fine-tune the visual language base model using the balanced sampled training data set to generate the multi-dimensional fine-grained reward model.

8. The multi-dimensional fine-grained reward device for image and video generation according to claim 5, characterized in that: The prediction module comprises: A weighting unit, used for mapping the actual binary judgment result into a preset characteristic value, and performing linear weighting on the preset characteristic value to obtain a weighted sum result; A classification unit, configured to classify the weighted sum result into a preset category label according to a pre-constructed human preference dataset; A prediction unit is used to perform linear regression using the weighted sum result and the preset category label to predict the visual preference result of the image or video to be tested.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-dimensional fine-grained reward method for image and video generation as described in any one of claims 1 to 4.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the multi-dimensional fine-grained reward method for image and video generation as described in any one of claims 1 to 4.

Images