Scoring model training method and device, image-text pair scoring method and device, equipment and medium

Abstract

The embodiment of the invention provides a scoring model training method and device, an image-text pair scoring method and device, equipment and a medium, a scoring model comprises a text coding network, a fusion network and a scoring network, the fusion network is used for representation extraction and image-text representation fusion, and the method comprises the steps that sample image-text pairs and label data corresponding to the sample image-text pairs are obtained, the sample image pair comprises a sample image description and a corresponding sample image; obtaining a first text representation through a text coding network by using the sample image description; using the sample image and the first text representation to obtain a first multi-modal representation through a fusion network; using the first multi-modal representation to obtain a prediction score through a scoring network; and adjusting a scoring model by using the difference between the prediction score and the label data so as to realize training to obtain the scoring model capable of evaluating the quality of the image-text pair, thereby realizing the evaluation of the quality of the image-text pair.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method, device, equipment and medium for scoring a scoring model and scoring a picture-text pair. Background Art

[0002] In the field of image generation, in order to better provide users with high-quality image generation services, it is crucial to improve the image generation capability of the image generation model, wherein the image generation model is used to utilize the image description (also referred to as a text prompt) to generate an image corresponding to the image description.

[0003] Considering that in the process of training the image generation model, the better the quality of the image-text pairs in the training data, the more conducive it is to improving the image generation ability of the image generation model. In view of this, in order to better improve the image generation ability of the image generation model, we can first determine the quality of the image-text pairs, and then use the image-text pairs with good quality to train the image generation model. Therefore, how to provide a method to evaluate the quality of the image-text pairs is crucial. Summary of the invention

[0004] One or more embodiments of the present disclosure provide a method, apparatus, device and medium for training a scoring model and scoring image-text pairs, so as to train a scoring model that can evaluate the quality of image-text pairs, thereby evaluating the quality of image-text pairs.

[0005] According to a first aspect, a scoring model training method is provided, wherein the scoring model comprises: a text encoding network, a fusion network and a scoring network, wherein the fusion network is used for representation extraction and image-text representation fusion, and the method comprises:

[0006] Obtaining a sample image-text pair and its corresponding label data, wherein the sample image pair includes a sample image description and its corresponding sample image;

[0007] Using the sample image description, through the text encoding network, a first text representation is obtained;

[0008] Using the sample image and the first text representation, through the fusion network, to obtain a first multimodal representation;

[0009] Using the first multimodal representation, through the scoring network, to obtain a prediction score;

[0010] The scoring model is adjusted using the difference between the predicted score and the labeled data.

[0011] In an optional implementation, the scoring model further includes an image coding network; obtaining the first multimodal representation includes: using the sample image to obtain the first image representation through the image coding network; using the first image representation and the first text representation to obtain the first multimodal representation through the fusion network. In this implementation, the first image representation is obtained through the image coding network, so as to better fuse the first image representation and the first text representation, and obtain the first multimodal representation with a better fusion effect, which provides a basis for improving the accuracy of the prediction results of the scoring model.

[0012] In an optional implementation, the fusion network includes several groups of extraction layers and fusion layers arranged in series; the obtaining of the first multimodal representation includes: using the sample image and the first text representation, through the several groups of extraction layers and fusion layers arranged in series, to obtain the first multimodal representation. This implementation can obtain a first multimodal representation with a better fusion effect, thereby providing a basis for subsequently improving the accuracy of the prediction results of the scoring model.

[0013] In an optional embodiment, the several groups of extraction layers and fusion layers include a target extraction layer and a target fusion layer; obtaining the first multimodal representation includes: using the current representation to obtain the first extraction representation through the target extraction layer, wherein the current representation is obtained based on the output representation of the previous fusion layer of the target extraction layer, or based on the sample image; using the first extraction representation and the first text representation to obtain the output representation of the target fusion layer through the target fusion layer, so as to obtain the first multimodal representation based on the output representation of the target fusion layer. This implementation method defines the processing process of several groups of extraction layers and fusion layers to obtain a first multimodal representation with a better fusion effect, thereby providing a basis for subsequently improving the accuracy of the prediction results of the scoring model.

[0014] In an optional embodiment, the scoring network includes: at least one scoring sub-network, the at least one scoring sub-network includes: a first scoring sub-network corresponding to a first image-text pair quality dimension; the sample image includes a sample image under the first image-text pair quality dimension, and the label data includes a label value corresponding to the sample image under the first image-text pair quality dimension; the obtaining of the prediction score includes: using the first multimodal representation, through the first scoring sub-network, obtaining a first prediction sub-score, and attributing it to the prediction score; the adjusting of the scoring model includes: using the difference between the first prediction sub-score and the label value corresponding to the sample image under the first image-text pair quality dimension to adjust the first scoring sub-network. The scoring model includes at least one scoring sub-network, each of which corresponds to a different image-text pair quality dimension. The sample image and its corresponding label data are used to describe the sample image under each image-text pair quality dimension, so as to train a scoring model for fine-grained and different-dimensional quality assessment of the image-text pair, thereby realizing the quality assessment of the image-text pair from different image-text pair quality dimensions.

[0015] In an optional implementation, the fusion network is implemented based on the U-net structure of the specified diffusion model, which can save the training cost of the scoring model to a certain extent and can better adapt to the feature space of the specified diffusion model.

[0016] In an optional implementation, the scoring model is used to score the image generated by the specified diffusion model and its corresponding image description, and / or to score the image-text pairs in the training data set corresponding to the specified diffusion model. This can achieve a more accurate assessment of the quality of the image-text pairs related to the specified diffusion model, and then continue to train the specified diffusion model in combination with the image-text pairs and their corresponding scores, which can better improve the image generation capability of the specified diffusion model to a certain extent.

[0017] According to a second aspect, a method for scoring a picture-text pair is provided, comprising:

[0018] Obtaining a target image-text pair to be rated, wherein the target image-text pair includes a target image description and a target image;

[0019] Using the target image description, a target text representation is obtained through a text encoding network of a target scoring model;

[0020] Using the target image and the target text representation, a target multimodal representation is obtained through a fusion network of the target scoring model, wherein the fusion network of the target scoring model is used to perform representation extraction and image-text representation fusion;

[0021] The target multimodal representation is used to obtain a target score through a scoring network of the target scoring model.

[0022] In an optional implementation, the fusion network of the target scoring model is implemented based on a U-net structure of a specified diffusion model so as to better adapt to the feature space of the specified diffusion model.

[0023] In an optional embodiment, the target image is obtained through the specified diffusion model based on the target image description, and / or the target image-text pair is obtained from a training data set corresponding to the specified diffusion model to obtain a more accurate score for the image-text pair, and then the specified diffusion model can be better fine-tuned in combination with the score to better improve the image generation capability of the specified diffusion model.

[0024] In an optional embodiment, the target scoring model also includes an image coding network, and obtaining the target multimodal representation includes: using the target image to obtain the target image representation through the image coding network of the target scoring model; using the target image representation and the target text representation to obtain the target multimodal representation through the fusion network of the target scoring model, so as to better map the target image representation and the target text representation to the same feature space, so as to obtain a multimodal representation with better fusion effect, thereby improving the accuracy of the prediction results of the scoring model.

[0025] In an optional embodiment, the fusion network of the target scoring model includes several groups of extraction layers and fusion layers arranged in series; the target multimodal representation is obtained, including: using the target image and the target text representation, through the several groups of extraction layers and fusion layers arranged in series, to obtain the target multimodal representation with better fusion effect, thereby better improving the accuracy of the prediction results of the scoring model.

[0026] In an optional implementation, the scoring network of the target scoring model includes: at least one scoring sub-network, each scoring sub-network corresponds to a different image-text pair quality dimension; obtaining the target score includes: utilizing the target multimodal representation to obtain the target sub-score under each image-text pair quality dimension through each scoring sub-network, so as to evaluate the quality of the image-text pair from different image-text pair quality dimensions, and obtain more fine-grained, different-dimensional scores for the image-text pairs.

[0027] In an optional implementation, the target scoring model is a scoring model trained based on the scoring model training method described in the first aspect, so that the target scoring model can perform relatively accurate quality assessment on image-text pairs and obtain a more accurate score.

[0028] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method described in the first aspect or the second aspect.

[0029] According to a fourth aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in the first aspect or the second aspect is implemented.

[0030] According to the method, device, equipment and medium for scoring model training and image-text pair scoring provided by the embodiments of the present disclosure, the scoring model includes: a text encoding network, a fusion network and a scoring network, the fusion network is used for representation extraction and image-text representation fusion, and the scoring model training method includes: obtaining sample image-text pairs and their corresponding label data, the sample image pairs include sample image descriptions and their corresponding sample images; using the sample image descriptions to obtain a first text representation through a text encoding network; using the sample image and the first text representation to obtain a first multimodal representation through a fusion network; using the first multimodal representation to obtain a predicted score through a scoring network; using the difference between the predicted score and the label data to adjust the scoring model.

[0031] In the above process, the fusion network of the scoring model can realize the extraction of image representation of the sample image, and the fusion of image representation and text representation to obtain a first multimodal representation, and then use the first multimodal representation to obtain a predicted score through the scoring network of the scoring model, and use the difference between the predicted score and the label data to adjust the scoring model so that the scoring model can evaluate the quality of the image-text pairs, and through the supervision of the label data, guide the scoring model to generate quality assessment results that are more in line with human feedback. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 A schematic diagram of an implementation framework of an embodiment disclosed in the present disclosure;

[0034] Figure 2 A schematic diagram of a flow chart of a scoring model training method provided in an embodiment;

[0035] Figure 3 Another schematic diagram of a flow chart of a scoring model training method provided in an embodiment;

[0036] Figure 4 A schematic flow chart of a method for scoring image-text pairs provided in an embodiment;

[0037] Figure 5 A schematic block diagram of a scoring model training device provided in an embodiment;

[0038] Figure 6 A schematic block diagram of a device for scoring image-text pairs provided in an embodiment;

[0039] Figure 7 A schematic block diagram of an electronic device provided in an embodiment. DETAILED DESCRIPTION

[0040] The technical solution of the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0041] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0042] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0043] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0044] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0045] The present disclosure discloses a method, device, apparatus and medium for scoring a scoring model training and a picture-text pair scoring. The following first introduces the application scenarios and technical concepts of the method, which are as follows:

[0046] As mentioned above, in the process of training the image generation model, the better the quality of the image-text pairs in the training data, the more conducive it is to improving the image generation ability of the image generation model. In view of this, in order to better improve the image generation ability of the image generation model, the quality of the image-text pairs can be determined first, and then the image generation model can be trained using the image-text pairs with good quality. Therefore, it is crucial to provide a method for evaluating the quality of the image-text pairs.

[0047] In view of this, the embodiments of the present disclosure provide a method, apparatus, device and medium for training a scoring model and scoring image-text pairs, so as to achieve the evaluation of the quality of image-text pairs. Figure 1 A schematic diagram of an implementation scenario according to an embodiment disclosed in the present disclosure is shown. In the implementation scenario, a schematic structure of a scoring model and an exemplary training process of the scoring model are shown. Among them, the scoring model includes: a text encoding network, a fusion network and a scoring network, and the fusion network is used to extract representations and perform image-text representation fusion. Its training process may include: obtaining sample image-text pairs and their corresponding label data, wherein the sample image pair includes a sample image description and its corresponding sample image; using the sample image description, through a text encoding network, to obtain a first text representation; using the sample image and the first text representation, through a fusion network, to obtain a first multimodal representation; using the first multimodal representation, through a scoring network, to obtain a predicted score; using the difference between the predicted score and the label data, to adjust the scoring model.

[0048] In the above process, the sample image and the first text representation are used to obtain a first multimodal representation of the fused image and text representation through a fusion network included in the scoring model for image representation extraction and image and text representation fusion. Then, the first multimodal representation is used to obtain a predicted score through the scoring network of the scoring model. The predicted score can characterize the quality of the sample image and text pair evaluated by the scoring model. Then, the scoring model is adjusted by using the difference between the predicted score and the label data that can characterize the actual quality of the sample image and text pair. Specifically, the scoring model is adjusted with the goal of reducing the difference between the predicted score and the label data so that the scoring model can evaluate the quality of the image and text pair, and through the supervision of the label data, the scoring model is guided to generate quality assessment results that are more in line with human feedback.

[0049] In the following, in conjunction with specific embodiments, the scoring model training, image-text pair scoring method, device, equipment and medium provided by the present disclosure are described in detail.

[0050] Figure 2The flowchart of the scoring model training method in one embodiment of the present disclosure is shown. The method can be implemented by a target program. The target program can be installed in an electronic device, and the electronic device can be implemented by any device, equipment, platform, equipment cluster, etc. with computing and processing capabilities. The scoring model includes: a text encoding network, a fusion network and a scoring network. The fusion network is used for representation extraction and image-text representation fusion. During the training process, Figure 2 As shown, the following steps S210-S250 may be included:

[0051] In step S210, a sample image-text pair and its corresponding label data are obtained, where the sample image pair includes a sample image description and its corresponding sample image.

[0052] In one implementation, in order to train a scoring model, a training data set for training the scoring model may be first obtained, the training data set including a number of image-text pairs and their corresponding label data, each image-text pair including an image description and its corresponding image, in one case, the image description may also be referred to as a text prompt, i.e., a prompt. The image-text pair may be referred to as a sample image-text pair, the image therein may be referred to as a sample image, and the image description therein may be referred to as a sample image description. The label data corresponding to the sample image-text pair may represent a human evaluation of the quality of the sample image-text pair.

[0053] In one implementation, the scoring network of the scoring model may include at least one scoring sub-network, wherein different scoring sub-networks correspond to different image-text pair quality dimensions, that is, at least one scoring sub-network may evaluate the quality of image-text pairs from different image-text pair quality dimensions. Accordingly, the training data set may include sample image-text pairs and label data for each specified task, i.e., different image-text pair quality dimensions. The target program may utilize sample image-text pairs and label data corresponding to different image-text pair quality dimensions to at least train the scoring sub-network corresponding to the image-text pair quality dimension. The label data may characterize the quality of the corresponding sample image-text pair under the corresponding image-text pair quality dimension. The image pair quality dimension may be any dimension that can characterize the quality of the image-text pair.

[0054] In one case, the image-text pair quality dimension corresponding to at least one of the scoring sub-networks may include, but is not limited to, image clarity dimension, image aesthetics dimension, and image-text matching dimension. In another case, the image pair quality dimension may also include: a comprehensive quality dimension of at least two of the image clarity dimension, image aesthetics dimension, and image-text matching dimension, that is, the scoring sub-network corresponding to the comprehensive quality dimension may comprehensively evaluate the quality of the image-text pair in the corresponding two dimensions. The image pair quality dimension may be set according to user needs.

[0055] In one case, the disclosed embodiment may also provide a process for obtaining the aforementioned training data set, wherein the process may include: the target program collects image descriptions; using the collected image descriptions, through the trained image generation model, obtains at least one generated image corresponding to each image description, and then based on each image description and its corresponding at least one generated image, determines the sample image corresponding to each image description, and the label data corresponding to each image description and its corresponding sample image. The image generation model is obtained by training based on image-text pairs, and is used to generate the corresponding image for the input image description.

[0056] The following takes any image description Ti in the acquired image description as an example to illustrate the process of determining the sample image corresponding to each image description, and the label data corresponding to each image description and its corresponding sample image. i is a positive integer.

[0057] Specifically, the image description Ti and at least one corresponding generated image are annotated by using a specific program or manual annotation. In one case, the image description Ti and at least one corresponding generated image can be annotated under at least one image-text pair quality dimension to obtain a sample image corresponding to the image description Ti under at least one image-text pair quality dimension, as well as label data corresponding to the image description Ti and the corresponding sample image.

[0058] In the case where an image description Ti corresponds to a generated image, the generated image can be determined as a sample image corresponding to the image description Ti, and the image description Ti and its corresponding sample image can be annotated. For example, under at least one image-text pair quality dimension, the sample image can be annotated as a positive sample image or a negative sample image under the image-text pair quality dimension, or the annotation can represent the quality score corresponding to the sample image under each image-text pair quality dimension, so as to obtain label data corresponding to the image description Ti and the sample image under at least one image-text pair quality dimension.

[0059] For example, under the image-text pair quality dimension 1, the sample image corresponding to the image description Ti can be annotated as a positive sample image or a negative sample image. For example, a first label value is annotated for the sample image corresponding to the image description Ti, which can characterize the sample image corresponding to the image description Ti as a positive sample image under the image-text pair quality dimension 1. Correspondingly, under the image-text pair quality dimension 1, the label data corresponding to the image description Ti and its corresponding sample image include the first label value. Under the image-text pair quality dimension 2, the sample image corresponding to the image description Ti can be annotated as a positive sample image or a negative sample image. For example, a second label value is annotated for the sample image corresponding to the image description Ti, which can characterize the sample image corresponding to the image description Ti as a negative sample image under the image-text pair quality dimension 2. Correspondingly, under the image-text pair quality dimension 2, the label data corresponding to the image description Ti and its corresponding sample image include the second label value. For another example, under the image-text pair quality dimension 1, the quality score 1 corresponding to the sample image corresponding to the image description Ti can be annotated. Accordingly, under the image-text pair quality dimension 1, the label data corresponding to the image description Ti and its corresponding sample image include the quality score 1; under the image-text pair quality dimension 2, the quality score 2 corresponding to the sample image corresponding to the image description Ti can be annotated. Accordingly, under the image-text pair quality dimension 2, the label data corresponding to the image description Ti and its corresponding sample image include the quality score 2.

[0060] Taking the quality dimension of the text and picture as the image clarity dimension as an example for explanation, one implementation method can be based on the clarity of the sample image corresponding to the image description Ti, labeling the sample image corresponding to the image description Ti as a positive sample image or a negative sample image under the image clarity dimension. For example, when the clarity of the sample image is relatively high (for example, not lower than a first clarity threshold), the sample image can be labeled as a positive sample image under the image clarity dimension. For example, a first label value is labeled for the sample image, and the first label value is, for example, 1. At this time, under the image clarity dimension, the label data corresponding to the image description Ti and its corresponding sample image include the first label value 1; for another example, when the clarity of the sample image is relatively low (for example, lower than a second clarity threshold, and the second clarity threshold is not greater than the first clarity threshold), the sample image can be labeled as a negative sample image under the image clarity dimension. For example, a second label value is labeled for the sample image, and the second label value is, for example, 0. At this time, under the image clarity dimension, the label data corresponding to the image description Ti and its corresponding sample image include the second label value 2.

[0061] Another implementation method is to mark the clarity score corresponding to the sample image corresponding to the image description Ti based on the clarity of the sample image. In this case, under the image clarity dimension, the label data corresponding to the image description Ti and its corresponding sample image include the clarity score, wherein the higher the clarity of the image, the larger the corresponding clarity score.

[0062] When the image description Ti corresponds to at least two generated images, it is possible to mark, for any image-text pair quality dimension j in at least one image-text pair quality dimension, which image among the at least two generated images is a positive sample image under the image-text pair quality dimension j, and / or which image is a negative sample image under the image-text pair quality dimension j.

[0063] For example, the generated images corresponding to the image description Ti include generated image 1, generated image 2, generated image 3 and generated image 4. The at least one image-text pair quality dimension includes, but is not limited to, image clarity dimension, image aesthetics dimension, image-text matching dimension, and a comprehensive quality dimension of the three dimensions of image clarity dimension, image aesthetics dimension and image-text matching dimension, which is referred to as the first comprehensive quality dimension.

[0064] Specifically, for example, with respect to the image clarity dimension, assuming that the generated image 1 has the best clarity, for example, the clarity score of the generated image 1 is the largest among the four images, the generated image 1 can be used as the positive sample image corresponding to the image description Ti in the image clarity dimension; the generated image 2 has the worst clarity, for example, the clarity score of the generated image 2 is the smallest among the four images, the generated image 2 can be used as the negative sample image corresponding to the image description Ti in the image clarity dimension.

[0065] Correspondingly, in one case, the image description Ti and its corresponding generated image 1 are taken as a set of sample image-text pairs under the image clarity dimension, and the label data corresponding to the sample image-text pair includes data representing the generated image 1 as a positive sample image under the image clarity dimension, for example, a first label value such as 1; and the image description Ti and its corresponding generated image 2 are taken as another set of sample image-text pairs under the image clarity dimension, and the label data corresponding to the sample image-text pair includes data representing the generated image 2 as a negative sample image under the image clarity dimension, for example, a second label value such as 0.

[0066] In another case, the image description Ti and its corresponding generated image 1 and generated image 2 can also be used as a set of sample image-text pairs under the image clarity dimension. Accordingly, the label data corresponding to the sample image-text pair includes data characterizing the generated image 1 as a positive sample image under the image clarity dimension, such as a first label value such as 1, and data characterizing the generated image 2 as a negative sample image under the image clarity dimension, such as a second label value such as 0.

[0067] For another example, with respect to the image aesthetic dimension, assuming that generated image 4 is the most beautiful, for example, the aesthetic score of generated image 4 is the largest among the four images, generated image 4 can be used as the positive sample image corresponding to the image description Ti in the image aesthetic dimension, and generated image 3 is the least beautiful, for example, the aesthetic score of generated image 3 is the smallest among the four images, generated image 3 can be used as the negative sample image corresponding to the image description Ti in the image aesthetic dimension.

[0068] Accordingly, in one case, the image description Ti and its corresponding generated image 4 are used as a set of sample image-text pairs under the image aesthetic dimension, and the label data corresponding to the sample image-text pair includes data representing the generated image 4 as a positive sample image under the image aesthetic dimension, for example, a first label value of 1; and the image description Ti and its corresponding generated image 3 are used as another set of sample image-text pairs under the image aesthetic dimension, and the label data corresponding to the sample image-text pair includes data representing the generated image 3 as a negative sample image under the image aesthetic dimension, for example, a second label value of 0. In another case, the image description Ti and its corresponding generated image 4 and generated image 3 can also be used as a set of sample image-text pairs under the image aesthetic dimension, and the label data corresponding to the sample image-text pair includes data representing the generated image 4 as a positive sample image under the image aesthetic dimension, for example, a first label value of 1, and data representing the generated image 3 as a negative sample image under the image aesthetic dimension, for example, a second label value of 0.

[0069] For another example, with respect to the image-text matching dimension, assuming that the generated image 4 is the best match to the image description Ti, for example, the matching value between the generated image 4 and the image description Ti is the largest matching value between the four images and the image description Ti, the generated image 4 is used as the positive sample image corresponding to the image description Ti in the image-text matching dimension, and the generated image 2 is the least match to the image description Ti, for example, the matching value between the generated image 2 and the image description Ti is the smallest matching value between the four images and the image description Ti, the generated image 2 is used as the negative sample image corresponding to the image description Ti in the image-text matching dimension.

[0070] Accordingly, in one case, the image description Ti and its corresponding generated image 4 are used as a set of sample image-text pairs under the image-text matching dimension, and the label data corresponding to the sample image-text pair includes data representing the generated image 4 as a positive sample image under the image-text matching dimension, for example, a first label value of 1; and the image description Ti and its corresponding generated image 2 are used as another set of sample image-text pairs under the image-text matching dimension, and the label data corresponding to the sample image-text pair includes data representing the generated image 2 as a negative sample image under the image-text matching dimension, for example, a second label value of 0. In another case, the image description Ti and its corresponding generated image 4 and generated image 2 can also be used as a set of sample image-text pairs under the image-text matching dimension, and the label data corresponding to the sample image-text pair includes data representing the generated image 4 as a positive sample image under the image-text matching dimension, for example, a first label value of 1, and data representing the generated image 2 as a negative sample image under the image-text matching dimension, for example, a second label value of 0.

[0071] For another example, with respect to the comprehensive quality dimensions of the three dimensions of image clarity, image aesthetics, and image-text matching, that is, the first comprehensive quality dimension, it is assumed that the comprehensive quality between the generated image 4 and the image description Ti is the best, that is, the generated image 4 and the image description Ti have the best overall quality in the three dimensions of image clarity, image aesthetics, and image-text matching. The generated image 4 is used as the positive sample image corresponding to the image description Ti in the first comprehensive quality dimension, and the comprehensive quality between the generated image 2 and the image description Ti is the worst, that is, the generated image 2 and the image description Ti have the worst overall quality in the three dimensions of image clarity, image aesthetics, and image-text matching. The generated image 2 is used as the negative sample image corresponding to the image description Ti in the first comprehensive quality dimension.

[0072] Accordingly, in one case, the image description Ti and its corresponding generated image 4 are used as a set of sample image-text pairs under the first comprehensive quality dimension, and the label data corresponding to the sample image-text pair includes data representing that the generated image 4 is a positive sample image under the first comprehensive quality dimension, for example, a first label value of 1; and the image description Ti and its corresponding generated image 2 are used as another set of sample image-text pairs under the first comprehensive quality dimension, and the label data corresponding to the sample image-text pair includes data representing that the generated image 2 is a negative sample image under the first comprehensive quality dimension, for example, a second label value of 0. In another case, the image description Ti and its corresponding generated image 4 and generated image 2 can also be used as a set of sample image-text pairs under the first comprehensive quality dimension, and the label data corresponding to the sample image-text pair includes data representing that the generated image 4 is a positive sample image under the first comprehensive quality dimension, for example, a first label value of 1, and data representing that the generated image 2 is a negative sample image under the first comprehensive quality dimension, for example, a second label value of 0.

[0073] By the above method, sample images corresponding to each image description and label data corresponding to each image description and its corresponding sample image can be obtained under at least one image-text pair quality dimension, thereby obtaining a training data set. It can be understood that between different image-text pair quality dimensions, the positive sample images corresponding to the image description Ti can be the same or different, and the negative sample images corresponding to the image description Ti can be the same or different.

[0074] The target program can obtain a sample image-text pair from the training data set, and then, in step S220, use the sample image description to obtain a first text representation through a text encoding network. In this step, the text encoding network can be a pre-trained text encoding network. The target program inputs the sample image description into the text encoding network, and performs text encoding on the sample image description through the text encoding network to obtain a first text representation corresponding to the sample image description. In one implementation, the text encoding network can be implemented based on the text encoder in the CLIP (Contrastive Language-Image Pre-Training) model.

[0075] Next, in step S230, the sample image and the first text representation are used to obtain a first multimodal representation through a fusion network. In one implementation, the target program can input the sample image and the first text representation into the fusion network, and the fusion network can extract the image representation of the sample image to obtain an extraction result, and then the first text representation is fused into the extraction result to obtain the first multimodal representation.

[0076] In one embodiment, the fusion network may include several groups of extraction layers and fusion layers arranged in series, wherein the extraction layer is used for representation extraction and the fusion layer is used for image-text representation fusion. Accordingly, in step S230, it may include: using the sample image and the first text representation, through several groups of extraction layers and fusion layers arranged in series, to obtain a first multimodal representation.

[0077] The plurality of groups of extraction layers and fusion layers may be N groups of extraction layers and fusion layers, where N is a positive integer. In one case, in order to obtain a multimodal representation with a better fusion effect, N may be a positive integer greater than 1.

[0078] Specifically, in one implementation, the plurality of groups of extraction layers and fusion layers may include a target extraction layer and a target fusion layer; accordingly, step S230 may include the following steps 11-12:

[0079] In step 11, a first extraction representation is obtained by using the current representation through the target extraction layer, wherein the current representation is obtained based on an output representation of a previous fusion layer of the target extraction layer, or based on a sample image.

[0080] In this step, if the target extraction layer is an extraction layer in the first group of extraction layers and fusion layers, for example, the first extraction layer, the current representation can be obtained based on the sample image, for example, by performing image representation extraction on the sample image, for example, the current representation is the first image representation mentioned later.

[0081] If the target extraction layer is an extraction layer not in the first group of extraction layers and fusion layers, for example, an extraction layer in the Mth group of extraction layers and fusion layers, for example, the Mth extraction layer, where M is a positive integer greater than 1, the current representation can be obtained based on the output representation of the previous fusion layer of the target extraction layer, that is, the fusion layer in the M-1th group of extraction layers and fusion layers.

[0082] In one implementation, a downsampling layer may also be included between each group of extraction layers and fusion layers. Accordingly, if the target extraction layer is the Mth extraction layer, the current representation may be a representation obtained by downsampling the output representation of the fusion layer in the M-1th group of extraction layers and fusion layers through the M-1th group of extraction layers and fusion layers and the downsampling layer between the Mth group of extraction layers and fusion layers.

[0083] In another implementation, a downsampling layer may be included between every two groups of extraction layers and fusion layers. Accordingly, if the target extraction layer is the Mth extraction layer and M is an odd number, the current representation may be a representation obtained by downsampling the output representation of the fusion layer in the M-1th group of extraction layers and fusion layers through the M-1th group of extraction layers and fusion layers and the downsampling layer between the Mth group of extraction layers and fusion layers; when M is an even number, the current representation may be the output representation of the fusion layer in the M-1th group of extraction layers and fusion layers.

[0084] In this step, the target program obtains the current representation, and can input the current representation into the target extraction layer, that is, the Mth extraction layer, and extract the representation of the current representation through the target extraction layer to obtain a first extracted representation.

[0085] After obtaining the first extraction representation, in step 12, the first extraction representation and the first text representation are used to obtain the output representation of the target fusion layer through the target fusion layer, and the first multimodal representation is obtained based on the output representation of the target fusion layer. In this step, the target program can input the first extraction representation and the first text representation into the target fusion layer, i.e., the Mth fusion layer, and fuse the first extraction representation and the first text representation through the target fusion layer, that is, fuse the first text representation into the first extraction representation, and obtain the output representation of the target fusion layer, and obtain the first multimodal representation based on the output representation of the target fusion layer.

[0086] Specifically, in one case, if the target fusion layer, that is, the Mth fusion layer, is the last fusion layer, the target program may directly use the output representation of the target fusion layer as the first multimodal representation.

[0087] In another case, if the target fusion layer, i.e., the Mth fusion layer, is not the last fusion layer, and when a downsampling layer is included between each group of extraction layers and the fusion layer, the target program can downsample the output representation of the target fusion layer to obtain a new current representation, and then use the new current representation to pass through the next extraction layer of the target fusion layer, i.e., the M+1th extraction layer, that is, input the new current representation into the M+1th extraction layer to obtain the extraction representation of the M+1th extraction layer, and use the extraction representation of the M+1th extraction layer and the first text representation to pass through the fusion layer corresponding to the M+1th extraction layer, i.e., the M+1th fusion layer, to obtain the output representation of the M+1th fusion layer; the target program then downsamples the output representation of the M+1th fusion layer to obtain a new current representation, and executes subsequent processes... and so on, until the output representation of the fusion layer at the end is obtained, which is the first multimodal representation.

[0088] In another case, if the target fusion layer, i.e., the Mth fusion layer, is not the last fusion layer, and when a downsampling layer is included between every two groups of extraction layers and fusion layers, when M is an odd number, the target program may use the output representation of the target fusion layer as the new current representation, and use the new current representation to obtain the extraction representation of the M+1th extraction layer through the M+1th extraction layer; and then use the extraction representation of the M+1th extraction layer and the first text representation to obtain the output representation of the M+1th fusion layer through the fusion layer in the M+1th group of extraction layers and fusion layers, i.e., the M+1th fusion layer; then the target program downsamples the output representation of the M+1th fusion layer to obtain a new current representation, and executes subsequent processes... and so on, until the output representation of the last fusion layer is obtained, which is the first multimodal representation.

[0089] When M is an even number, the target program can downsample the output representation of the target fusion layer to obtain a new current representation, and then use the new current representation to pass through the next extraction layer of the target fusion layer, i.e., the M+1th extraction layer, that is, input the new current representation into the M+1th extraction layer to obtain the extracted representation of the M+1th extraction layer, and then use the extracted representation of the M+1th extraction layer and the first text representation to pass through the fusion layer corresponding to the M+1th extraction layer, i.e., the M+1th fusion layer, to obtain the output representation of the M+1th fusion layer, use the output representation of the M+1th fusion layer as the new current representation, and execute subsequent processes... and so on, until the output representation of the fusion layer at the end is obtained, which is the first multimodal representation.

[0090] In the above implementation, a first multimodal representation with better fusion effect can be obtained, which can provide a basis for subsequently improving the accuracy of the prediction results of the scoring model.

[0091] In one implementation, the target fusion layer may be based on a cross-attention mechanism, that is, a cross-attention operation is used to fuse the first extracted representation and the first text representation, so as to achieve better fusion of the first extracted representation and the first text representation.

[0092] In one embodiment, the fusion network is implemented based on the U-net structure of the specified diffusion model. Specifically, the fusion network can be implemented as the encoder part in the U-net structure based on the specified diffusion model. In this implementation, the specified diffusion model is a model trained based on image-text pairs, and its image representation space and text representation space are aligned. The fusion network is implemented directly based on the U-net structure of the specified diffusion model, which can save the training cost of the scoring model to a certain extent and make the scoring model better adapt to the feature space of the specified diffusion model.

[0093] In one embodiment, the scoring model can be used to score images generated by a specified diffusion model and their corresponding image descriptions, and / or to score image-text pairs in a training data set corresponding to the specified diffusion model, so as to achieve a more accurate assessment of the quality of the image-text pairs related to the specified diffusion model, and then continue to train the specified diffusion model in combination with the image-text pairs and their corresponding scores, which can better improve the image generation capability of the specified diffusion model to a certain extent. The specified diffusion model can generate an image corresponding to the image description based on the image description.

[0094] After obtaining the first multimodal representation, in step S240, the first multimodal representation is used to obtain a prediction score through a scoring network. In this step, the target program inputs the first multimodal representation into the scoring network, processes the first multimodal representation through the scoring network, and obtains a prediction score corresponding to the sample image-text pair. The prediction score can represent the quality evaluated by the scoring model for the sample image-text pair. For example, it can represent the quality evaluated by the scoring model for the sample image-text pair under the aforementioned image-text pair quality dimension. In one case, the larger the prediction score, the better the quality of the sample image-text pair evaluated by the scoring model. The value range of the prediction score is [0, 1]. In one implementation, the scoring network can be implemented by MLP (multi-layer perceptron).

[0095] After obtaining the prediction score, in step S250, the scoring model is adjusted using the difference between the prediction score and the label data. In this step, the target program can use a specified loss function to adjust the scoring model using the difference between the prediction score and the label data. The specified loss function may include, but is not limited to, a cross entropy loss function and an MSE (mean square error) loss function, etc. In one implementation, the target program can use a specified loss function to construct a prediction loss corresponding to the scoring model using the difference between the prediction score and the label data, wherein the prediction loss is positively correlated with the difference between the prediction score and the label data. Then, the scoring model is adjusted with the goal of minimizing the prediction loss, that is, minimizing the difference between the prediction score and the label data.

[0096] In one implementation, the text encoding network and the fusion network (and the image encoding network mentioned later) and the image encoding network of the scoring model are pre-trained networks, and accordingly, adjusting the scoring model may include: adjusting the parameters of the scoring network. In another implementation, in the case of setting the text encoding network and the fusion network (and the image encoding network mentioned later) of the scoring model to be adjusted, adjusting the scoring model may include: adjusting the parameters of the text encoding network, the fusion network (and the image encoding network mentioned later) and the scoring network.

[0097] In one embodiment, in order to achieve a more fine-grained quality assessment of the image-text pair, the scoring network may include: at least one scoring sub-network, wherein each scoring sub-network corresponds to a different image-text pair quality dimension, for example, the scoring network may include but is not limited to the scoring sub-network corresponding to the aforementioned image clarity dimension, the scoring sub-network corresponding to the image aesthetics dimension, and the scoring sub-network corresponding to the image-text matching degree dimension. Among them, the scoring sub-network corresponding to the image clarity dimension is used to perform quality assessment on the image-text pair under the image clarity dimension, the scoring sub-network corresponding to the image aesthetics dimension is used to perform quality assessment on the image-text pair under the image aesthetics dimension, and the scoring sub-network corresponding to the image-text matching degree dimension is used to perform quality assessment on the image-text pair under the image-text matching degree dimension.

[0098] In addition, the scoring network may also include other dimensions for evaluating the quality of image-text pairs, for example, it may also include scoring sub-networks corresponding to comprehensive quality dimensions of at least two of the image clarity dimension, the image aesthetics dimension, and the image-text matching dimension. This type of scoring sub-network can perform comprehensive quality evaluation of image-text pairs in at least two of the image clarity dimension, the image aesthetics dimension, and the image-text matching dimension.

[0099] In one case, the number and corresponding dimensions of the scoring sub-networks included in the scoring network can be set according to user requirements. The label data corresponding to the sample image-text pair is related to the dimensions corresponding to the scoring sub-networks included in the scoring network. In one implementation, each scoring sub-network can be implemented by an MLP (multi-layer perceptron).

[0100] In one implementation, each of the scoring sub-networks corresponds to a different image-text pair quality dimension, and different image-text pair quality dimensions may correspond to corresponding sample image-text pairs. The target program may use the sample image-text pairs corresponding to the different image-text pair quality dimensions and their corresponding label data to adjust and train the corresponding scoring sub-network. It is understandable that the data processing process and the corresponding adjustment process of different scoring sub-networks are similar. The data processing process and the corresponding adjustment process of any first scoring sub-network in at least one scoring sub-network are used as an example for explanation. Specifically, the at least one scoring sub-network includes: a first scoring sub-network corresponding to a first image-text pair quality dimension, and the first image-text pair quality dimension can be any dimension of the at least one image-text pair quality dimension mentioned above; the sample image in the sample image-text pair includes the sample image under the first image-text pair quality dimension, and the label data corresponding to the sample image-text pair includes the label value corresponding to the sample image under the first image-text pair quality dimension; accordingly, in step S240, it can include: using the first multimodal representation, through the first scoring sub-network, to obtain a first prediction sub-score, and classifying it into the prediction score; in this step, the target program inputs the first multimodal representation into the first scoring sub-network, obtains the first prediction sub-score of the sample image-text pair under the first image-text pair quality dimension, classifies it into the prediction score, and thus obtains the prediction score.

[0101] Then in step S250, it includes: using the difference between the first prediction sub-score and the label value corresponding to the sample image under the first image-text pair quality dimension, adjusting the first scoring sub-network. In this step, a specified loss function can be used to determine the first prediction sub-loss by using the difference between the first prediction sub-score and the label value corresponding to the sample image under the first image-text pair quality dimension, and include it in the aforementioned prediction loss; the first scoring sub-network is adjusted with the goal of minimizing the prediction loss, that is, minimizing the first prediction sub-loss, that is, minimizing the difference between the first prediction sub-score and the label value corresponding to the sample image under the first image-text pair quality dimension.

[0102] It can be understood that the training sample set includes sample image-text pairs corresponding to the image-text pair quality dimensions corresponding to each scoring sub-network, and the sample image-text pairs corresponding to the image-text pair quality dimensions corresponding to each scoring sub-network can be used to train and adjust each scoring sub-network in the scoring model to improve the accuracy of the scores output by each scoring sub-network, so that the scores output by each scoring sub-network are more consistent with the quality evaluation results of human feedback. Each scoring sub-network is used to evaluate the quality of the image-text pair under each image-text pair quality dimension.

[0103] Let's review the execution process of the aforementioned steps S210-S250. The above embodiment is explained by taking a sample image-text pair as an example. In another embodiment, the above steps S210-S240 can also be performed for a batch of samples, i.e., multiple sample image-text pairs, to obtain the prediction score corresponding to each sample image-text pair, and then adjust the scoring model in step S250 based on the difference between the prediction score corresponding to each sample image-text pair and the label data corresponding to the sample image-text pair. In this embodiment, the scoring model is adjusted for a batch of samples, which can reduce the number of adjustments to the scoring model and make it easier to implement the training process.

[0104] The scoring model is adjusted until the scoring model reaches a preset convergence condition to obtain a trained scoring model. The preset convergence condition may include but is not limited to the number of parameter adjustments of the scoring model exceeding a preset number threshold, or the predicted loss of the determined scoring model being lower than a preset loss threshold. In one implementation, the scoring model reaching a preset convergence condition may include: each scoring sub-network reaches a preset convergence condition, for example: the number of adjustments of each scoring sub-network exceeds a preset number threshold, or the loss determined for each scoring sub-network is lower than a preset loss threshold.

[0105] In the above process, the sample image and the first text representation are used to obtain a first multimodal representation of the fused image and text representation through a fusion network included in the scoring model for image representation extraction and image and text representation fusion. Then, the first multimodal representation is used to obtain a predicted score through the scoring network of the scoring model. The predicted score can characterize the quality of the sample image and text pair evaluated by the scoring model. Then, the scoring model is adjusted by using the difference between the predicted score and the label data that can characterize the actual quality of the sample image and text pair. Specifically, the scoring model is adjusted with the goal of reducing the difference between the predicted score and the label data so that the scoring model can evaluate the quality of the image and text pair, and through the supervision of the label data, the scoring model is guided to generate quality assessment results that are more in line with human feedback.

[0106] In one embodiment, the sample image may include a positive sample image and / or a negative sample image, and the label data includes a first label value corresponding to the positive sample image and / or a second label value corresponding to the negative sample image; accordingly, in step S230, the process of obtaining the first multimodal representation includes: using the first text representation and the positive sample image to obtain a positive sample multimodal representation through a fusion network; and / or using the first text representation and the negative sample image to obtain a negative sample multimodal representation through a fusion network; wherein the specific implementation process can refer to the aforementioned process of obtaining the first multimodal representation.

[0107] Then in step S240, the process of obtaining the prediction score includes: using the multimodal representation of the positive sample to obtain a first prediction score through a scoring network; and / or using the multimodal representation of the negative sample to obtain a second prediction score through a scoring network; wherein the specific implementation process can refer to the aforementioned process of obtaining the prediction score.

[0108] Finally, in step S250, the process of adjusting the scoring model includes: adjusting the scoring model using the difference between the first prediction score and the first label value, and / or the difference between the second prediction score and the second label value. In this step, the scoring model is adjusted, for example, the parameters of the scoring network are adjusted, with the goal of reducing the difference between the first prediction score and the first label value, and / or reducing the difference between the second prediction score and the second label value.

[0109] This allows the scoring model to distinguish the quality of sample images corresponding to the image description, so as to better ensure the accuracy of the prediction results of the scoring model, that is, the scores predicted for the image-text pairs.

[0110] In one embodiment, the scoring model may also include an image encoding network, such as Figure 3 As shown, the method may include the following steps S310-S360:

[0111] In step S310, a sample image-text pair and its corresponding label data are obtained, wherein the sample image pair includes a sample image description and its corresponding sample image. The implementation process of step S310 is similar to Figure 2 The implementation process of step S210 is similar and will not be described in detail here.

[0112] In step S320, the sample image description is used to obtain a first text representation through a text encoding network. The implementation process of step S320 is similar to Figure 2 The implementation process of step S220 is similar and will not be described in detail here.

[0113] In step S330, the sample image is used to obtain a first image representation through an image coding network. In this step, the target program can input the sample image into the image coding network, and perform image coding on the sample image through the image coding network to obtain the first image representation. The image coding network is a trained image coding network. In one implementation, the image coding network can be a coding network implemented based on VAE (Variational Autoencoder).

[0114] Next, in step S340, the first image representation and the first text representation are used to obtain a first multimodal representation through a fusion network. In this step, the target program can input the first image representation and the first text representation into the fusion network, extract the image representation of the first image representation through the fusion network, obtain the corresponding extraction result, and fuse the extraction result and the first text representation, that is, fuse the first text representation into the extraction result, to obtain the first multimodal representation. The specific implementation process of step S340 can be the specific implementation process of obtaining the first multimodal representation in the aforementioned embodiment, which will not be repeated here.

[0115] Through the image coding network, a first image representation is obtained, and then the first image representation and the first text representation are fused to obtain a first multimodal representation with a better fusion effect, so as to provide a basis for improving the accuracy of the prediction results of the scoring model.

[0116] Then, in step S350, the first multimodal representation is used to obtain a prediction score through a scoring network. And in step S360, the difference between the prediction score and the label data is used to adjust the scoring model. The implementation process of step S350 is the same as Figure 2 The implementation process of step S240 is similar to that of step S360. Figure 2 The implementation process of step S250 is similar and will not be described in detail here.

[0117] Corresponding to the above method embodiment, the embodiment of the present disclosure also provides a method for scoring image-text pairs, which can be implemented by a first program. The first program can be the same physical program as the aforementioned target program, or can be a different physical program from the aforementioned target program. The first program can be installed in an electronic device, and the electronic device can be implemented by any device, equipment, platform, device cluster, etc. with computing and processing capabilities. Figure 4 As shown, the following steps S410-S440 may be included:

[0118] It can be understood that the method for scoring the image-text pair can be a method for scoring the image-text pair based on a target scoring model, and the target scoring model can include a text encoder, a fusion network, and a scoring network, and can also include the image encoder mentioned later. In one embodiment, the target scoring model can be a scoring model trained based on the scoring model training method provided in the aforementioned embodiment, that is, it can be a scoring model trained in the aforementioned embodiment.

[0119] In step S410, a target image-text pair to be rated is obtained, wherein the target image-text pair includes a target image description and a target image. In one implementation, the target image-text pair can be any image-text pair that needs to be evaluated for quality.

[0120] After obtaining the target image-text pair to be scored, in step S420, the target image description is used to obtain the target text representation through the text encoding network of the target scoring model. The process of obtaining the target text representation can refer to the process of obtaining the first text representation in the above embodiment, which will not be repeated here.

[0121] Next, in step S430, the target image and target text representations are used to obtain a target multimodal representation through a fusion network of a target scoring model, wherein the fusion network of the target scoring model is used to extract representations and perform image-text representation fusion. The process of obtaining the target multimodal representation can refer to the process of obtaining the first multimodal representation in the aforementioned embodiment, and will not be described in detail here.

[0122] In one embodiment, the fusion network of the target scoring model is implemented based on the U-net structure of the specified diffusion model. Specifically, it can be implemented based on the encoder part in the U-net structure of the specified diffusion model. In this implementation, the specified diffusion model is a model trained based on image-text pairs, which can make the target scoring model better adapt to the feature space of the specified diffusion model, and can achieve a more accurate assessment of the quality of the relevant image-text pairs of the specified diffusion model. Accordingly, in one embodiment, the target image is obtained based on the target image description through the specified diffusion model, and / or the target image-text pair is obtained from the training data set corresponding to the specified diffusion model.

[0123] In one embodiment, the target scoring model may further include an image coding network. Accordingly, in step S430, the following steps 21-22 may be included:

[0124] In step 21, the target image is used to obtain the target image representation through the image encoding network of the target scoring model. The process of obtaining the target image representation can refer to the process of obtaining the first image representation in the above embodiment, which will not be described in detail here.

[0125] Next, in step 22, the target image representation and the target text representation are used to obtain the target multimodal representation through the fusion network of the target scoring model. The process of obtaining the target multimodal representation can refer to the process of obtaining the first multimodal representation in the above embodiment, which will not be described in detail here.

[0126] In this implementation, a target multimodal representation with better fusion effect can be obtained, which can provide a basis for obtaining a quality score of the target image-text pair with better accuracy.

[0127] In one embodiment, the fusion network of the target scoring model may include several sets of extraction layers and fusion layers arranged in series; accordingly, in step S430, it may include: using the target image and target text representation, through several sets of extraction layers and fusion layers arranged in series, to obtain the target multimodal representation. The process can refer to the process of obtaining the first multimodal representation in the above embodiment, which will not be described in detail here.

[0128] After obtaining the target multimodal representation, in step S440, the target multimodal representation is used to obtain the target score through the scoring network of the target scoring model. In this step, the first program inputs the target multimodal representation into the scoring network of the target scoring model, and processes the target multimodal representation through the scoring network of the target scoring model to obtain a target score that can characterize the quality of the target image-text pair.

[0129] In one embodiment, the scoring network of the target scoring model includes: at least one scoring sub-network, each scoring sub-network corresponds to a different image-text pair quality dimension, and each scoring sub-network can evaluate the quality of the target image-text pair under the image-text pair quality dimension corresponding to the scoring sub-network; in step S440, it can include: using the target multimodal representation to obtain the target sub-score under each image-text pair quality dimension through each scoring sub-network. The target sub-score under each image-text pair quality dimension can represent the quality of the target image-text pair evaluated by the target scoring model under the corresponding image-text pair quality dimension. The value range of each target sub-score can be set to [0, 1], where the larger the target sub-score, the better the quality of the target image-text pair under the image-text pair quality dimension corresponding to the target sub-score.

[0130] In the above process, the target scoring model can be used to obtain a target score that can better characterize the quality of the target image-text pair and has a higher accuracy. In addition, when the scoring network of the target scoring model includes multiple scoring sub-networks, the target scoring model can be used to obtain a more fine-grained target score that characterizes the quality of the target image-text pair under different image-text pair quality dimensions, so as to more intuitively characterize the quality of the target image-text pair.

[0131] The foregoing describes specific embodiments of the present disclosure, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims may be performed in an order different from that in the embodiments, and the desired results may still be achieved. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible, or may be advantageous.

[0132] Corresponding to the above method embodiment, the present disclosure embodiment provides a scoring model training device 500, whose schematic block diagram is as follows: Figure 5 As shown, it is used to execute Figure 2 In the method shown, the scoring model includes: a text encoding network, a fusion network and a scoring network, the fusion network is used for representation extraction and image-text representation fusion, and the device 500 includes:

[0133] A first acquisition module 510 is configured to acquire a sample image-text pair and its corresponding label data, wherein the sample image pair includes a sample image description and its corresponding sample image;

[0134] A first obtaining module 520 is configured to obtain a first text representation by using the sample image description through the text encoding network;

[0135] The second obtaining module 530 is configured to obtain a first multimodal representation by using the sample image and the first text representation through the fusion network:

[0136] A third obtaining module 540 is configured to obtain a prediction score by using the first multimodal representation through the scoring network;

[0137] The adjustment module 550 is configured to adjust the scoring model using the difference between the prediction score and the label data.

[0138] In an optional implementation, the scoring model further includes an image coding network;

[0139] The second obtaining module 530 is specifically configured to use the sample image to obtain a first image representation through the image encoding network; and use the first image representation and the first text representation to obtain the first multimodal representation through the fusion network.

[0140] In an optional embodiment, the fusion network includes several groups of extraction layers and fusion layers arranged in series;

[0141] The second obtaining module 530 is specifically configured to obtain a first multimodal representation by using the sample image and the first text representation through the plurality of groups of extraction layers and fusion layers arranged in series.

[0142] In an optional embodiment, the plurality of groups of extraction layers and fusion layers include target extraction layers and target fusion layers;

[0143] The second obtaining module 530 is specifically configured to obtain a first extraction representation by using the current representation through the target extraction layer, wherein the current representation is obtained based on an output representation of a previous fusion layer of the target extraction layer, or based on the sample image;

[0144] The first extraction representation and the first text representation are used to obtain an output representation of the target fusion layer through the target fusion layer, so as to obtain the first multimodal representation based on the output representation of the target fusion layer.

[0145] In an optional implementation, the scoring network includes: at least one scoring sub-network, the at least one scoring sub-network includes: a first scoring sub-network corresponding to a first image-text pair quality dimension; the sample image includes a sample image under the first image-text pair quality dimension, and the label data includes a label value corresponding to the sample image under the first image-text pair quality dimension;

[0146] The third obtaining module 540 is specifically configured to obtain a first prediction sub-score by using the first multimodal representation through the first scoring sub-network, and include it in the prediction score;

[0147] The adjustment module 550 is specifically configured to adjust the first scoring sub-network by using the difference between the first prediction sub-score and the label value corresponding to the sample image under the first image-text pair quality dimension.

[0148] In an optional implementation, the fusion network is implemented based on a U-net structure of a specified diffusion model.

[0149] In an optional implementation, the scoring model is used to score images generated by the specified diffusion model and their corresponding image descriptions, and / or to score image-text pairs in a training data set corresponding to the specified diffusion model.

[0150] Corresponding to the above method embodiment, the present disclosure embodiment provides a device 600 for scoring image-text pairs, and its schematic block diagram is as follows: Figure 6 As shown, it is used to execute Figure 4 The method shown, the device 600 includes:

[0151] The second acquisition module 610 is configured to acquire a target image-text pair to be rated, wherein the target image-text pair includes a target image description and a target image;

[0152] A fourth obtaining module 620 is configured to obtain a target text representation by using the target image description through a text encoding network of a target scoring model;

[0153] A fifth obtaining module 630 is configured to obtain a target multimodal representation by using the target image and the target text representation through a fusion network of the target scoring model, wherein the fusion network of the target scoring model is used for representation extraction and image-text representation fusion;

[0154] The sixth obtaining module 640 is configured to obtain a target score by using the target multimodal representation through the scoring network of the target scoring model.

[0155] In an optional implementation, the fusion network of the target scoring model is implemented based on a U-net structure of a specified diffusion model.

[0156] In an optional implementation, the target image is obtained through the specified diffusion model based on the target image description, and / or the target image-text pair is obtained from a training data set corresponding to the specified diffusion model.

[0157] In an optional implementation, the target scoring model further includes an image coding network, and the fifth obtaining module 630 is specifically configured to obtain a target image representation by using the target image through the image coding network of the target scoring model;

[0158] The target multimodal representation is obtained by utilizing the target image representation and the target text representation through a fusion network of the target scoring model.

[0159] In an optional embodiment, the fusion network of the target scoring model includes several groups of extraction layers and fusion layers arranged in series; the fifth obtaining module 630 is specifically configured to use the target image and the target text representation to obtain the target multimodal representation through the several groups of extraction layers and fusion layers arranged in series.

[0160] In an optional implementation, the scoring network of the target scoring model includes: at least one scoring sub-network, each scoring sub-network corresponding to a different image-text pair quality dimension;

[0161] The sixth obtaining module 640 is specifically configured to utilize the target multimodal representation to obtain the target sub-score of each image-text pair quality dimension through each scoring sub-network.

[0162] In an optional implementation, the target scoring model is a scoring model trained by the scoring model training device described in the aforementioned embodiment.

[0163] The above device embodiments correspond to the method embodiments. For specific descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For specific descriptions, please refer to the corresponding method embodiments.

[0164] The present disclosure provides an electronic device, including a memory and a processor, wherein the memory stores an executable code, and when the processor executes the executable code, the following is implemented: Figure 2 , Figure 3 or Figure 4 The method shown below. Figure 7 , which shows a schematic structural diagram of an electronic device suitable for implementing an embodiment of the present disclosure. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0165] like Figure 7 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. In RAM 703, various programs and data required for the operation of the electronic device are also stored. The processing device 701, ROM 702, and RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0166] Typically, the following devices may be connected to the I / O interface 705: input devices 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, etc.; output devices 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 708 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 709. The communication devices 709 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Figure 7 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 7 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0167] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 709, or installed from a storage device 708, or installed from a ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0168] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute the following steps provided in the present disclosure: Figure 2 , Figure 3 or Figure 4 The method shown.

[0169] It should be noted that the computer-readable medium described in the embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0170] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains a sample image-text pair and its corresponding label data, wherein the sample image pair includes a sample image description and its corresponding sample image; uses the sample image description to obtain a first text representation through the text encoding network of the scoring model; uses the sample image and the first text representation to obtain a first multimodal representation through the fusion network of the scoring model; uses the first multimodal representation to obtain a predicted score through the scoring network of the scoring model; and uses the difference between the predicted score and the label data to adjust the scoring model. Alternatively, the electronic device is configured to: obtain a target image-text pair to be scored, wherein the target image-text pair includes a target image description and a target image; utilize the target image description to obtain a target text representation through a text encoding network of a target scoring model; utilize the target image and the target text representation to obtain a target multimodal representation through a fusion network of the target scoring model, wherein the fusion network of the target scoring model is used to perform representation extraction and image-text representation fusion; utilize the target multimodal representation to obtain a target score through a scoring network of the target scoring model.

[0171] Computer program code for performing the operations of the disclosed embodiments may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0172] Each embodiment in the present disclosure is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the storage medium and computing device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0173] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present invention may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0174] The specific implementation methods described above further describe the purpose, technical solutions and beneficial effects of the embodiments of the present invention in detail. It should be understood that the above description is only a specific implementation method of the embodiments of the present invention and is not intended to limit the scope of protection of the present invention. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solution of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A scoring model training method, the scoring model comprising: A text encoding network, a fusion network and a scoring network, wherein the fusion network is used for representation extraction and image-text representation fusion, and the method comprises: Obtaining a sample image-text pair and its corresponding label data, wherein the sample image pair includes a sample image description and its corresponding sample image; Using the sample image description, through the text encoding network, a first text representation is obtained; Using the sample image and the first text representation, through the fusion network, to obtain a first multimodal representation; Using the first multimodal representation, through the scoring network, to obtain a prediction score; The scoring model is adjusted using the difference between the predicted score and the labeled data.

2. The method of claim 1, wherein the scoring model further comprises an image coding network; The obtaining of the first multimodal representation comprises: Using the sample image through the image encoding network, a first image representation is obtained; The first multimodal representation is obtained by using the first image representation and the first text representation through the fusion network.

3. The method of claim 1, wherein: The fusion network includes several groups of extraction layers and fusion layers arranged in series; The obtaining of the first multimodal representation comprises: The sample image and the first text representation are used to obtain a first multimodal representation by serially arranging the plurality of extraction layers and fusion layers.

4. The method of claim 3, wherein: The several groups of extraction layers and fusion layers include, Target extraction layer and target fusion layer; The obtaining of the first multimodal representation comprises: Using the current representation, through the target extraction layer, a first extraction representation is obtained, wherein the current representation is obtained based on an output representation of a previous fusion layer of the target extraction layer, or based on the sample image; The first extraction representation and the first text representation are used to obtain an output representation of the target fusion layer through the target fusion layer, so as to obtain the first multimodal representation based on the output representation of the target fusion layer.

5. The method according to any one of claims 1 to 4, wherein: The scoring network includes: at least one scoring sub-network, the at least one scoring sub-network includes: a first scoring sub-network corresponding to a first image-text pair quality dimension; the sample image includes a sample image under the first image-text pair quality dimension, and the label data includes a label value corresponding to the sample image under the first image-text pair quality dimension; The obtaining of the prediction score comprises: Using the first multimodal representation, through the first scoring subnetwork, a first prediction subscore is obtained, and the first prediction subscore is included in the prediction score; The adjusting the scoring model comprises: The first scoring subnetwork is adjusted using a difference between the first prediction sub-score and a label value corresponding to the sample image under the first image-text pair quality dimension.

6. The method according to any one of claims 1 to 4, wherein: The fusion network is implemented based on the U-net structure of the specified diffusion model.

7. The method of claim 6, wherein: The scoring model is used to score the image generated by the specified diffusion model and its corresponding image description, and / or to score the image-text pairs in the training data set corresponding to the specified diffusion model.

8. A method for scoring a picture-text pair, comprising: Obtaining a target image-text pair to be rated, wherein the target image-text pair includes a target image description and a target image; Using the target image description, a target text representation is obtained through a text encoding network of a target scoring model; Using the target image and the target text representation, a target multimodal representation is obtained through a fusion network of the target scoring model, wherein the fusion network of the target scoring model is used to perform representation extraction and image-text representation fusion; The target multimodal representation is used to obtain a target score through a scoring network of the target scoring model.

9. The method of claim 8, wherein: The fusion network of the target scoring model is implemented based on the U-net structure of the specified diffusion model.

10. The method of claim 9, wherein: The target image is obtained through the specified diffusion model based on the target image description, and / or the target image-text pair is obtained from a training data set corresponding to the specified diffusion model.

11. The method of claim 8, wherein: The target scoring model also includes an image encoding network, and obtaining a target multimodal representation includes: Using the target image, through the image encoding network of the target scoring model, a target image representation is obtained; The target multimodal representation is obtained by utilizing the target image representation and the target text representation through a fusion network of the target scoring model.

12. The method of claim 8, wherein: The fusion network of the target scoring model includes several groups of extraction layers and fusion layers arranged in series; The obtaining of the target multimodal representation comprises: The target multimodal representation is obtained by using the target image and the target text representation through the plurality of groups of extraction layers and fusion layers arranged in series.

13. The method of claim 8, wherein: The scoring network of the target scoring model includes: at least one scoring sub-network, each scoring sub-network corresponds to a different image-text pair quality dimension; Obtaining the target score includes: By using the target multimodal representation and through each scoring sub-network, the target sub-score of each image-text pair under the quality dimension is obtained.

14. The method according to any one of claims 8 to 13, wherein: The target scoring model is a scoring model trained by the scoring model training method according to any one of claims 1 to 7.

15. A scoring model training device, the scoring model comprising: A text encoding network, a fusion network and a scoring network, wherein the fusion network is used for representation extraction and image-text representation fusion, and the device comprises: A first acquisition module is configured to acquire sample image-text pairs and their corresponding label data, wherein the sample image pairs include a sample image description and its corresponding sample image; A first obtaining module is configured to obtain a first text representation by using the sample image description through the text encoding network; A second obtaining module is configured to obtain a first multimodal representation by using the sample image and the first text representation through the fusion network; A third obtaining module is configured to obtain a prediction score by using the first multimodal representation through the scoring network; An adjustment module is configured to adjust the scoring model using the difference between the prediction score and the label data.

16. A device for scoring a picture-text pair, comprising: A second acquisition module is configured to acquire a target image-text pair to be scored, wherein the target image-text pair includes a target image description and a target image; A fourth obtaining module is configured to obtain a target text representation by using the target image description through a text encoding network of a target scoring model; A fifth obtaining module is configured to obtain a target multimodal representation by using the target image and the target text representation through a fusion network of the target scoring model, wherein the fusion network of the target scoring model is used to perform representation extraction and image-text representation fusion; The sixth obtaining module is configured to obtain a target score by utilizing the target multimodal representation through the scoring network of the target scoring model.

17. An electronic device comprising a memory and a processor, wherein: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 14 is implemented.

18. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 14.

Images