Scoring model training method and device, image-text pair scoring method and device, equipment and medium

Abstract

The embodiment of the invention provides a scoring model training and image-text pair scoring method and device, equipment and a medium. The method comprises the steps that a sample image-text pair comprising a first image description of a first language, a sample image corresponding to the first image description and a second image description of a second language and label data of the sample image-text pair are acquired; obtaining a first text representation through a first language encoder by using the first image description; using the second image description to obtain a second text representation through a second language encoder; using the sample image to obtain a sample image representation through an image encoder; using the first text representation, the second text representation and the sample image representation to obtain a first multi-modal representation through a multi-modal fusion network; using the first multi-modal representation to obtain a prediction score through a scoring network; and adjusting the scoring model comprising the above structure by using the difference between the prediction score and the label data so as to train and obtain the scoring model capable of performing quality evaluation on the image-text pair.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method, device, equipment and medium for scoring a scoring model and scoring a picture-text pair. Background Art

[0002] In the field of image generation, an image description (also called a text prompt) can be used to generate an image corresponding to the image description through an image generation model based on content generation technology. In the above process, the stronger the image generation capability of the image generation model, the better the quality of the generated image.

[0003] In the process of training the image generation model, the quality of the image-text pairs used is crucial to improving the image generation capability of the image generation model. Accordingly, in order to improve the image generation capability of the image generation model, the image generation model can be trained using image-text pairs of good quality. Then, how to provide a method for evaluating the quality of image-text pairs becomes an urgent problem to be solved. Summary of the invention

[0004] One or more embodiments of the present disclosure provide a method, apparatus, device and medium for training a scoring model and scoring image-text pairs, so as to train a scoring model that can evaluate the quality of image-text pairs, thereby evaluating the quality of image-text pairs.

[0005] According to a first aspect, a scoring model training method is provided, wherein the scoring model comprises: a first language encoder, an image encoder, a second language encoder, a multimodal fusion network and a scoring network, and the method comprises:

[0006] Acquire a sample image-text pair and acquire label data corresponding to the sample image-text pair, wherein the sample image-text pair includes a first image description in a first language, and a sample image corresponding to the first image description and a second image description in a second language;

[0007] Using the first image description, through the first language encoder, to obtain a first text representation;

[0008] Using the second image description, through the second language encoder, to obtain a second text representation;

[0009] Using the sample image, through the image encoder, to obtain a sample image representation;

[0010] Using the first text representation, the second text representation and the sample image representation, through the multimodal fusion network, a first multimodal representation is obtained;

[0011] Using the first multimodal representation, through the scoring network, to obtain a prediction score;

[0012] The scoring model is adjusted using the difference between the predicted score and the labeled data.

[0013] In an optional implementation, the obtaining of the sample image-text pair includes: obtaining the first image description and the sample image; using the first image description to obtain the second image description in the second language through a first language model; and using the first image description, the sample image, and the second image description as the sample image-text pair. In this implementation, the first image description can be converted into a language type through a first language model to ensure that a more accurate second image description in the second language with the same semantics as the first image description can be obtained.

[0014] In an optional implementation, the obtaining of the sample image-text pair includes: obtaining the second image description and the sample image; using the second image description to obtain the first image description in the first language through a second language model; and using the second image description, the sample image, and the first image description as the sample image-text pair. In this implementation, the second image description can be converted into a language type through a second language model to ensure that a more accurate first image description in the first language with the same semantics as the second image description can be obtained.

[0015] In an optional embodiment, obtaining the first multimodal representation includes: using the first text representation and the sample image representation to obtain a first intermediate representation through the multimodal fusion network; using the second text representation and the sample image representation to obtain a second intermediate representation through the multimodal fusion network; fusing the first intermediate representation and the second intermediate representation to obtain the first multimodal representation. In this implementation, the first intermediate representation is obtained by fusing the first text representation and the sample image representation, and the second intermediate representation is obtained by fusing the second text representation and the sample image representation; and the first intermediate representation and the second intermediate representation are then fused to obtain the first multimodal representation, so as to obtain a first multimodal representation that is fused with the image representation and the text representation of the image description in different languages ​​with the same semantics, so that the first multimodal representation can better and more accurately represent the quality of the image-text pair consisting of the image and the image description, so that the prediction score obtained by the scoring model using the first multimodal representation is more accurate and can better reflect human feedback.

[0016] In an optional embodiment, the scoring network includes: at least one scoring sub-network, the at least one scoring sub-network includes: a first scoring sub-network corresponding to a first image-text pair quality dimension; the sample image includes a sample image under the first image-text pair quality dimension, and the label data includes a label value corresponding to the sample image under the first image-text pair quality dimension; the obtaining of the predicted score includes: using the first multimodal representation, through the first scoring sub-network, obtaining a first predicted sub-score, and incorporating it into the predicted score; the adjusting of the scoring model includes: using the difference between the first predicted sub-score and the label value corresponding to the sample image under the first image-text pair quality dimension to adjust the first scoring sub-network. In this implementation, the scoring model includes at least one scoring sub-network, each of which corresponds to a different image-text pair quality dimension. Using the sample images under each image-text pair quality dimension and their corresponding label data, a scoring model for fine-grained, different-dimensional quality assessment of the image-text pair can be trained, thereby realizing the assessment of the quality of the image-text pair from different image-text pair quality dimensions.

[0017] According to a second aspect, a method for scoring a picture-text pair is provided, comprising:

[0018] Obtaining a target image-text pair to be scored, wherein the target image-text pair includes a third image description in a first language and a fourth image description in a second language corresponding to the third image description and a target image;

[0019] Using the third image description, through a first language encoder of a target scoring model, to obtain a third text representation;

[0020] Using the fourth image description, through the second language encoder of the target scoring model, to obtain a fourth text representation;

[0021] Using the target image, through the image encoder of the target scoring model, to obtain a target image representation;

[0022] Using the third text representation, the fourth text representation and the target image representation, a target multimodal representation is obtained through a multimodal fusion network of the target scoring model;

[0023] The target multimodal representation is used to obtain a target score through a scoring network of the target scoring model.

[0024] In an optional implementation, the step of obtaining the target image-text pair to be scored includes: obtaining the third image description and the target image; using the third image description and the first language model to obtain the fourth image description in the second language, so as to obtain the target image-text pair. The first language model is used to ensure that the fourth image description obtained is more accurate and more consistent with the true semantics of the third image description, that is, the fourth image description having the same semantics as the third image description is obtained.

[0025] In an optional implementation, the step of obtaining the target image-text pair to be scored includes: obtaining the fourth image description and the target image; using the fourth image description and a second language model to obtain the third image description in the first language, so as to obtain the target image-text pair. The second language model is used to ensure that the obtained third image description is more accurate and more consistent with the true semantics of the fourth image description, that is, the third image description having the same semantics as the fourth image description is obtained.

[0026] In an optional embodiment, obtaining the target multimodal representation includes: using the third text representation and the target image representation to obtain a third intermediate representation through the multimodal fusion network of the target scoring model; using the fourth text representation and the target image representation to obtain a fourth intermediate representation through the multimodal fusion network of the target scoring model; fusing the third intermediate representation and the fourth intermediate representation to obtain the target multimodal representation, so as to obtain a target multimodal representation that is fused with the target image representation and the text representation of the image description in different languages ​​with the same semantics, so that the target multimodal representation can better and more accurately represent the quality of the image-text pair consisting of the image and the image description, thereby making the score obtained by the scoring model using the target multimodal representation more accurate and more reflective of human feedback.

[0027] In an optional implementation, the scoring network of the target scoring model includes: at least one scoring sub-network, each scoring sub-network corresponds to a different image-text pair quality dimension; the obtaining of the target score includes: using the target multimodal representation, through each scoring sub-network, to obtain the target sub-score under each image-text pair quality dimension. In this implementation, the scoring model includes at least one scoring sub-network, each of which corresponds to a different image-text pair quality dimension, which can realize the evaluation of the image-text pair quality from different image-text pair quality dimensions, and obtain a more fine-grained, different-dimensional score for the image-text pair.

[0028] In an optional embodiment, the target scoring model is a scoring model trained based on the scoring model training method described in the first aspect, so that the scoring model can perform relatively accurate quality assessment on the image description in the first language and its corresponding image, and can perform relatively accurate quality assessment on the image description in the second language and its corresponding image, and the obtained score has higher accuracy.

[0029] According to a third aspect, a scoring model training device is provided, wherein the scoring model comprises: a first language encoder, an image encoder, a second language encoder, a multimodal fusion network and a scoring network, and the device comprises:

[0030] A first acquisition module is configured to acquire a sample image-text pair and acquire label data corresponding to the sample image-text pair, wherein the sample image-text pair includes a first image description in a first language, and a sample image corresponding to the first image description and a second image description in a second language;

[0031] A first obtaining module configured to obtain a first text representation by using the first image description through the first language encoder;

[0032] A second obtaining module is configured to obtain a second text representation by using the second image description through the second language encoder;

[0033] A third obtaining module is configured to obtain a sample image representation by using the sample image through the image encoder;

[0034] A fourth obtaining module is configured to obtain a first multimodal representation by using the first text representation, the second text representation and the sample image representation through the multimodal fusion network;

[0035] a fifth obtaining module configured to obtain a prediction score by using the first multimodal representation through the scoring network;

[0036] An adjustment module is configured to adjust the scoring model using the difference between the prediction score and the label data.

[0037] According to a fourth aspect, a device for scoring a picture-text pair is provided, comprising:

[0038] A second acquisition module is configured to acquire a target image-text pair to be scored, wherein the target image-text pair includes a third image description in a first language and a fourth image description in a second language corresponding to the third image description and a target image;

[0039] a sixth obtaining module configured to obtain a third text representation by using the third image description through a first language encoder of a target scoring model;

[0040] a seventh obtaining module configured to obtain a fourth text representation by using the fourth image description through a second language encoder of the target scoring model;

[0041] an eighth obtaining module configured to obtain a target image representation by using the target image through an image encoder of the target scoring model;

[0042] A ninth obtaining module, configured to obtain a target multimodal representation by using the third text representation, the fourth text representation and the target image representation through a multimodal fusion network of the target scoring model;

[0043] The tenth obtaining module is configured to obtain a target score by utilizing the target multimodal representation through the scoring network of the target scoring model.

[0044] According to a fifth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method described in the first aspect or the second aspect.

[0045] According to a sixth aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in the first aspect or the second aspect is implemented.

[0046] According to the method, device, equipment and medium for scoring model training and image-text pair scoring provided by the embodiments of the present disclosure, the scoring model includes: a first language encoder, an image encoder, a second language encoder, a multimodal fusion network and a scoring network, and the training process includes: obtaining a sample image-text pair and obtaining label data corresponding to the sample image-text pair, the sample image-text pair includes a first image description in a first language, and a sample image corresponding to the first image description and a second image description in a second language; using the first image description, through a first language encoder, to obtain a first text representation; using the second image description, through a second language encoder, to obtain a second text representation; using the sample image, through an image encoder, to obtain a sample image representation; using the first text representation, the second text representation and the sample image representation, through a multimodal fusion network, to obtain a first multimodal representation; using the first multimodal representation, through a scoring network, to obtain a predicted score; using the difference between the predicted score and the label data, to adjust the scoring model.

[0047] In the above process, the first language encoder and the second language encoder of the scoring model can be used to respectively perform text encoding on image descriptions of different language types with the same semantics to obtain respective text representations of image descriptions of different language types with the same semantics. Then, the respective text representations of image descriptions of different language types with the same semantics are used to fuse with the sample image representation to obtain a first multimodal representation, which contains the features of image descriptions of different language types, which can better represent the true meaning of the image description, and further can better reflect the quality of the corresponding image-text pairs. The first multimodal representation is used to obtain a prediction score, and then the difference between the prediction score and the label data is used to adjust the scoring model to better improve the accuracy of the scoring model's evaluation results on the image-text pair quality, and the scoring model can be enabled to perform relatively accurate quality evaluation on image-text pairs consisting of image descriptions of different language types, i.e., the first language and the second language, and their images. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 A schematic diagram of an implementation framework of an embodiment disclosed in the present disclosure;

[0050] Figure 2 A schematic diagram of a flow chart of a scoring model training method provided in an embodiment;

[0051] Figure 3 A schematic flow chart of a method for scoring image-text pairs provided in an embodiment;

[0052] Figure 4 A schematic block diagram of a scoring model training device provided in an embodiment;

[0053] Figure 5 A schematic block diagram of a device for scoring image-text pairs provided in an embodiment;

[0054] Figure 6 A schematic block diagram of an electronic device provided in an embodiment. DETAILED DESCRIPTION

[0055] The technical solution of the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0056] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0057] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0058] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0059] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0060] The present disclosure discloses a method, device, apparatus and medium for scoring a scoring model training and a picture-text pair scoring. The following first introduces the application scenarios and technical concepts of the method, which are as follows:

[0061] As mentioned above, in the process of training the image generation model, the quality of the image-text pairs used is more significant in improving the image generation capability of the image generation model. Accordingly, in order to improve the image generation capability of the image generation model, the image generation model can be trained using image-text pairs of good quality. Then, how to provide a method for evaluating the quality of image-text pairs becomes an urgent problem to be solved.

[0062] In view of this, the embodiments of the present disclosure provide a method, apparatus, device and medium for training a scoring model and scoring image-text pairs, so as to train a scoring model that can accurately evaluate the quality of image-text pairs, and the scoring model can accurately evaluate the quality of image descriptions of different language types and image-text pairs composed of their images.

[0063] Figure 1A schematic diagram of an implementation scenario according to an embodiment disclosed in the present disclosure is shown. In the implementation scenario, a schematic structure of a scoring model and an exemplary training process of the scoring model are shown. The scoring model includes: a first language encoder, an image encoder, a second language encoder, a multimodal fusion network, and a scoring network. The training process may include: obtaining sample image-text pairs and their corresponding label data, wherein the sample image-text pairs include a first image description in a first language, and a sample image corresponding to the first image description and a second image description in a second language, the first language is different from the second language, and the first image description in the first language and the second image description in the second language represent the same semantics.

[0064] Using the first image description, a first text representation corresponding to the first image description is obtained through a first language encoder; using the second image description, a second text representation corresponding to the second image description is obtained through a second language encoder; using the sample image, a sample image representation corresponding to the sample image is obtained through an image encoder. In one implementation, the first language encoder is a pre-trained encoder that can encode the image description in the first language to obtain a text representation corresponding to the image description in the first language. The second language encoder is a pre-trained encoder that can encode the image description in the second language to obtain a text representation corresponding to the image description in the second language. The image encoder is a pre-trained encoder that can encode the image to obtain an image representation corresponding to the image.

[0065] After obtaining the first text representation, the second text representation and the sample image representation, the first text representation, the second text representation and the sample image representation are used to obtain a first multimodal representation through a multimodal fusion network. The first multimodal representation is fused with the first text representation, the second text representation and the sample image representation. Afterwards, the first multimodal representation is used to obtain a predicted score through a scoring network; and then the difference between the predicted score and the label data is used to adjust the scoring model.

[0066] In the above process, the first language encoder and the second language encoder of the scoring model can be used to respectively perform text encoding on image descriptions of different language types with the same semantics to obtain respective text representations of image descriptions of different language types with the same semantics. Then, the respective text representations of image descriptions of different language types with the same semantics are used to fuse with the sample image representation to obtain a first multimodal representation, which contains the features of image descriptions of different language types, which can better represent the true meaning of the image description, and further can better reflect the quality of the corresponding image-text pairs. The first multimodal representation is used to obtain a prediction score, and then the difference between the prediction score and the label data is used to adjust the scoring model to better improve the accuracy of the scoring model's evaluation results on the image-text pair quality, and the scoring model can be enabled to perform relatively accurate quality evaluation on image-text pairs consisting of image descriptions of different language types, i.e., the first language and the second language, and their images.

[0067] In the following, in conjunction with specific embodiments, the scoring model training, image-text pair scoring method, device, equipment and medium provided by the present disclosure are described in detail.

[0068] Figure 2 The flowchart of the scoring model training method in one embodiment of the present disclosure is shown. The method can be implemented by a target program. The target program can be installed in an electronic device, and the electronic device can be implemented by any device, equipment, platform, equipment cluster, etc. with computing and processing capabilities. The scoring model includes: a first language encoder, an image encoder, a second language encoder, a multimodal fusion network and a scoring network. During the training process, Figure 2 As shown, the following steps S210-S270 may be included:

[0069] In step S210, a sample image-text pair is obtained, and label data corresponding to the sample image-text pair is obtained. The sample image-text pair includes a first image description in a first language, a sample image corresponding to the first image description, and a second image description in a second language.

[0070] In one implementation, in order to train the scoring model, a training data set for training the scoring model may be first obtained, the training data set including a number of image-text pairs and their corresponding label data, each image-text pair including an image description in a first language, and an image corresponding to the image description in the first language and a second image description in a second language, wherein the image-text pair may be referred to as a sample image-text pair, and the image therein may be referred to as a sample image. It is understandable that the first language and the second language are different types of languages, for example, the first language may be English, and the second language may be Chinese or Russian or other languages; for another example, the first language may be Chinese, and the second language may be English or other languages.

[0071] Among them, for the image description in the first language and the image description in the second language included in a sample image pair, the two have the same semantics and represent the same meaning.

[0072] In one case, the disclosed embodiment may also provide a process for obtaining the aforementioned training data set, wherein the process may include: the target program collects image descriptions in the first language and collects image descriptions in the second language; using each of the collected image descriptions in the first language and the second language, through the trained image generation model, at least one generated image corresponding to each image description is obtained, and then based on each image description and its corresponding at least one generated image, the sample image and label data corresponding to each image description are determined. The image generation model is used to generate a corresponding image for the input image description. The following takes any image description Ti in the image description in the first language and the image description in the second language as an example to illustrate the process of determining the sample image and label data corresponding to the image description. The i is a positive integer.

[0073] Specifically, the image description Ti and at least one corresponding generated image are annotated by using a specific program or manual annotation. In one case, the image description Ti and at least one corresponding generated image can be annotated under at least one preset image-text pair quality dimension to obtain a sample image corresponding to the image description Ti and corresponding label data. The preset image pair quality dimension can be any dimension that can characterize the quality of the image-text pair.

[0074] In one implementation, the image pair quality dimension may include, but is not limited to, image clarity dimension, image aesthetics dimension, and image-text matching dimension. The image pair quality dimension may also include: a comprehensive quality dimension of at least two of the image clarity dimension, image aesthetics dimension, and image-text matching dimension. The image pair quality dimension may be set according to user needs.

[0075] In the case where the image description Ti corresponds to a generated image, the generated image can be labeled as a positive sample image or a negative sample image under at least one image-text pair quality dimension, or the labeling can represent the quality score corresponding to the generated image under the image-text pair quality dimension, so as to obtain label data corresponding to the image description Ti; and the generated image can be used as the sample image corresponding to the image description Ti.

[0076] Accordingly, in one case, the label data corresponding to the image description Ti may include a label value representing that the sample image corresponding to the image description Ti is a positive sample image or a negative sample image under at least one image-text pair quality dimension, for example, including a first label value representing that the sample image corresponding to the image description Ti is a positive sample image under image-text pair quality dimension 1, including a first label value representing that the sample image corresponding to the image description Ti is a positive sample image under image-text pair quality dimension 2, including a second label value representing that the sample image corresponding to the image description Ti is a negative sample image under image-text pair quality dimension 3. In another case, the label data corresponding to the image description Ti may also include the quality scores of the sample images corresponding to the image description Ti under at least one image-text pair quality dimension.

[0077] Taking the quality dimension as the image clarity dimension as an example, the generated image is labeled as a positive sample image or a negative sample image under the image clarity dimension based on the clarity of the generated image. When the clarity of the generated image is high (for example, not lower than the first clarity threshold), the generated image can be labeled as a positive sample image under the image clarity dimension, for example, a first label value is labeled for the generated image, and the first label value is, for example, 1; when the clarity of the generated image is low (for example, lower than the second clarity threshold, and the second clarity threshold is not greater than the first clarity threshold), the generated image can be labeled as a negative sample image under the image clarity dimension, for example, a second label value is labeled for the generated image, and the second label value is, for example, 0, to obtain the label data corresponding to the image description Ti. It is also possible to label the clarity score corresponding to the generated image based on the clarity of the generated image as the quality score of the generated image under the image clarity dimension to obtain the label data corresponding to the image description Ti, wherein the higher the clarity of the image, the larger the corresponding clarity score.

[0078] In the case where the image description Ti corresponds to at least two generated images, the generated images corresponding to the image description Ti as positive samples in each image-text pair quality dimension and / or the generated images as negative samples in each image-text pair quality dimension can be screened out from the at least two generated images in at least one image-text pair quality dimension, and label data corresponding to the image description Ti is obtained, and the screened generated images as positive samples in each image-text pair quality dimension and / or the generated images as negative samples are used as sample images corresponding to the image description Ti, wherein the generated images as positive samples can be called positive sample images, and the generated images as negative samples can be called negative sample images.

[0079] For example, the generated images corresponding to the image description Ti include generated image 1, generated image 2, generated image 3 and generated image 4. The at least one image-text pair quality dimension includes an image clarity dimension, an image aesthetics dimension, an image-text matching dimension, and a comprehensive quality dimension of the three dimensions of image clarity dimension, image aesthetics dimension and image-text matching dimension, which is called the first comprehensive quality dimension.

[0080] Specifically, for example, with respect to the image clarity dimension, from generated image 1, generated image 2, generated image 3 and generated image 4 corresponding to the image description Ti, the generated images corresponding to the image description Ti are screened out as positive samples and / or generated images as negative samples in the image clarity dimension. For example, if generated image 1 has the best clarity, such as if the clarity score of generated image 1 is the largest among the four images, generated image 1 can be used as the positive sample image in the image clarity dimension corresponding to the image description Ti. If generated image 2 has the worst clarity, such as if the clarity score of generated image 2 is the smallest among the four images, generated image 2 can be used as the negative sample image in the image clarity dimension corresponding to the image description Ti.

[0081] Correspondingly, the label data corresponding to the image description Ti may include: label data corresponding to the image clarity dimension, for example, including a first label value representing that the generated image 1 is a positive sample image corresponding to the image description Ti in the image clarity dimension, and a second label value representing that the generated image 2 is a negative sample image corresponding to the image description Ti in the image clarity dimension.

[0082] For another example, with respect to the image aesthetic dimension, from generated image 1, generated image 2, generated image 3 and generated image 4 corresponding to the image description Ti, the generated images corresponding to the image description Ti as positive samples and generated images as negative samples in the image aesthetic dimension are screened out. For example, generated image 4 is the most beautiful, for example, the aesthetic score of generated image 4 is the largest among the four images, and generated image 4 can be used as the positive sample image in the image aesthetic dimension corresponding to the image description Ti. Generated image 3 is the least beautiful, for example, the aesthetic score of generated image 3 is the smallest among the four images, and generated image 3 can be used as the negative sample image in the image aesthetic dimension corresponding to the image description Ti.

[0083] Correspondingly, the label data corresponding to the image description Ti may include: label data corresponding to the image aesthetic dimension, for example, including a first label value representing that the generated image 4 is a positive sample image corresponding to the image description Ti under the image aesthetic dimension, and a second label value representing that the generated image 3 is a negative sample image corresponding to the image description Ti under the image aesthetic dimension.

[0084] For another example, with respect to the image-text matching dimension, from generated image 1, generated image 2, generated image 3 and generated image 4 corresponding to the image description Ti, the generated images corresponding to the image description Ti as positive samples and generated images as negative samples in the image-text matching dimension are screened out. For example, generated image 4 is the most matched with the image description Ti, for example, the matching value between generated image 4 and the image description Ti is the largest matching value between the four images and the image description Ti respectively, and generated image 4 is used as the positive sample image corresponding to the image description Ti in the image-text matching dimension. Generated image 2 is the least matched with the image description Ti, for example, the matching value between generated image 2 and the image description Ti is the smallest matching value between the four images and the image description Ti respectively, and generated image 2 is used as the negative sample image corresponding to the image description Ti in the image-text matching dimension.

[0085] Correspondingly, the label data corresponding to the image description Ti may include: label data corresponding to the image-text matching dimension, for example, including a first label value representing that the generated image 4 is a positive sample image corresponding to the image description Ti in the image-text matching dimension, and a second label value representing that the generated image 2 is a negative sample image corresponding to the image description Ti in the image-text matching dimension.

[0086] For another example, for the comprehensive quality dimension of the three dimensions of image clarity dimension, image aesthetics dimension, and image-text matching degree, which is called the first comprehensive quality dimension, from generated image 1, generated image 2, generated image 3, and generated image 4 corresponding to the image description Ti, the generated images as positive samples and the generated images as negative samples under the first comprehensive quality dimension corresponding to the image description Ti are screened out. For example, the comprehensive quality of generated image 4 and image description Ti is the best, that is, the overall quality of generated image 4 and image description Ti is the best in the three dimensions of image clarity dimension, image aesthetics dimension, and image-text matching degree. Generated image 4 is used as the positive sample image corresponding to the image description Ti in the first comprehensive quality dimension. The comprehensive quality of generated image 2 and image description Ti is the worst, that is, the overall quality of generated image 2 and image description Ti is the worst in the three dimensions of image clarity dimension, image aesthetics dimension, and image-text matching degree. Generated image 2 is used as the negative sample image corresponding to the image description Ti in the first comprehensive quality dimension.

[0087] Correspondingly, the label data corresponding to the image description Ti may include: the label data corresponding to the aforementioned first comprehensive quality dimension, for example, including a first label value representing that the generated image 4 is a positive sample image under the first comprehensive quality dimension corresponding to the image description Ti, and a second label value representing that the generated image 2 is a negative sample image under the first comprehensive quality dimension corresponding to the image description Ti.

[0088] Through the above method, sample images and label data corresponding to any image description Ti in each image description in the first language and each image description in the second language can be obtained. Among them, the sample image corresponding to the image description Ti may include, at least one sample image under the image-text pair quality dimension, specifically, may include, positive sample images and / or negative sample images under each image-text pair quality dimension. Correspondingly, the label data corresponding to the image description Ti may include: label data corresponding to at least one image-text pair quality dimension, and the label data corresponding to each image-text pair quality dimension include: the label value corresponding to the sample image corresponding to the image description Ti under the image-text pair quality dimension, for example, including, the first label value corresponding to the positive sample image under each image-text pair quality dimension, and / or the second label value corresponding to the negative sample image under each image-text pair quality dimension. Among them, the first label value is different from the second label value.

[0089] It can be understood that, between different image-text pair quality dimensions, the positive sample images corresponding to the image description Ti may be the same or different, and the negative sample images corresponding to the image description Ti may be the same or different.

[0090] Afterwards, for the image description in the first language, the image description in the first language can be translated using a designated translator to obtain an image description in the second language corresponding to the image description in the first language; then, the image description in the first language, the image description in the second language obtained by translating the image description in the first language, and the sample image corresponding to the description in the first language are combined to obtain a sample image-text pair corresponding to the image description in the first language.

[0091] In one implementation, there is one sample image-text pair corresponding to the image description in the first language, which may include the image description in the first language, the image description in the second language translated from the image description in the first language, and all sample images corresponding to the image description in the first language, that is, sample images under the quality dimension of each image-text pair corresponding to the image description in the first language. Subsequently, the label data corresponding to the image description in the first language is used as the label data corresponding to the sample image-text pair containing the image description in the first language.

[0092] In another implementation, there may be multiple sample image-text pairs corresponding to the image description in the first language, and each sample image-text pair corresponds to one of the aforementioned image-text pair quality dimensions. Accordingly, each sample image-text pair may include: an image description in the first language, an image description in the second language obtained by translating the image description in the first language, and sample images corresponding to the image description in the first language, and sample images under the image-text pair quality dimension corresponding to the sample image-text pair. Subsequently, based on the label data corresponding to the image description in the first language, the label data corresponding to each of the multiple sample image-text pairs corresponding to the image description in the first language are determined.

[0093] Continuing with the above example: for example, if the image description Ti is any image description in the image description in the first language, the image description Ti may correspond to four sample image-text pairs, for example, including: a sample image-text pair corresponding to the image clarity dimension, including: the image description Ti and its corresponding translation obtained in another language, and the generated image 1 as a positive sample image and / or the generated image 2 as a negative sample image under the image clarity dimension; a sample image-text pair corresponding to the image aesthetics dimension, including: the image description Ti and its corresponding translation obtained in another language, and the generated image 4 as a positive sample image and / or the generated image 3 as a negative sample image under the image aesthetics dimension; a sample image-text pair corresponding to the image-text matching dimension, including: the image description Ti and its corresponding translation obtained in another language, and the generated image 4 as a positive sample image and / or the generated image 2 as a negative sample image under the image-text matching dimension; a sample image-text pair corresponding to the first comprehensive quality dimension, including: the image description Ti and its corresponding translation obtained in another language, and the generated image 4 as a positive sample image and / or the generated image 2 as a negative sample image under the first comprehensive quality dimension.

[0094] Correspondingly, for example, the label data corresponding to the sample image-text pair corresponding to the image definition dimension includes: a first label value representing the generated image 1 as a positive sample image under the image definition dimension, and / or a second label value representing the generated image 2 as a negative sample image under the image definition dimension. The label data corresponding to the sample image-text pair corresponding to the image aesthetic dimension includes: a first label value representing the generated image 4 as a positive sample image under the image definition dimension, and / or a second label value representing the generated image 3 as a negative sample image under the image definition dimension.

[0095] In one embodiment, in order to ensure the accuracy of the image description in the first language and the image description in the second language in the sample image-text pair, it is necessary to ensure that the image description in the first language and the image description in the second language have closer or identical semantics. In order to ensure that the semantics of the image description in the second language obtained by translation of the image description in the first language are highly accurate, that is, to ensure that the image description in the second language has the same semantics as the image description in the first language, specifically, in step S210, the process of obtaining the sample image-text pair may include steps 11-13:

[0096] In step 11, a first image description and a sample image are obtained. The sample image may include a positive sample image and / or a negative sample image under at least one image-text pair quality dimension. The first image description is an image description in a first language.

[0097] Then, in step 12, the first image description is used to obtain a second image description in a second language through a first language model. In this step, in order to make the semantic accuracy of the image description in the second language translated by using the first image description higher, so that the semantics of the image description in the second language translated is closer to or the same as the semantics of the first image description, the first image description is translated using the first language model to obtain a second image description in the second language corresponding to the first image description. Among them, the first language model is a model based on natural language processing technology that has been trained with a large amount of corpus in advance. The first language model is used to translate the corpus in the first language into the corpus in the second language that is closer to or the same as the semantics of the first language. For example, the first language is Chinese, and accordingly, the first image description is a Chinese image description, and the second language is English, and accordingly, the first language model can translate Chinese into English. For example, the first image description is "donkey meat fire", and through the first language model, the second image description in the second language can be obtained as "baked cake of donkey meat".

[0098] Then, in step 13, the first image description, the sample image, and the second image description are used as sample image-text pairs. In this step, the target program uses the first image description in the first language and its corresponding sample image and the second image description in the second language as sample image-text pairs to obtain training data required for training the scoring model.

[0099] For the image description in the second language, the image description in the second language can be translated using a designated translator to obtain an image description in the first language corresponding to the image description in the second language; then, the image description in the second language, the image description in the first language obtained by translating the image description in the second language, and the sample image corresponding to the description in the second language are combined to obtain a sample image-text pair corresponding to the image description in the second language. The process of obtaining the sample image-text pair corresponding to the image description in the second language can refer to the aforementioned process of obtaining the sample image-text pair corresponding to the image description in the first language, and will not be described in detail here.

[0100] In one embodiment, in order to ensure the accuracy of the image description in the first language and the image description in the second language in the sample image-text pair, it is necessary to ensure that the image description in the first language and the image description in the second language have closer or identical semantics. In order to ensure that the semantics of the image description in the first language obtained by translation for the image description in the second language are more accurate, that is, the image description in the first language and the image description in the second language have identical semantics. Specifically, in step S210, the process of obtaining the sample image-text pair may include steps 21-23:

[0101] In step 21, a second image description and a sample image are obtained. The sample image may include a positive sample image and / or a negative sample image under at least one image-text pair quality dimension. The second image description is an image description in a second language.

[0102] Then, in step 22, the second image description is used to obtain the first image description in the first language through the second language model. In this step, in order to make the semantic accuracy of the image description in the first language obtained by translating the second image description higher, so that the semantics of the translated image description in the first language is closer to or the same as the semantics of the second image description, the second image description is translated using the second language model to obtain the first image description in the first language corresponding to the second image description. The second language model is a model based on natural language processing technology that has been pre-trained with a large amount of corpus, and the second language model is used to translate the corpus in the second language into the corpus in the first language that is closer to or the same as the semantics of the second language.

[0103] Then, in step 23, the second image description, the sample image and the first image description are used as sample image-text pairs. In this step, the target program uses the second image description in the second language and its corresponding sample image and the first image description in the first language as sample image-text pairs to obtain training data required for training the scoring model.

[0104] Through the above method, for any collected image description in the first language, its corresponding sample image and image description in the second language and label data can be obtained to obtain the corresponding sample image-text pairs and their corresponding label data; and for any collected image description in the second language, its corresponding sample image and image description in the first language and label data can be obtained to obtain the corresponding sample image-text pairs and their corresponding label data, thereby obtaining a training data set including several sample image-text pairs and their corresponding label data.

[0105] The target program can obtain any sample image-text pair from the training data set, and obtain the label data corresponding to the sample image-text pair, and then perform the subsequent scoring model training process based on the obtained sample image-text pair and its corresponding label data. Subsequently, the image description in the first language of the sample image-text pair is referred to as the first image description, and the image description in the second language of the sample image-text pair is referred to as the second image description.

[0106] In step S220, the first image description is used to obtain a first text representation through a first language encoder. In this step, the target program can input the first image description into the first language encoder, encode the first image description through the first language encoder, and obtain a first text representation corresponding to the first image description. The first language encoder is an encoder that encodes the image description in the first language, and the first language encoder is a pre-trained encoder. In one implementation, the first language encoder can be implemented by a text encoder in a CLIP (Contrastive Language-Image Pre-Training) model.

[0107] Next, in step S230, the second image description is used to obtain a second text representation through a second language encoder. In this step, the target program can input the second image description into the second language encoder, encode the second image description through the second language encoder, and obtain a second text representation corresponding to the second image description. The second language encoder is an encoder that encodes the image description in the second language, and the second language encoder is a pre-trained encoder. In one implementation, the second language encoder can be implemented by a text encoder in a CLIP (Contrastive Language-Image Pre-Training) model.

[0108] Next, in step S240, the sample image is used to obtain a sample image representation through an image encoder. In this step, the target program can input the sample image into the image encoder, encode the sample image through the image encoder, and obtain a sample image representation corresponding to the sample image. The image encoder is a pre-trained encoder. In one implementation, the image encoder can be implemented by an image encoder in a CLIP (Contrastive Language-Image Pre-Training) model.

[0109] After obtaining the first text representation corresponding to the first image description, the second text representation corresponding to the second image description, and the sample image representation corresponding to the sample image, in step S250, the first text representation, the second text representation, and the sample image representation are used to obtain a first multimodal representation through a multimodal fusion network. In this step, the target program inputs the first text representation, the second text representation, and the sample image representation into the multimodal fusion network, and fuses the first text representation, the second text representation, and the sample image representation through the multimodal fusion network to obtain a first multimodal representation. The first multimodal representation is a fusion of the first text representation, the second text representation, and the sample image representation.

[0110] In one embodiment, step S250 may include the following steps 31-33:

[0111] In step 31, the first text representation and the sample image representation are used to obtain a first intermediate representation through a multimodal fusion network. In this step, the target program can input the first text representation and the sample image representation into the multimodal fusion network, and fuse the first text representation and the sample image representation through the multimodal fusion network to obtain a first intermediate representation. In one implementation, the process of fusing the first text representation and the sample image representation through the multimodal fusion network can include: fusing the first text representation and the sample image representation through the multimodal fusion network using a dot product operation.

[0112] Then, in step 32, the second text representation and the sample image representation are used to obtain a second intermediate representation through a multimodal fusion network. In this step, the target program can input the second text representation and the sample image representation into the multimodal fusion network, and fuse the second text representation and the sample image representation through the multimodal fusion network to obtain a second intermediate representation. In one implementation, the process of fusing the second text representation and the sample image representation through the multimodal fusion network can include: fusing the second text representation and the sample image representation through the multimodal fusion network using a dot product operation.

[0113] Then, in step 33, the first intermediate representation and the second intermediate representation are fused to obtain a first multimodal representation. In this step, the target program can use any specified fusion method to fuse the first intermediate representation and the second intermediate representation to obtain the first multimodal representation, for example, the first intermediate representation and the second intermediate representation are added to fuse the first intermediate representation and the second intermediate representation to obtain the first multimodal representation.

[0114] In the above process, the first multimodal representation obtained integrates the sample image representation and the text representation of the image description in different languages ​​with the same semantics, so that the first multimodal representation can better and more accurately represent the quality of the image-text pair consisting of the image and the image description, thereby making the prediction score obtained by the scoring model using the first multimodal representation more accurate and more able to reflect human feedback.

[0115] After obtaining the first multimodal representation, in step S260, the first multimodal representation is used to obtain a prediction score through a scoring network. In this step, the target program inputs the first multimodal representation into the scoring network, processes the first multimodal representation through the scoring network, and obtains a prediction score corresponding to the sample image-text pair. The prediction score can characterize the quality of the sample image-text pair, for example, it can characterize the quality of the sample image-text pair under the aforementioned image-text pair quality dimension. In one case, the larger the prediction score is, the better the quality of the sample image-text pair evaluated by the scoring model. The value range of the prediction score is [0, 1]. In one implementation, the scoring network can be implemented by MLP (multi-layer perceptron).

[0116] After obtaining the predicted score, in step S270, the scoring model is adjusted using the difference between the predicted score and the label data. In this step, the target program can use a specified loss function to adjust the scoring model using the difference between the predicted score and the label data. The specified loss function may include, but is not limited to, a cross entropy loss function and an MSE (mean square error) loss function, etc. In one implementation, the target program can use a specified loss function to construct a prediction loss corresponding to the scoring model using the difference between the predicted score and the label data, wherein the prediction loss is positively correlated with the difference between the predicted score and the label data. Then, the scoring model is adjusted with the goal of minimizing the prediction loss, that is, minimizing the difference between the predicted score and the label data.

[0117] In one implementation, adjusting the scoring model may include: adjusting parameters of a scoring network. In another implementation, in the case of setting a first language encoder, an image encoder, a second language encoder, and a multimodal fusion network that need to adjust the scoring model, adjusting the scoring model may include: adjusting the parameters of the first language encoder, the image encoder, the second language encoder, the multimodal fusion network, and the scoring network.

[0118] In one embodiment, in order to achieve a more fine-grained quality assessment of the image-text pair, the scoring network may include: at least one scoring sub-network, wherein each scoring sub-network corresponds to a different image-text pair quality dimension, for example, the scoring network may include but is not limited to the scoring sub-network corresponding to the aforementioned image clarity dimension, the scoring sub-network corresponding to the image aesthetics dimension, and the scoring sub-network corresponding to the image-text matching degree dimension. Among them, the scoring sub-network corresponding to the image clarity dimension is used to perform quality assessment on the image-text pair under the image clarity dimension, the scoring sub-network corresponding to the image aesthetics dimension is used to perform quality assessment on the image-text pair under the image aesthetics dimension, and the scoring sub-network corresponding to the image-text matching degree dimension is used to perform quality assessment on the image-text pair under the image-text matching degree dimension.

[0119] In addition, the scoring network may also include other dimensions for evaluating the quality of image-text pairs, for example, it may also include scoring sub-networks corresponding to comprehensive quality dimensions of at least two of the image clarity dimension, the image aesthetics dimension, and the image-text matching dimension. This type of scoring sub-network can perform comprehensive quality evaluation of image-text pairs in at least two of the image clarity dimension, the image aesthetics dimension, and the image-text matching dimension.

[0120] In one case, the number and corresponding dimensions of the scoring sub-networks included in the scoring network can be set according to user requirements. The label data corresponding to the sample image-text pair is related to the dimensions corresponding to the scoring sub-networks included in the scoring network. In one implementation, each scoring sub-network can be implemented by an MLP (multi-layer perceptron).

[0121] In one implementation, the label data includes label values ​​corresponding to the sample images under each image-text pair quality dimension, and step S260 may include: using the first multimodal representation, through each scoring sub-network, obtaining the prediction sub-score corresponding to each scoring sub-network, and including it in the prediction score. Then, in step S270, using the prediction sub-score of each scoring sub-network and the label values ​​corresponding to the sample images under each image-text pair quality dimension corresponding to each scoring sub-network in the label data, each scoring sub-network is adjusted.

[0122] In another implementation, each of the scoring sub-networks corresponds to a different image-text pair quality dimension, and different image-text pair quality dimensions may correspond to corresponding sample image-text pairs. The target program may use the sample image-text pairs corresponding to the different image-text pair quality dimensions and their corresponding label data to adjust and train the corresponding scoring sub-network. It can be understood that the data processing process and the corresponding adjustment process of different scoring sub-networks are similar. The data processing process and the corresponding adjustment process of any first scoring sub-network in at least one scoring sub-network are used as an example for explanation. Specifically, the at least one scoring sub-network includes: a first scoring sub-network corresponding to a first image-text pair quality dimension, and the first image-text pair quality dimension can be any dimension of the at least one image-text pair quality dimension mentioned above; the sample image in the sample image-text pair includes the sample image under the first image-text pair quality dimension, and the label data corresponding to the sample image-text pair includes the label value corresponding to the sample image under the first image-text pair quality dimension; accordingly, in step S260, it includes: using the first multimodal representation, through the first scoring sub-network, to obtain a first prediction sub-score, and including it in the prediction score; in this step, the target program inputs the first multimodal representation into the first scoring sub-network, obtains the first prediction sub-score of the sample image-text pair under the first image-text pair quality dimension, includes it in the prediction score, and obtains the prediction score.

[0123] Then in step S270, it includes: using the difference between the first prediction sub-score and the label value corresponding to the sample image under the first image-text pair quality dimension, adjusting the first scoring sub-network. In this step, a specified loss function can be used to determine the first prediction sub-loss by using the difference between the first prediction sub-score and the label value corresponding to the sample image under the first image-text pair quality dimension, and include it in the aforementioned prediction loss; the first scoring sub-network is adjusted with the goal of minimizing the prediction loss, that is, minimizing the first prediction sub-loss, that is, minimizing the difference between the first prediction sub-score and the label value corresponding to the sample image under the first image-text pair quality dimension.

[0124] In one implementation, when setting a first language encoder, an image encoder, a second language encoder, and a multimodal fusion network in which the scoring model needs to be adjusted, the difference between the first prediction sub-score and the label value corresponding to the sample image under the first image-text pair quality dimension can be used to adjust the first scoring sub-network, the first language encoder, the image encoder, the second language encoder, and the multimodal fusion network.

[0125] It can be understood that each sample image-text pair in the training sample set may include a first image description in a first language and its corresponding second image description in a second language, as well as a sample image under at least one of the aforementioned image-text pair quality dimensions. Using the sample image-text pairs and the label data corresponding to the sample image-text pairs, each scoring sub-network corresponding to each image pair quality dimension in the scoring model can be trained and adjusted to improve the accuracy of the scores output by each scoring sub-network. Each scoring sub-network is used to evaluate the quality of the image-text pair under each image-text pair quality dimension.

[0126] In one embodiment, the sample image may include a positive sample image and / or a negative sample image, and the label data includes a first label value corresponding to the positive sample image and / or a second label value corresponding to the negative sample image; accordingly, in step S240, the process of obtaining the sample image representation includes: using the positive sample image to obtain the positive sample image representation through an image encoder; and / or using the negative sample image to obtain the negative sample image representation through an image encoder, wherein the specific implementation process can refer to the aforementioned process of obtaining the sample image representation.

[0127] Next, in step S250, the process of obtaining the first multimodal representation includes: using the first text representation, the second text representation and the positive sample image representation to obtain the positive sample multimodal representation through a multimodal fusion network; and / or using the first text representation, the second text representation and the negative sample image representation to obtain the negative sample multimodal representation through a multimodal fusion network; wherein the specific implementation process can refer to the aforementioned process of obtaining the first multimodal representation.

[0128] Then in step S260, the process of obtaining the prediction score includes: using the multimodal representation of the positive sample to obtain a first prediction score through a scoring network; and / or using the multimodal representation of the negative sample to obtain a second prediction score through a scoring network; wherein the specific implementation process can refer to the aforementioned process of obtaining the prediction score.

[0129] Finally, in step S270, the process of adjusting the scoring model includes: adjusting the scoring model using the difference between the first prediction score and the first label value, and / or the difference between the second prediction score and the second label value. In this step, the scoring model is adjusted, for example, the parameters of the scoring network are adjusted, with the goal of reducing the difference between the first prediction score and the first label value, and / or reducing the difference between the second prediction score and the second label value.

[0130] This allows the scoring model to distinguish the quality of different sample images corresponding to the first image description in the first language and its corresponding second image description in the second language, so as to better ensure the accuracy of the prediction result of the scoring model, that is, the predicted score for the image-text pair.

[0131] Let's review the execution process of the aforementioned steps S210-S270. The above embodiment is explained by taking a sample image-text pair as an example. In another embodiment, the above steps S210-S260 can also be performed for a batch of samples, i.e., multiple sample image-text pairs, to obtain the prediction scores corresponding to each sample image-text pair, and then adjust the scoring model in step S270 based on the difference between the prediction scores corresponding to each sample image-text pair and the label data corresponding to the sample image-text pair. In this embodiment, the scoring model is adjusted for a batch of samples, which can reduce the number of adjustments to the scoring model and make it easier to implement the training process.

[0132] The scoring model is adjusted until the scoring model reaches a preset convergence condition to obtain a trained scoring model. The preset convergence condition may include but is not limited to the number of parameter adjustments of the scoring model exceeding a preset number threshold, or the predicted loss of the determined scoring model being lower than a preset loss threshold. In one implementation, the scoring model reaching a preset convergence condition may include: each scoring sub-network reaches a preset convergence condition, for example: the number of adjustments of each scoring sub-network exceeds a preset number threshold, or the loss determined for each scoring sub-network is lower than a preset loss threshold.

[0133] In this embodiment, the first language encoder and the second language encoder of the scoring model can be used to respectively perform text encoding on image descriptions of different language types with the same semantics to obtain respective text representations of the image descriptions of different language types with the same semantics. Then, the respective text representations of the image descriptions of different language types with the same semantics are used to fuse with the sample image representation to obtain a first multimodal representation, which includes the respective features of the image descriptions of different language types, which can better represent the true meaning of the image description and can better reflect the quality of the image-text pair. The first multimodal representation is used to obtain a prediction score, and then the difference between the prediction score and the label data is used to adjust the scoring model to better improve the accuracy of the scoring model's evaluation results on the image-text pair quality, and the scoring model can be enabled to perform relatively accurate quality evaluation on image descriptions of different language types (i.e., the first language and the second language) and image-text pairs composed of the images.

[0134] Corresponding to the above method embodiment, the embodiment of the present disclosure also provides a method for scoring image-text pairs, which can be implemented by a first program. The first program can be the same physical program as the aforementioned target program, or can be a different physical program from the aforementioned target program, such as Figure 3 As shown, the method may include the following steps S310-S360:

[0135] It can be understood that the method for scoring the image-text pair can be a method for scoring the image-text pair based on a target scoring model, and the target scoring model can include a first language encoder, an image encoder, a second language encoder, a multimodal fusion network, and a scoring network. In one embodiment, the target scoring model can be a scoring model trained based on the scoring model training method provided in the aforementioned embodiment, that is, it can be a scoring model trained in the aforementioned embodiment.

[0136] In step S310, a target image-text pair to be scored is obtained, wherein the target image-text pair includes a third image description in a first language, a fourth image description in a second language corresponding to the third image description, and a target image. The target image-text pair can be any image-text pair that needs to be scored for quality.

[0137] In one implementation, in order to better improve the quality of user service, the user can input any image description in the first language and its corresponding image, or the image description in the second language and its corresponding image, and the first program provides translation services for the image description to obtain the target image-text pair to be scored. Specifically, in one embodiment, in step S310, the following steps 41-42 may be included: In step 41, a third image description and a target image are obtained. In step 42, the third image description is used to obtain a fourth image description in the second language through the first language model to obtain a target image-text pair. Among them, the process of obtaining the target image-text pair can refer to the aforementioned process of obtaining the sample image-text pair, which will not be repeated here.

[0138] In another embodiment, step S310 may include the following steps 51-52: In step 51, a fourth image description and a target image are obtained. In step 52, the third image description in the first language is obtained by using the fourth image description through the second language model to obtain a target image-text pair. The process of obtaining the target image-text pair can refer to the process of obtaining the sample image-text pair mentioned above, and will not be described in detail here.

[0139] After obtaining the target image-text pair, in step S320, the third image description is used to obtain a third text representation through the first language encoder of the target scoring model. The process of obtaining the third text representation can refer to the process of obtaining the first text representation, which will not be described in detail here.

[0140] Then, in step S330, the fourth image description is used to obtain a fourth text representation through the second language encoder of the target scoring model. The process of obtaining the fourth text representation can refer to the process of obtaining the fourth text representation described above, and will not be described in detail here.

[0141] Next, in step S340, the target image is used to obtain the target image representation through the image encoder of the target scoring model. The process of obtaining the target image representation can refer to the process of obtaining the sample image representation mentioned above, and will not be described in detail here.

[0142] Then, in step S350, the third text representation, the fourth text representation and the target image representation are used to obtain a target multimodal representation through a multimodal fusion network of a target scoring model.

[0143] Specifically, in one embodiment, step S350 may include the following steps 61-63:

[0144] In step 61, the third text representation and the target image representation are used to obtain a third intermediate representation through a multimodal fusion network of a target scoring model. The process of obtaining the third intermediate representation can refer to the process of obtaining the first intermediate representation, which will not be described in detail here.

[0145] In step 62, the fourth text representation and the target image representation are used to obtain a fourth intermediate representation through a multimodal fusion network of the target scoring model. The process of obtaining the fourth intermediate representation can refer to the process of obtaining the second intermediate representation, which will not be described in detail here.

[0146] In step 63, the third intermediate representation and the fourth intermediate representation are fused to obtain the target multimodal representation. The process of obtaining the target multimodal representation can refer to the process of obtaining the first multimodal representation, which will not be described in detail here.

[0147] After obtaining the target multimodal representation, in step S360, the target multimodal representation is used to obtain the target score through the scoring network of the target scoring model. In this step, the first program inputs the target multimodal representation into the scoring network of the target scoring model, and processes the target multimodal representation through the scoring network of the target scoring model to obtain a target score that can characterize the quality of the target image-text pair.

[0148] In one embodiment, the scoring network of the target scoring model may include: at least one scoring sub-network, each scoring sub-network corresponds to a different image-text pair quality dimension mentioned in the aforementioned embodiment, and each scoring sub-network can evaluate the quality of the target image-text pair under the image-text pair quality dimension corresponding to the scoring sub-network; in step S360, it may include: using the target multimodal representation to obtain the target sub-score under each image-text pair quality dimension through each scoring sub-network. The target sub-score under each image-text pair quality dimension can represent the quality of the target image-text pair evaluated by the target scoring model under the corresponding image-text pair quality dimension. The value range of the target sub-score can be set to [0, 1], wherein the larger the target sub-score, the better the quality of the target image-text pair under the image-text pair quality dimension corresponding to the target sub-score.

[0149] In the above process, the target scoring model can be used to obtain a target score that can better characterize the quality of the target image-text pair and has a higher accuracy. In addition, when the scoring network of the target scoring model includes multiple scoring sub-networks, the target scoring model can be used to obtain a more fine-grained target score that characterizes the quality of the target image-text pair under different image-text pair quality dimensions, so as to more intuitively characterize the quality of the target image-text pair.

[0150] The foregoing describes specific embodiments of the present disclosure, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims may be performed in an order different from that in the embodiments, and the desired results may still be achieved. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible, or may be advantageous.

[0151] Corresponding to the above method embodiment, the present disclosure embodiment provides a scoring model training device 400, whose schematic block diagram is as follows: Figure 4 As shown, it is used to execute Figure 2 The method shown, the scoring model includes: a first language encoder, an image encoder, a second language encoder, a multimodal fusion network and a scoring network, and the device 400 includes:

[0152] A first acquisition module 410 is configured to acquire a sample image-text pair and acquire label data corresponding to the sample image-text pair, wherein the sample image-text pair includes a first image description in a first language, and a sample image corresponding to the first image description and a second image description in a second language;

[0153] A first obtaining module 420 is configured to obtain a first text representation by using the first image description through the first language encoder;

[0154] A second obtaining module 430 is configured to obtain a second text representation by using the second image description through the second language encoder;

[0155] A third obtaining module 440 is configured to obtain a sample image representation by using the sample image through the image encoder;

[0156] A fourth obtaining module 450 is configured to obtain a first multimodal representation by using the first text representation, the second text representation and the sample image representation through the multimodal fusion network;

[0157] A fifth obtaining module 460 is configured to obtain a prediction score by using the first multimodal representation through the scoring network;

[0158] The adjustment module 470 is configured to adjust the scoring model using the difference between the prediction score and the label data.

[0159] In an optional implementation, the first acquisition module 410 is specifically configured to acquire the first image description and the sample image;

[0160] Using the first image description and a first language model, obtain the second image description in the second language;

[0161] The first image description, the sample image and the second image description are used as the sample image-text pair.

[0162] In an optional implementation, the first acquisition module 410 is specifically configured to acquire the second image description and the sample image;

[0163] Using the second image description and a second language model, obtain the first image description in the first language;

[0164] The second image description, the sample image and the first image description are used as the sample image-text pair.

[0165] In an optional implementation, the fourth obtaining module 450 is specifically configured to obtain a first intermediate representation by using the first text representation and the sample image representation through the multimodal fusion network;

[0166] Using the second text representation and the sample image representation, through the multimodal fusion network, to obtain a second intermediate representation;

[0167] The first intermediate representation and the second intermediate representation are fused to obtain the first multimodal representation.

[0168] In an optional implementation, the scoring network includes: at least one scoring sub-network, the at least one scoring sub-network includes: a first scoring sub-network corresponding to a first image-text pair quality dimension; the sample image includes a sample image under the first image-text pair quality dimension, and the label data includes a label value corresponding to the sample image under the first image-text pair quality dimension;

[0169] The fifth obtaining module 460 is specifically configured to obtain a first prediction sub-score by using the first multimodal representation through the first scoring sub-network, and include it in the prediction score;

[0170] The adjustment module 470 is specifically configured to adjust the first scoring sub-network by using the difference between the first prediction sub-score and the label value corresponding to the sample image under the first image-text pair quality dimension.

[0171] Corresponding to the above method embodiment, the present disclosure embodiment provides a device 500 for scoring image-text pairs, and its schematic block diagram is as follows: Figure 5 As shown, it is used to execute Figure 3 The method shown, the device 500 includes:

[0172] The second acquisition module 510 is configured to acquire a target image-text pair to be scored, wherein the target image-text pair includes a third image description in a first language and a fourth image description in a second language corresponding to the third image description and a target image;

[0173] A sixth obtaining module 520 is configured to obtain a third text representation by using the third image description through a first language encoder of a target scoring model;

[0174] A seventh obtaining module 530 is configured to obtain a fourth text representation by using the fourth image description through a second language encoder of the target scoring model;

[0175] An eighth obtaining module 540 is configured to obtain a target image representation by using the target image through an image encoder of the target scoring model;

[0176] A ninth obtaining module 550 is configured to obtain a target multimodal representation by using the third text representation, the fourth text representation and the target image representation through a multimodal fusion network of the target scoring model;

[0177] The tenth obtaining module 560 is configured to obtain a target score by using the target multimodal representation through the scoring network of the target scoring model.

[0178] In an optional implementation, the second acquisition module 510 is specifically configured to acquire the third image description and the target image;

[0179] The fourth image description in the second language is obtained by using the third image description through a first language model to obtain the target image-text pair.

[0180] In an optional implementation, the second acquisition module 510 is specifically configured to acquire the fourth image description and the target image;

[0181] The third image description in the first language is obtained by using the fourth image description through a second language model to obtain the target image-text pair.

[0182] In an optional implementation, the eighth obtaining module 540 is specifically configured to obtain a third intermediate representation by using the third text representation and the target image representation through a multimodal fusion network of the target scoring model;

[0183] Using the fourth text representation and the target image representation, a fourth intermediate representation is obtained through a multimodal fusion network of the target scoring model;

[0184] The third intermediate representation and the fourth intermediate representation are fused to obtain the target multimodal representation.

[0185] In an optional implementation, the scoring network of the target scoring model includes: at least one scoring sub-network, each scoring sub-network corresponding to a different image-text pair quality dimension;

[0186] The tenth obtaining module 560 is specifically configured to utilize the target multimodal representation to obtain the target sub-score of each image-text pair quality dimension through each scoring sub-network.

[0187] In an optional implementation, the target scoring model is a scoring model trained by the scoring model training device described in the aforementioned embodiment.

[0188] The above device embodiments correspond to the method embodiments. For specific descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For specific descriptions, please refer to the corresponding method embodiments.

[0189] The present disclosure provides an electronic device, including a memory and a processor, wherein the memory stores an executable code, and when the processor executes the executable code, the following is implemented: Figure 2 or Figure 3 The method shown below. Figure 6 , which shows a schematic structural diagram of an electronic device suitable for implementing an embodiment of the present disclosure. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0190] like Figure 6 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In RAM 603, various programs and data required for the operation of the electronic device are also stored. The processing device 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0191] Typically, the following devices may be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, etc.; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Figure 6 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 6 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0192] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0193] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute the following steps provided in the present disclosure: Figure 2 or Figure 3 The method shown.

[0194] It should be noted that the computer-readable medium described in the embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0195] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains a sample image-text pair and obtains label data corresponding to the sample image-text pair, wherein the sample image-text pair includes a first image description in a first language, and a sample image corresponding to the first image description and a second image description in a second language; uses the first image description to obtain a first text representation through a first language encoder of a scoring model; uses the second image description to obtain a second text representation through a second language encoder of the scoring model; uses the sample image to obtain a sample image representation through an image encoder of the scoring model; uses the first text representation, the second text representation and the sample image representation to obtain a first multimodal representation through a multimodal fusion network of the scoring model; uses the first multimodal representation to obtain a prediction score through a scoring network of the scoring model; uses the difference between the prediction score and the label data to adjust the scoring model. Alternatively, the electronic device is configured to: obtain a target image-text pair to be scored, wherein the target image-text pair includes a third image description in a first language and a fourth image description in a second language corresponding to the third image description and a target image; utilize the third image description to obtain a third text representation through a first language encoder of a target scoring model; utilize the fourth image description to obtain a fourth text representation through a second language encoder of the target scoring model; utilize the target image to obtain a target image representation through an image encoder of the target scoring model; utilize the third text representation, the fourth text representation and the target image representation to obtain a target multimodal representation through a multimodal fusion network of the target scoring model; utilize the target multimodal representation to obtain a target score through a scoring network of the target scoring model.

[0196] Computer program code for performing the operations of the disclosed embodiments may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0197] Each embodiment in the present disclosure is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the storage medium and computing device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0198] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present invention may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0199] The specific implementation methods described above further describe the purpose, technical solutions and beneficial effects of the embodiments of the present invention in detail. It should be understood that the above description is only a specific implementation method of the embodiments of the present invention and is not intended to limit the scope of protection of the present invention. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solution of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A scoring model training method, the scoring model comprising: A first language encoder, an image encoder, a second language encoder, a multimodal fusion network and a scoring network, the method comprising: Acquire a sample image-text pair and acquire label data corresponding to the sample image-text pair, wherein the sample image-text pair includes a first image description in a first language, and a sample image corresponding to the first image description and a second image description in a second language; Using the first image description, through the first language encoder, to obtain a first text representation; Using the second image description, through the second language encoder, to obtain a second text representation; Using the sample image, through the image encoder, to obtain a sample image representation; Using the first text representation, the second text representation and the sample image representation, through the multimodal fusion network, a first multimodal representation is obtained; Using the first multimodal representation, through the scoring network, to obtain a prediction score; The scoring model is adjusted using the difference between the predicted score and the labeled data.

2. The method according to claim 1, wherein obtaining sample image-text pairs comprises: Obtaining the first image description and the sample image; Using the first image description and a first language model, obtain the second image description in the second language; The first image description, the sample image and the second image description are used as the sample image-text pair.

3. The method according to claim 1, wherein obtaining sample image-text pairs comprises: Acquire the second image description and the sample image; Using the second image description and a second language model, obtain the first image description in the first language; The second image description, the sample image and the first image description are used as the sample image-text pair.

4. The method of claim 1, wherein: The obtaining of the first multimodal representation comprises: Using the first text representation and the sample image representation, through the multimodal fusion network, to obtain a first intermediate representation; Using the second text representation and the sample image representation, through the multimodal fusion network, to obtain a second intermediate representation; The first intermediate representation and the second intermediate representation are fused to obtain the first multimodal representation.

5. The method of claim 1, wherein: The scoring network includes: at least one scoring sub-network, the at least one scoring sub-network includes: a first scoring sub-network corresponding to a first image-text pair quality dimension; the sample image includes a sample image under the first image-text pair quality dimension, and the label data includes a label value corresponding to the sample image under the first image-text pair quality dimension; The obtaining of the prediction score comprises: Using the first multimodal representation, through the first scoring subnetwork, a first prediction subscore is obtained, and the first prediction subscore is included in the prediction score; The step of adjusting the scoring model comprises: The first scoring subnetwork is adjusted using a difference between the first prediction sub-score and a label value corresponding to the sample image under the first image-text pair quality dimension.

6. A method for scoring a picture-text pair, comprising: Acquire a target image-text pair to be scored, wherein the target image-text pair includes a third image description in a first language and a fourth image description in a second language corresponding to the third image description and a target image; Using the third image description, through a first language encoder of a target scoring model, to obtain a third text representation; Using the fourth image description, through the second language encoder of the target scoring model, to obtain a fourth text representation; Using the target image, through the image encoder of the target scoring model, to obtain a target image representation; Using the third text representation, the fourth text representation and the target image representation, a target multimodal representation is obtained through a multimodal fusion network of the target scoring model; The target multimodal representation is used to obtain a target score through a scoring network of the target scoring model.

7. The method according to claim 6, wherein obtaining the target image-text pair to be scored comprises: Acquire the third image description and the target image; The fourth image description in the second language is obtained by using the third image description through a first language model to obtain the target image-text pair.

8. The method according to claim 6, wherein obtaining the target image-text pair to be scored comprises: Acquire the fourth image description and the target image; The third image description in the first language is obtained by using the fourth image description through a second language model to obtain the target image-text pair.

9. The method of claim 6, wherein: The obtaining of the target multimodal representation comprises: Using the third text representation and the target image representation, a third intermediate representation is obtained through a multimodal fusion network of the target scoring model; Using the fourth text representation and the target image representation, a fourth intermediate representation is obtained through a multimodal fusion network of the target scoring model; The third intermediate representation and the fourth intermediate representation are fused to obtain the target multimodal representation.

10. The method of claim 6, wherein: The scoring network of the target scoring model includes: at least one scoring sub-network, each scoring sub-network corresponds to a different image-text pair quality dimension; Obtaining the target score includes: By using the target multimodal representation and through each scoring sub-network, the target sub-score of each image-text pair under the quality dimension is obtained.

11. The method according to any one of claims 7 to 10, wherein: The target scoring model is a scoring model trained by the scoring model training method according to any one of claims 1 to 5.

12. A scoring model training device, the scoring model comprising: A first language encoder, an image encoder, a second language encoder, a multimodal fusion network and a scoring network, the device comprising: A first acquisition module is configured to acquire a sample image-text pair and acquire label data corresponding to the sample image-text pair, wherein the sample image-text pair includes a first image description in a first language, and a sample image corresponding to the first image description and a second image description in a second language; A first obtaining module configured to obtain a first text representation by using the first image description through the first language encoder; A second obtaining module is configured to obtain a second text representation by using the second image description through the second language encoder; A third obtaining module is configured to obtain a sample image representation by using the sample image through the image encoder; A fourth obtaining module is configured to obtain a first multimodal representation by using the first text representation, the second text representation and the sample image representation through the multimodal fusion network; a fifth obtaining module configured to obtain a prediction score by using the first multimodal representation through the scoring network; An adjustment module is configured to adjust the scoring model using the difference between the prediction score and the label data.

13. A device for scoring a picture-text pair, comprising: A second acquisition module is configured to acquire a target image-text pair to be scored, wherein the target image-text pair includes a third image description in a first language and a fourth image description in a second language corresponding to the third image description and a target image; a sixth obtaining module configured to obtain a third text representation by using the third image description through a first language encoder of a target scoring model; a seventh obtaining module configured to obtain a fourth text representation by using the fourth image description through a second language encoder of the target scoring model; an eighth obtaining module configured to obtain a target image representation by using the target image through an image encoder of the target scoring model; A ninth obtaining module, configured to obtain a target multimodal representation by using the third text representation, the fourth text representation and the target image representation through a multimodal fusion network of the target scoring model; The tenth obtaining module is configured to obtain a target score by utilizing the target multimodal representation through the scoring network of the target scoring model.

14. An electronic device comprising a memory and a processor, wherein: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 11 is implemented.

15. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 11.

Images