Scoring model training method and apparatus, image-text pair scoring method and apparatus, device, and medium

Abstract

Embodiments of the present disclosure provide a scoring model training method and apparatus, an image-text pair scoring method and apparatus, a device, and a medium. The method comprises: obtaining a sample image-text pair comprising a first image description of a first language, a sample image corresponding to the first image description, and a second image description of a second language, and label data of the sample image-text pair; using the first image description to obtain a first text representation by means of a first language encoder; using the second image description to obtain a second text representation by means of a second language encoder; using the sample image to obtain a sample image representation by means of an image encoder; using the first text representation, the second text representation, and the sample image representation to obtain a first multi-modal representation by means of a multi-modal fusion network; using the first multi-modal representation to obtain a prediction score by means of a scoring network; and using a difference between the prediction score and the label data to adjust a scoring model comprising the structure, so as to perform training to obtain a scoring model capable of performing quality evaluation on an image-text pair.

Description

Scoring model training, image-text scoring method, device, equipment and medium

[0001] This application claims priority to Chinese Patent Application No. 202311293797.4 filed on October 8, 2023, and the contents of the above-mentioned Chinese patent application disclosure are hereby incorporated by reference in their entirety as a part of this application. Technical Field

[0002] The present disclosure relates to a method, apparatus, device and medium for training a scoring model and scoring image-text pairs. Background Art

[0003] In the field of image generation, image descriptions (also known as "prompts") can be used to generate images corresponding to these descriptions using an image generation model based on content generation technology. In this process, the stronger the image generation capabilities of the image generation model, the better the quality of the generated images.

[0004] During the training of an image generation model, the quality of the image-text pairs used is crucial to improving the model's image generation capabilities. Accordingly, to improve the model's image generation capabilities, it can be trained using high-quality image-text pairs. Therefore, providing a method for evaluating the quality of image-text pairs is an urgent problem.

[0005] Summary of the Invention

[0006] One or more embodiments of the present disclosure provide a scoring model training method, wherein the scoring model includes: a first language encoder, an image encoder, a second language encoder, a multimodal fusion network, and a scoring network. The method includes:

[0007] Obtaining a sample image-text pair and label data corresponding to the sample image-text pair, wherein the sample image-text pair includes a first image description in a first language and a sample image corresponding to the first image description and a second image description in a second language;

[0008] Using the first image description, through the first language encoder, to obtain a first text representation;

[0009] Using the second image description, passing it through the second language encoder, to obtain a second text representation;

[0010] Using the sample image, through the image encoder, to obtain a sample image representation;

[0011] Obtaining a first multimodal representation by using the first text representation, the second text representation, and the sample image representation through the multimodal fusion network;

[0012] Using the first multimodal representation, through the scoring network, to obtain a prediction score;

[0013] The scoring model is adjusted using the difference between the predicted score and the labeled data.

[0014] In an optional embodiment, obtaining a sample image-text pair includes: obtaining the first image description and the sample image; utilizing the first image description and a first language model to obtain the second image description in the second language; and using the first image description, the sample image, and the second image description as the sample image-text pair. In this implementation, utilizing the first image description and a first language model to convert the first image description into a different language ensures a more accurate second image description in the second language with the same semantics as the first image description.

[0015] In an optional embodiment, obtaining a sample image-text pair includes: obtaining the second image description and the sample image; utilizing the second image description through a second language model to obtain the first image description in the first language; and using the second image description, the sample image, and the first image description as the sample image-text pair. In this implementation, utilizing the second image description through a second language model allows for language conversion of the second image description to ensure a more accurate first image description in the first language that has the same semantics as the second image description.

[0016] In an optional embodiment, obtaining the first multimodal representation includes: using the first text representation and the sample image representation to obtain a first intermediate representation through the multimodal fusion network; using the second text representation and the sample image representation to obtain a second intermediate representation through the multimodal fusion network; fusing the first intermediate representation and the second intermediate representation to obtain the first multimodal representation. In this implementation, the first intermediate representation is obtained by fusing the first text representation and the sample image representation, and the second intermediate representation is obtained by fusing the second text representation and the sample image representation; and the first intermediate representation and the second intermediate representation are further fused to obtain the first multimodal representation, so as to obtain a first multimodal representation that is a fusion of the image representation and the text representation of the image description in different languages ​​with the same semantics, so that the first multimodal representation can better and more accurately represent the quality of the image-text pair consisting of the image and the image description, thereby making the prediction score obtained by the scoring model using the first multimodal representation more accurate and more able to reflect human feedback.

[0017] In an optional embodiment, the scoring network includes: at least one scoring sub-network, the at least one scoring sub-network includes: a first scoring sub-network corresponding to the first image-text pair quality dimension; the sample image includes a sample image under the first image-text pair quality dimension, and the label data includes a label value corresponding to the sample image under the first image-text pair quality dimension; the obtaining of the predicted score includes: using the first multimodal representation, through the first scoring sub-network, obtaining a first prediction sub-score, and incorporating it into the predicted score; the adjusting of the scoring model includes: using the difference between the first prediction sub-score and the label value corresponding to the sample image under the first image-text pair quality dimension to adjust the first scoring sub-network. In this implementation, the scoring model includes at least one scoring sub-network, each of which corresponds to a different image-text pair quality dimension. Using the sample images under each image-text pair quality dimension and their corresponding label data, a scoring model for fine-grained, different-dimensional quality assessment of the image-text pair can be trained, thereby realizing the quality assessment of the image-text pair from different image-text pair quality dimensions.

[0018] One or more embodiments of the present disclosure provide a method for scoring image-text pairs, including:

[0019] Obtaining a target image-text pair to be scored, wherein the target image-text pair includes a third image description in a first language, a fourth image description in a second language corresponding to the third image description, and a target image;

[0020] Using the third image description, passing it through a first language encoder of a target scoring model to obtain a third text representation;

[0021] Using the fourth image description, passing it through a second language encoder of the target scoring model to obtain a fourth text representation;

[0022] Using the target image, through the image encoder of the target scoring model, to obtain a target image representation;

[0023] Using the third text representation, the fourth text representation, and the target image representation, a target multimodal representation is obtained through a multimodal fusion network of the target scoring model;

[0024] The target multimodal representation is used to obtain a target score through the scoring network of the target scoring model.

[0025] In an optional embodiment, obtaining the target image-text pair to be scored includes: obtaining the third image description and the target image; and using the third image description to obtain the fourth image description in the second language using a first language model to obtain the target image-text pair. The first language model ensures that the obtained fourth image description is more accurate and more consistent with the true semantics of the third image description, that is, the obtained fourth image description has the same semantics as the third image description.

[0026] In an optional embodiment, obtaining the target image-text pair to be scored includes: obtaining the fourth image description and the target image; and using the fourth image description to obtain the third image description in the first language through a second language model, thereby obtaining the target image-text pair. The second language model ensures that the obtained third image description is more accurate and more consistent with the actual semantics of the fourth image description, that is, the obtained third image description has the same semantics as the fourth image description.

[0027] In an optional embodiment, obtaining the target multimodal representation includes: using the third text representation and the target image representation to obtain a third intermediate representation through the multimodal fusion network of the target scoring model; using the fourth text representation and the target image representation to obtain a fourth intermediate representation through the multimodal fusion network of the target scoring model; fusing the third intermediate representation and the fourth intermediate representation to obtain the target multimodal representation, so as to obtain a target multimodal representation that is fused with the target image representation and the text representation of the image description in different languages ​​with the same semantics, so that the target multimodal representation can better and more accurately represent the quality of the image-text pair consisting of the image and the image description, thereby making the score obtained by the scoring model using the target multimodal representation more accurate and more able to reflect human feedback.

[0028] In an optional embodiment, the scoring network of the target scoring model includes: at least one scoring sub-network, each scoring sub-network corresponding to a different image-text pair quality dimension; and obtaining the target score includes: utilizing the target multimodal representation to obtain the target sub-score under each image-text pair quality dimension through each scoring sub-network. In this implementation, the scoring model includes at least one scoring sub-network, each corresponding to a different image-text pair quality dimension, which can evaluate the quality of the image-text pair from different image-text pair quality dimensions, and obtain more fine-grained, different-dimensional scores for the image-text pair.

[0029] In an optional embodiment, the target scoring model is a scoring model trained based on the scoring model training method provided in one or more of the above embodiments, so that the scoring model can perform relatively accurate quality assessment on the image description in the first language and its corresponding image, and can perform relatively accurate quality assessment on the image description in the second language and its corresponding image, and the accuracy of the obtained score is higher.

[0030] One or more embodiments of the present disclosure provide a scoring model training device, wherein the scoring model includes: a first language encoder, an image encoder, a second language encoder, a multimodal fusion network, and a scoring network. The device includes:

[0031] A first acquisition module is configured to acquire a sample image-text pair and acquire label data corresponding to the sample image-text pair, wherein the sample image-text pair includes a first image description in a first language and a sample image corresponding to the first image description and a second image description in a second language;

[0032] a first obtaining module configured to obtain a first text representation by using the first image description through the first language encoder;

[0033] a second obtaining module configured to obtain a second text representation by using the second image description through the second language encoder;

[0034] a third obtaining module configured to obtain a sample image representation by using the sample image through the image encoder;

[0035] a fourth obtaining module configured to obtain a first multimodal representation by using the first text representation, the second text representation, and the sample image representation through the multimodal fusion network;

[0036] a fifth obtaining module configured to obtain a prediction score using the first multimodal representation through the scoring network;

[0037] An adjustment module is configured to adjust the scoring model using the difference between the prediction score and the label data.

[0038] One or more embodiments of the present disclosure provide a device for scoring image-text pairs, including:

[0039] A second acquisition module is configured to acquire a target image-text pair to be scored, wherein the target image-text pair includes a third image description in a first language, a fourth image description in a second language corresponding to the third image description, and a target image;

[0040] a sixth obtaining module configured to obtain a third text representation by using the third image description through a first language encoder of a target scoring model;

[0041] a seventh obtaining module configured to obtain a fourth text representation by using the fourth image description through a second language encoder of the target scoring model;

[0042] an eighth obtaining module configured to obtain a target image representation by using the target image through an image encoder of the target scoring model;

[0043] a ninth obtaining module configured to obtain a target multimodal representation by using the third text representation, the fourth text representation, and the target image representation through a multimodal fusion network of the target scoring model;

[0044] The tenth obtaining module is configured to obtain a target score by using the target multimodal representation through the scoring network of the target scoring model.

[0045] An embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute the method provided in one or more of the above embodiments.

[0046] An embodiment of the present disclosure provides an electronic device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method provided by one or more of the above embodiments is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] To more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0048] FIG1 is a schematic diagram of an implementation framework of an embodiment disclosed in the present disclosure;

[0049] FIG2 is a schematic diagram of a flow chart of a scoring model training method provided in an embodiment;

[0050] FIG3 is a flow chart of a method for scoring image-text pairs provided by an embodiment;

[0051] FIG4 is a schematic block diagram of a scoring model training device provided in an embodiment;

[0052] FIG5 is a schematic block diagram of an apparatus for scoring image-text pairs provided in an embodiment; and

[0053] FIG6 is a schematic block diagram of an electronic device provided by an embodiment. DETAILED DESCRIPTION

[0054] The technical solutions of the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0055] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0056] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0057] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0058] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0059] The present disclosure discloses a method, apparatus, device, and medium for training a scoring model and scoring image-text pairs. The following first introduces the application scenarios and technical concepts of the method, specifically as follows:

[0060] As previously mentioned, when training an image generation model, the quality of the image-text pairs used significantly improves the model's image generation capabilities. Accordingly, to improve the model's image generation capabilities, it can be trained using high-quality image-text pairs. Therefore, providing a method for evaluating the quality of image-text pairs is a pressing issue.

[0061] In view of this, the embodiments of the present disclosure provide a method, apparatus, device and medium for training a scoring model and scoring image-text pairs, so as to train a scoring model that can accurately evaluate the quality of image-text pairs, and the scoring model can accurately evaluate the quality of image descriptions of different language types and image-text pairs composed of their images.

[0062] FIG1 shows a schematic diagram of an implementation scenario according to an embodiment disclosed in the present disclosure. In the implementation scenario, a schematic structure of a scoring model and an exemplary training process of the scoring model are shown. The scoring model includes: a first language encoder, an image encoder, a second language encoder, a multimodal fusion network, and a scoring network. The training process may include: obtaining sample image-text pairs and their corresponding label data, wherein the sample image-text pairs include a first image description in a first language, and a sample image corresponding to the first image description and a second image description in a second language, the first language is different from the second language, and the first image description in the first language and the second image description in the second language represent the same semantics.

[0063] Using the first image description, a first text representation corresponding to the first image description is obtained through a first language encoder; using the second image description, a second text representation corresponding to the second image description is obtained through a second language encoder; using the sample image, a sample image representation corresponding to the sample image is obtained through an image encoder. In one implementation, the first language encoder is a pretrained encoder that can encode the image description in the first language to obtain the text representation corresponding to the image description in the first language. The second language encoder is a pretrained encoder that can encode the image description in the second language to obtain the text representation corresponding to the image description in the second language. The image encoder is a pretrained encoder that can encode the image to obtain the image representation corresponding to the image.

[0064] After obtaining the first text representation, the second text representation, and the sample image representation, the first, second, and sample image representations are then fed into a multimodal fusion network to generate a first multimodal representation. This first multimodal representation fuses the first, second, and sample image representations. The first multimodal representation is then fed into a scoring network to generate a predicted score. The difference between the predicted score and the labeled data is then used to adjust the scoring model.

[0065] In the above process, the first language encoder and the second language encoder of the scoring model can be used to respectively perform text encoding on image descriptions of different language types with the same semantics to obtain respective text representations of image descriptions of different language types with the same semantics. Then, the respective text representations of image descriptions of different language types with the same semantics are used to fuse with the sample image representation to obtain a first multimodal representation. The first multimodal representation contains the features of image descriptions of different language types, which can better represent the true meaning of the image description and further better reflect the quality of the corresponding image-text pairs. The first multimodal representation is used to obtain a prediction score, and then the difference between the prediction score and the label data is used to adjust the scoring model to better improve the accuracy of the scoring model's evaluation results on the image-text pair quality, and the scoring model can perform relatively accurate quality evaluation on image-text pairs consisting of image descriptions of different language types, i.e., the first language and the second language, and their images.

[0066] The following describes in detail the scoring model training and the method, device, equipment, and medium for scoring image-text pairs provided by the present disclosure in conjunction with specific embodiments.

[0067] FIG2 shows a flow chart of a scoring model training method according to an embodiment of the present disclosure. The method can be implemented by a target program. The target program can be installed in an electronic device, and the electronic device can be implemented by any device, equipment, platform, device cluster, etc. having computing and processing capabilities. The scoring model includes: a first language encoder, an image encoder, a second language encoder, a multimodal fusion network, and a scoring network. During the training process, as shown in FIG2 , the method can include the following steps S210-S270:

[0068] In step S210 , a sample image-text pair is obtained, and label data corresponding to the sample image-text pair is obtained. The sample image-text pair includes a first image description in a first language, a sample image corresponding to the first image description, and a second image description in a second language.

[0069] In one implementation, to train a scoring model, a training dataset for training the scoring model can be first obtained. The training dataset includes a number of image-text pairs and their corresponding label data. Each image-text pair includes an image description in a first language, an image corresponding to the image description in the first language, and a second image description in a second language. The image-text pair can be referred to as a "sample image-text pair," and the image therein can be referred to as a "sample image." It is understood that the first language and the second language can be different types of languages. For example, the first language can be English, and the second language can be Chinese, Russian, or another language. For another example, the first language can be Chinese, and the second language can be English or another language.

[0070] The image description in the first language and the image description in the second language included in a sample image pair have the same semantics and represent the same meaning.

[0071] In one case, the embodiment of the present disclosure may also provide a process for obtaining the aforementioned training data set, wherein the process may include: the target program collects image descriptions in the first language and image descriptions in the second language; using each of the collected image descriptions in the first language and the second language, through the trained image generation model, obtains at least one generated image corresponding to each image description, and then determines the sample image and label data corresponding to each image description based on each image description and its corresponding at least one generated image. The image generation model is used to generate the corresponding image for the input image description. The following uses any image description Ti in the image description in the first language and the image description in the second language as an example to illustrate the process of determining the sample image and label data corresponding to the image description. i is a positive integer.

[0072] Specifically, the image description Ti and its corresponding at least one generated image are annotated using a specific program or manual annotation. In one embodiment, the image description Ti and its corresponding at least one generated image can be annotated based on at least one preset image-text pair quality dimension to obtain a sample image corresponding to the image description Ti and corresponding label data. The preset image pair quality dimension can be any dimension that can characterize the quality of an image-text pair.

[0073] In one implementation, the image pair quality dimensions may include, but are not limited to, image clarity, image aesthetics, and image-text matching. The image pair quality dimensions may also include a comprehensive quality dimension of at least two of these dimensions. The image pair quality dimensions can be configured based on user needs.

[0074] When an image description Ti corresponds to a generated image, the generated image can be labeled as a positive sample image or a negative sample image under at least one image-text pair quality dimension, or the labeling can represent the quality score corresponding to the generated image under the image-text pair quality dimension, so as to obtain label data corresponding to the image description Ti; and the generated image can be used as the sample image corresponding to the image description Ti.

[0075] Accordingly, in one case, the label data corresponding to the image description Ti may include a label value representing whether the sample image corresponding to the image description Ti is a positive sample image or a negative sample image under at least one image-text pair quality dimension, for example, including a first label value representing that the sample image corresponding to the image description Ti is a positive sample image under image-text pair quality dimension 1, including a first label value representing that the sample image corresponding to the image description Ti is a positive sample image under image-text pair quality dimension 2, and including a second label value representing that the sample image corresponding to the image description Ti is a negative sample image under image-text pair quality dimension 3. In another case, the label data corresponding to the image description Ti may also include the quality scores of the sample images corresponding to the image description Ti under at least one image-text pair quality dimension.

[0076] Taking the quality dimension as the image clarity dimension as an example, the generated image is labeled as a positive sample image or a negative sample image under the image clarity dimension based on the clarity of the generated image. When the clarity of the generated image is high (for example, not lower than the first clarity threshold), the generated image can be labeled as a positive sample image under the image clarity dimension, for example, a first label value is labeled for the generated image, and the first label value is, for example, 1; when the clarity of the generated image is low (for example, lower than the second clarity threshold, the second clarity threshold is not greater than the first clarity threshold), the generated image can be labeled as a negative sample image under the image clarity dimension, for example, a second label value is labeled for the generated image, and the second label value is, for example, 0, to obtain the label data corresponding to the image description Ti. It is also possible to label the clarity score corresponding to the generated image based on the clarity of the generated image as the quality score of the generated image under the image clarity dimension to obtain the label data corresponding to the image description Ti, wherein the higher the clarity of the image, the larger the corresponding clarity score.

[0077] In the case where the image description Ti corresponds to at least two generated images, the generated images corresponding to the image description Ti as positive samples and / or generated images as negative samples in each image-text pair quality dimension can be screened out from the at least two generated images in at least one image-text pair quality dimension, and label data corresponding to the image description Ti can be obtained. The screened generated images as positive samples and / or generated images as negative samples in each image-text pair quality dimension can be used as sample images corresponding to the image description Ti, wherein the generated images as positive samples can be called positive sample images, and the generated images as negative samples can be called negative sample images.

[0078] For example, the generated images corresponding to the image description Ti include generated image 1, generated image 2, generated image 3, and generated image 4. The at least one image-text pair quality dimension includes an image clarity dimension, an image aesthetics dimension, an image-text matching dimension, and a comprehensive quality dimension of the three dimensions of image clarity dimension, image aesthetics dimension, and image-text matching dimension, which is referred to as the first comprehensive quality dimension.

[0079] Specifically, for example, with respect to the image clarity dimension, from generated image 1, generated image 2, generated image 3, and generated image 4 corresponding to the image description Ti, the generated images corresponding to the image description Ti are screened out as positive samples and / or generated images as negative samples in the image clarity dimension. For example, if generated image 1 has the best clarity, for example, the clarity score of generated image 1 is the largest among the four images, generated image 1 can be used as the positive sample image corresponding to the image description Ti in the image clarity dimension. If generated image 2 has the worst clarity, for example, the clarity score of generated image 2 is the smallest among the four images, generated image 2 can be used as the negative sample image corresponding to the image description Ti in the image clarity dimension.

[0080] Correspondingly, the label data corresponding to the image description Ti may include: label data corresponding to the image clarity dimension, for example, including a first label value representing that the generated image 1 is a positive sample image corresponding to the image description Ti in the image clarity dimension, and a second label value representing that the generated image 2 is a negative sample image corresponding to the image description Ti in the image clarity dimension.

[0081] For example, for the image aesthetic dimension, from generated image 1, generated image 2, generated image 3 and generated image 4 corresponding to the image description Ti, the generated images corresponding to the image description Ti in the image aesthetic dimension as positive samples and generated images as negative samples are screened out. For example, generated image 4 is the most beautiful, for example, the aesthetic score of generated image 4 is the largest among the four images, and generated image 4 can be used as the positive sample image corresponding to the image description Ti in the image aesthetic dimension. Generated image 3 is the least beautiful, for example, the aesthetic score of generated image 3 is the smallest among the four images, and generated image 3 can be used as the negative sample image corresponding to the image description Ti in the image aesthetic dimension.

[0082] Correspondingly, the label data corresponding to the image description Ti may include: label data corresponding to the image aesthetic dimension, for example, including a first label value representing that the generated image 4 is a positive sample image corresponding to the image description Ti under the image aesthetic dimension, and a second label value representing that the generated image 3 is a negative sample image corresponding to the image description Ti under the image aesthetic dimension.

[0083] For example, with respect to the image-text matching dimension, from generated image 1, generated image 2, generated image 3 and generated image 4 corresponding to the image description Ti, the generated images corresponding to the image description Ti as positive samples and generated images as negative samples in the image-text matching dimension are screened out. For example, if the generated image 4 is the best match with the image description Ti, for example, the matching value between the generated image 4 and the image description Ti is the largest matching value between the four images and the image description Ti respectively, the generated image 4 is used as the positive sample image corresponding to the image description Ti in the image-text matching dimension. If the generated image 2 is the least match with the image description Ti, for example, the matching value between the generated image 2 and the image description Ti is the smallest matching value between the four images and the image description Ti respectively, the generated image 2 is used as the negative sample image corresponding to the image description Ti in the image-text matching dimension.

[0084] Correspondingly, the label data corresponding to the image description Ti may include: label data corresponding to the image-text matching dimension, for example, including a first label value representing that the generated image 4 is a positive sample image corresponding to the image description Ti under the image-text matching dimension, and a second label value representing that the generated image 2 is a negative sample image corresponding to the image description Ti under the image-text matching dimension.

[0085] For another example, for the comprehensive quality dimension of the three dimensions of image clarity, image aesthetics, and image-text matching, which is called the first comprehensive quality dimension, from generated image 1, generated image 2, generated image 3, and generated image 4 corresponding to the image description Ti, the generated images as positive samples and generated images as negative samples under the first comprehensive quality dimension corresponding to the image description Ti are screened out. For example, the comprehensive quality of generated image 4 and image description Ti is the best, that is, the overall quality of generated image 4 and image description Ti is the best in the three dimensions of image clarity, image aesthetics, and image-text matching. The generated image 4 is used as the positive sample image corresponding to the image description Ti under the first comprehensive quality dimension. The comprehensive quality of generated image 2 and image description Ti is the worst, that is, the overall quality of generated image 2 and image description Ti is the worst in the three dimensions of image clarity, image aesthetics, and image-text matching. The generated image 2 is used as the negative sample image corresponding to the image description Ti under the first comprehensive quality dimension.

[0086] Correspondingly, the label data corresponding to the image description Ti may include: the label data corresponding to the aforementioned first comprehensive quality dimension, for example, including a first label value representing that the generated image 4 is a positive sample image under the first comprehensive quality dimension corresponding to the image description Ti, and a second label value representing that the generated image 2 is a negative sample image under the first comprehensive quality dimension corresponding to the image description Ti.

[0087] Through the above-mentioned method, the sample images and label data corresponding to any image description Ti in each first language image description and each second language image description can be obtained. The sample images corresponding to the image description Ti may include at least one sample image under the image-text pair quality dimension, and specifically may include positive sample images and / or negative sample images under each image-text pair quality dimension. Accordingly, the label data corresponding to the image description Ti may include: label data corresponding to at least one image-text pair quality dimension, and the label data corresponding to each image-text pair quality dimension include: the label value corresponding to the sample image corresponding to the image description Ti under the image-text pair quality dimension, for example, including the first label value corresponding to the positive sample image under each image-text pair quality dimension, and / or the second label value corresponding to the negative sample image under each image-text pair quality dimension. The first label value is different from the second label value.

[0088] It can be understood that, between different image-text pair quality dimensions, the positive sample images corresponding to the image description Ti may be the same or different, and the negative sample images corresponding to the image description Ti may be the same or different.

[0089] Afterwards, for the image description in the first language, the image description in the first language can be translated using a designated translator to obtain an image description in the second language corresponding to the image description in the first language; then, the image description in the first language, the image description in the second language obtained by translating the image description in the first language, and the sample image corresponding to the description in the first language are combined to obtain a sample image-text pair corresponding to the image description in the first language.

[0090] In one implementation, there is one sample image-text pair corresponding to the image description in the first language, which may include the image description in the first language, the image description in the second language translated from the image description in the first language, and all sample images corresponding to the image description in the first language, that is, sample images under the quality dimension of each image-text pair corresponding to the image description in the first language. Subsequently, the label data corresponding to the image description in the first language is used as the label data corresponding to the sample image-text pair containing the image description in the first language.

[0091] In another implementation, there may be multiple sample image-text pairs corresponding to the image description in the first language, and each sample image-text pair corresponds to one of the aforementioned image-text pair quality dimensions. Accordingly, each sample image-text pair may include: an image description in the first language, an image description in the second language obtained by translating the image description in the first language, and a sample image corresponding to the image description in the first language, and a sample image under the image-text pair quality dimension corresponding to the sample image-text pair. Subsequently, based on the label data corresponding to the image description in the first language, the label data corresponding to each of the multiple sample image-text pairs corresponding to the image description in the first language is determined.

[0092] Continuing with the above example: for example, the image description Ti is any image description in the first language, and the image description Ti can correspond to four sample image-text pairs, for example, including: a sample image-text pair corresponding to the image clarity dimension, including: the image description Ti and its corresponding translated image description in another language, as well as the generated image 1 as a positive sample image and / or the generated image 2 as a negative sample image under the image clarity dimension; a sample image-text pair corresponding to the image aesthetics dimension, including: the image description Ti and its corresponding translated image description in another language, as well as the generated image 4 as a positive sample image and / or the generated image 3 as a negative sample image under the image aesthetics dimension; a sample image-text pair corresponding to the image-text matching dimension, including: the image description Ti and its corresponding translated image description in another language, as well as the generated image 4 as a positive sample image and / or the generated image 2 as a negative sample image under the image-text matching dimension; a sample image-text pair corresponding to the first comprehensive quality dimension, including: the image description Ti and its corresponding translated image description in another language, as well as the generated image 4 as a positive sample image and / or the generated image 2 as a negative sample image under the first comprehensive quality dimension.

[0093] Correspondingly, for example, the label data corresponding to the sample image-text pair corresponding to the image clarity dimension includes: a first label value representing generated image 1 as a positive sample image in the image clarity dimension, and / or a second label value representing generated image 2 as a negative sample image in the image clarity dimension. The label data corresponding to the sample image-text pair corresponding to the image aesthetic dimension includes: a first label value representing generated image 4 as a positive sample image in the image clarity dimension, and / or a second label value representing generated image 3 as a negative sample image in the image clarity dimension.

[0094] In one embodiment, in order to ensure the accuracy of the image description in the first language and the image description in the second language in the sample image-text pair, it is necessary to ensure that the image description in the first language and the image description in the second language have closer or identical semantics. In order to ensure that the semantics of the image description in the second language obtained by translation of the image description in the first language are more accurate, that is, to ensure that the image description in the second language has the same semantics as the image description in the first language, specifically, in step S210, the process of obtaining the sample image-text pair may include steps 11-13.

[0095] In step 11, a first image description and a sample image are obtained. The sample image may include a positive sample image and / or a negative sample image under at least one image-text pair quality dimension. The first image description is an image description in a first language.

[0096] Then, in step 12, the first image description is used to obtain a second image description in a second language using a first language model. In this step, to achieve higher semantic accuracy for the image description in the second language obtained by translating the first image description, and to ensure that the semantics of the translated image description in the second language are closer to or identical to those of the first image description, the first image description is translated using the first language model to obtain a second image description in the second language corresponding to the first image description. The first language model is a natural language processing model that has been pre-trained on a large amount of corpus. This first language model is used to translate corpus in the first language into corpus in the second language that is semantically closer to or identical to the first language. For example, if the first language is Chinese, the first image description is in Chinese, and if the second language is English, the first language model can translate Chinese into English. For example, if the first image description is "donkey meat hot pot," the first language model can obtain a second image description in the second language as "baked cake of donkey meat."

[0097] Next, in step 13, the first image description, the sample image, and the second image description are used as sample image-text pairs. In this step, the target program uses the first image description in the first language and its corresponding sample image and the second image description in the second language as sample image-text pairs to obtain the training data required to train the scoring model.

[0098] For a second-language image description, the second-language image description can be translated using a designated translator to obtain a first-language image description corresponding to the second-language image description. The second-language image description, the first-language image description obtained by translating the second-language image description, and a sample image corresponding to the second-language image description are then combined to obtain a sample image-text pair corresponding to the second-language image description. The process of obtaining the sample image-text pair corresponding to the second-language image description can be referenced to the aforementioned process of obtaining the sample image-text pair corresponding to the first-language image description and is not further described here.

[0099] In one embodiment, in order to ensure the accuracy of the image description in the first language and the image description in the second language in the sample image-text pair, it is necessary to ensure that the image description in the first language and the image description in the second language have closer or identical semantics. In order to ensure that the semantics of the image description in the first language obtained by translation are more accurate for the image description in the second language, that is, it is ensured that the image description in the first language and the image description in the second language have the same semantics. Specifically, in step S210, the process of obtaining the sample image-text pair may include steps 21-23.

[0100] In step 21, a second image description and a sample image are obtained. The sample image may include a positive sample image and / or a negative sample image under at least one image-text pair quality dimension. The second image description is an image description in a second language.

[0101] Then, in step 22, the second image description is used to obtain a first image description in the first language using a second language model. In this step, in order to increase the semantic accuracy of the first language image description obtained by translating the second image description into a more semantically accurate first language image description, and to ensure that the semantics of the translated first language image description are closer to or identical to the semantics of the second image description, the second image description is translated using the second language model to obtain the first image description in the first language corresponding to the second image description. The second language model is a natural language processing-based model that has been pre-trained with a large amount of corpus. The second language model is used to translate the second language corpus into a first language corpus that is more semantically similar to or identical to the second language corpus.

[0102] Next, in step 23, the second image description, the sample image, and the first image description are used as sample image-text pairs. In this step, the target program uses the second image description in the second language and its corresponding sample image and the first image description in the first language as sample image-text pairs to obtain the training data required to train the scoring model.

[0103] Through the above method, for any collected image description in the first language, its corresponding sample image and image description in the second language and label data can be obtained to obtain the corresponding sample image-text pair and its corresponding label data; and for any collected image description in the second language, its corresponding sample image and image description in the first language and label data can be obtained to obtain the corresponding sample image-text pair and its corresponding label data, thereby obtaining a training data set including several sample image-text pairs and their corresponding label data.

[0104] The target program can obtain any sample image-text pair from the training dataset and the corresponding label data for the sample image-text pair. It then performs the subsequent scoring model training process based on the obtained sample image-text pair and its corresponding label data. Subsequently, the image description in the first language of the sample image-text pair is referred to as the first image description, and the image description in the second language of the sample image-text pair is referred to as the second image description.

[0105] In step S220, the first image description is used to obtain a first text representation through a first language encoder. In this step, the target program can input the first image description into the first language encoder, and encode the first image description through the first language encoder to obtain a first text representation corresponding to the first image description. The first language encoder is an encoder that encodes image descriptions in the first language, and the first language encoder is a pre-trained encoder. In one implementation, the first language encoder can be implemented by a text encoder in a CLIP (Contrastive Language-Image Pre-Training) model.

[0106] Then, in step S230, the second image description is used to obtain a second text representation through a second language encoder. In this step, the target program can input the second image description into the second language encoder, and encode the second image description through the second language encoder to obtain a second text representation corresponding to the second image description. The second language encoder is an encoder that encodes image descriptions in a second language, and the second language encoder is a pre-trained encoder. In one implementation, the second language encoder can be implemented by a text encoder in a CLIP (Contrastive Language-Image Pre-Training) model.

[0107] Next, in step S240, the sample image is used to obtain a sample image representation through an image encoder. In this step, the target program can input the sample image into the image encoder, which encodes the sample image to obtain a sample image representation corresponding to the sample image. The image encoder is a pre-trained encoder. In one implementation, the image encoder can be implemented using the image encoder in the CLIP (Contrastive Language-Image Pre-Training) model.

[0108] After obtaining the first text representation corresponding to the first image description, the second text representation corresponding to the second image description, and the sample image representation corresponding to the sample image, in step S250, the first text representation, the second text representation, and the sample image representation are used to obtain a first multimodal representation through a multimodal fusion network. In this step, the target program inputs the first text representation, the second text representation, and the sample image representation into the multimodal fusion network, and fuses the first text representation, the second text representation, and the sample image representation through the multimodal fusion network to obtain a first multimodal representation. The first multimodal representation is a fusion of the first text representation, the second text representation, and the sample image representation.

[0109] In one embodiment, step S250 may include the following steps 31-33.

[0110] In step 31, the first text representation and the sample image representation are used to obtain a first intermediate representation through a multimodal fusion network. In this step, the target program may input the first text representation and the sample image representation into the multimodal fusion network, and fuse the first text representation and the sample image representation through the multimodal fusion network to obtain the first intermediate representation. In one implementation, the process of fusing the first text representation and the sample image representation through the multimodal fusion network may include: fusing the first text representation and the sample image representation through the multimodal fusion network using a dot product operation.

[0111] Next, in step 32, the second text representation and the sample image representation are used to obtain a second intermediate representation through a multimodal fusion network. In this step, the target program can input the second text representation and the sample image representation into the multimodal fusion network, and fuse the second text representation and the sample image representation through the multimodal fusion network to obtain a second intermediate representation. In one implementation, the process of fusing the second text representation and the sample image representation through the multimodal fusion network can include: fusing the second text representation and the sample image representation through the multimodal fusion network using a dot product operation.

[0112] Then, in step 33, the first intermediate representation and the second intermediate representation are fused to obtain a first multimodal representation. In this step, the target program can use any specified fusion method to fuse the first intermediate representation and the second intermediate representation to obtain the first multimodal representation, for example, by adding the first intermediate representation and the second intermediate representation to fuse the first intermediate representation and the second intermediate representation to obtain the first multimodal representation.

[0113] In the above process, the first multimodal representation obtained is a fusion of the sample image representation and the text representation of the image description in different languages ​​with the same semantics, so that the first multimodal representation can better and more accurately represent the quality of the image-text pair consisting of the image and the image description, thereby making the prediction score obtained by the scoring model using the first multimodal representation more accurate and more able to reflect human feedback.

[0114] After obtaining the first multimodal representation, in step S260, the first multimodal representation is used to obtain a prediction score through a scoring network. In this step, the target program inputs the first multimodal representation into the scoring network, processes the first multimodal representation through the scoring network, and obtains a prediction score corresponding to the sample image-text pair. The prediction score can characterize the quality of the sample image-text pair, for example, it can characterize the quality of the sample image-text pair under the aforementioned image-text pair quality dimension. In one case, the larger the prediction score, the better the quality of the sample image-text pair evaluated by the scoring model. The value range of the prediction score is [0, 1]. In one implementation, the scoring network can be implemented by MLP (multi-layer perceptron).

[0115] After obtaining the predicted score, in step S270, the scoring model is adjusted using the difference between the predicted score and the label data. In this step, the target program can use a specified loss function to adjust the scoring model using the difference between the predicted score and the label data. The specified loss function may include but is not limited to: a cross entropy loss function and an MSE (mean square error) loss function, etc. In one implementation, the target program can use a specified loss function to construct a prediction loss corresponding to the scoring model using the difference between the predicted score and the label data, wherein the prediction loss is positively correlated with the difference between the predicted score and the label data. The scoring model is then adjusted with the goal of minimizing the prediction loss, that is, minimizing the difference between the predicted score and the label data.

[0116] In one implementation, adjusting the scoring model may include adjusting parameters of a scoring network. In another implementation, when a first language encoder, an image encoder, a second language encoder, and a multimodal fusion network are provided for which adjustment of the scoring model is required, adjusting the scoring model may include adjusting parameters of the first language encoder, the image encoder, the second language encoder, the multimodal fusion network, and the scoring network.

[0117] In one embodiment, in order to achieve a more fine-grained quality assessment of image-text pairs, the scoring network may include: at least one scoring sub-network, wherein each scoring sub-network corresponds to a different image-text pair quality dimension. For example, the scoring network may include, but is not limited to, the aforementioned scoring sub-network corresponding to the image clarity dimension, the scoring sub-network corresponding to the image aesthetics dimension, and the scoring sub-network corresponding to the image-text matching dimension. The scoring sub-network corresponding to the image clarity dimension is used to perform quality assessment of image-text pairs under the image clarity dimension, the scoring sub-network corresponding to the image aesthetics dimension is used to perform quality assessment of image-text pairs under the image aesthetics dimension, and the scoring sub-network corresponding to the image-text matching dimension is used to perform quality assessment of image-text pairs under the image matching dimension.

[0118] In addition, the scoring network may also include other dimensions for evaluating the quality of image-text pairs, for example, it may also include scoring sub-networks corresponding to comprehensive quality dimensions of at least two of the image clarity dimension, the image aesthetics dimension, and the image-text matching dimension. This type of scoring sub-network can perform comprehensive quality evaluation of image-text pairs in at least two of the image clarity dimension, the image aesthetics dimension, and the image-text matching dimension.

[0119] In one embodiment, the number and corresponding dimensions of the scoring sub-networks included in the scoring network can be set according to user needs. The label data corresponding to the sample image-text pair is related to the corresponding dimensions of the scoring sub-networks included in the scoring network. In one implementation, each scoring sub-network can be implemented using an MLP (Multi-Layer Perceptron).

[0120] In one implementation, the label data includes label values ​​corresponding to the sample image under each image-text pair quality dimension. Step S260 may include: utilizing the first multimodal representation to obtain prediction sub-scores corresponding to each scoring sub-network through each scoring sub-network, and incorporating the predicted sub-scores into the prediction score. Then, step S270 utilizes the prediction sub-scores of each scoring sub-network and the label values ​​corresponding to the sample image under each image-text pair quality dimension corresponding to each scoring sub-network in the label data to adjust each scoring sub-network.

[0121] In another implementation, each scoring sub-network corresponds to a different image-text pair quality dimension. Each image-text pair quality dimension can correspond to a corresponding sample image-text pair. The target program can use the sample image-text pairs corresponding to the different image-text pair quality dimensions and their corresponding label data to adjust and train the corresponding scoring sub-network. It is understood that the data processing and corresponding adjustment processes for different scoring sub-networks are similar. The data processing and corresponding adjustment processes for any first scoring sub-network in at least one scoring sub-network are used as an example to illustrate. Specifically, the at least one scoring sub-network includes: a first scoring sub-network corresponding to the first image-text pair quality dimension, and the first image-text pair quality dimension can be any dimension of the at least one image-text pair quality dimension mentioned above; the sample image in the sample image-text pair includes the sample image under the first image-text pair quality dimension, and the label data corresponding to the sample image-text pair includes the label value corresponding to the sample image under the first image-text pair quality dimension; accordingly, in step S260, it includes: using the first multimodal representation, through the first scoring sub-network, obtaining a first prediction sub-score, and classifying it into the prediction score; in this step, the target program inputs the first multimodal representation into the first scoring sub-network, obtains the first prediction sub-score of the sample image-text pair under the first image-text pair quality dimension, and classifies it into the prediction score, that is, obtains the prediction score.

[0122] Next, step S270 includes adjusting the first scoring sub-network using the difference between the first prediction sub-score and the label value corresponding to the sample image under the first image-text pair quality dimension. In this step, a specified loss function can be used to determine a first prediction sub-loss using the difference between the first prediction sub-score and the label value corresponding to the sample image under the first image-text pair quality dimension, and include it in the aforementioned prediction loss; the first scoring sub-network is adjusted with the goal of minimizing the prediction loss, i.e., minimizing the first prediction sub-loss, i.e., minimizing the difference between the first prediction sub-score and the label value corresponding to the sample image under the first image-text pair quality dimension.

[0123] In one implementation, when setting a first language encoder, an image encoder, a second language encoder, and a multimodal fusion network that require adjustment of the scoring model, the first scoring sub-network, the first language encoder, the image encoder, the second language encoder, and the multimodal fusion network can be adjusted by utilizing the difference between the first prediction sub-score and the label value corresponding to the sample image under the first image-text pair quality dimension.

[0124] It is understood that each sample image-text pair in the training sample set may include a first image description in a first language and its corresponding second image description in a second language, as well as sample images under at least one of the aforementioned image-text pair quality dimensions. Using these sample image-text pairs and the label data corresponding to these sample image-text pairs, the scoring sub-networks corresponding to each image-pair quality dimension in the scoring model can be trained and adjusted to improve the accuracy of the scores output by each scoring sub-network. Each scoring sub-network is used to evaluate the quality of the image-text pair under each image-text pair quality dimension.

[0125] In one embodiment, the sample image may include a positive sample image and / or a negative sample image, and the label data includes a first label value corresponding to the positive sample image and / or a second label value corresponding to the negative sample image; accordingly, in step S240, the process of obtaining the sample image representation includes: using the positive sample image to obtain the positive sample image representation through an image encoder; and / or using the negative sample image to obtain the negative sample image representation through an image encoder, wherein the specific implementation process can refer to the aforementioned process of obtaining the sample image representation.

[0126] Then, in step S250, the process of obtaining the first multimodal representation includes: using the first text representation, the second text representation and the positive sample image representation to obtain the positive sample multimodal representation through a multimodal fusion network; and / or using the first text representation, the second text representation and the negative sample image representation to obtain the negative sample multimodal representation through a multimodal fusion network; wherein, the specific implementation process can refer to the aforementioned process of obtaining the first multimodal representation.

[0127] Then, in step S260, the process of obtaining the prediction score includes: using the multimodal representation of the positive sample to obtain a first prediction score through a scoring network; and / or using the multimodal representation of the negative sample to obtain a second prediction score through a scoring network; wherein, the specific implementation process can refer to the aforementioned process of obtaining the prediction score.

[0128] Finally, in step S270, the process of adjusting the scoring model includes adjusting the scoring model using the difference between the first prediction score and the first label value, and / or the difference between the second prediction score and the second label value. In this step, the scoring model is adjusted, for example, by adjusting the parameters of the scoring network, with the goal of reducing the difference between the first prediction score and the first label value, and / or reducing the difference between the second prediction score and the second label value.

[0129] This allows the scoring model to distinguish the quality of different sample images corresponding to the first image description in the first language and its corresponding second image description in the second language, so as to better ensure the accuracy of the prediction results of the scoring model, that is, the predicted scores for the image-text pairs.

[0130] Let's review the execution process of the aforementioned steps S210-S270. The above embodiment is explained using a sample image-text pair as an example. In another embodiment, the above steps S210-S260 can also be performed for a batch of samples, that is, multiple sample image-text pairs, to obtain the corresponding prediction score for each sample image-text pair. Then, in step S270, the scoring model is adjusted based on the difference between the corresponding prediction score of each sample image-text pair and the label data corresponding to the sample image-text pair. In this embodiment, the scoring model is adjusted for a batch of samples. This can reduce the number of adjustments to the scoring model and make it easier to implement the training process.

[0131] Adjust the scoring model until the scoring model reaches a preset convergence condition, thereby obtaining a trained scoring model. The preset convergence condition may include, but is not limited to, the number of parameter adjustments of the scoring model exceeding a preset threshold, or the predicted loss of the determined scoring model being lower than a preset loss threshold. In one implementation, the scoring model reaching the preset convergence condition may include: each scoring sub-network reaching the preset convergence condition, for example: the number of adjustments of each scoring sub-network exceeds a preset threshold, or the loss determined for each scoring sub-network is lower than a preset loss threshold.

[0132] In this embodiment, the first language encoder and the second language encoder of the scoring model can be used to respectively perform text encoding on image descriptions of different language types with the same semantics to obtain text representations of the image descriptions of different language types with the same semantics. The text representations of the image descriptions of different language types with the same semantics are then fused with the sample image representation to obtain a first multimodal representation. The first multimodal representation includes the characteristics of the image descriptions of different language types, which can better represent the true meaning of the image description and better reflect the quality of the image-text pair. The first multimodal representation is used to obtain a prediction score, and then the difference between the prediction score and the label data is used to adjust the scoring model to better improve the accuracy of the scoring model's evaluation results on the image-text pair quality. The scoring model can also perform relatively accurate quality evaluation on image descriptions of different language types (i.e., the first language and the second language) and image-text pairs composed of the images.

[0133] Corresponding to the above method embodiment, the embodiment of the present disclosure also provides a method for scoring image-text pairs. The method can be implemented by a first program. The first program can be the same physical program as the aforementioned target program, or a different physical program from the aforementioned target program. As shown in Figure 3, the method can include the following steps S310-S360.

[0134] It is understood that the image-text pair scoring method can be a method for scoring image-text pairs based on a target scoring model, which can include a first language encoder, an image encoder, a second language encoder, a multimodal fusion network, and a scoring network. In one embodiment, the target scoring model can be a scoring model trained using the scoring model training method provided in the aforementioned embodiment, that is, the trained scoring model in the aforementioned embodiment.

[0135] In step S310, a target image-text pair to be rated is obtained, wherein the target image-text pair includes a third image description in a first language, a fourth image description in a second language corresponding to the third image description, and a target image. The target image-text pair can be any image-text pair that needs to be rated.

[0136] In one implementation, in order to better improve the quality of user service, the user can input any image description in the first language and its corresponding image, or an image description in the second language and its corresponding image, and the first program provides translation services for the image description to obtain a target image-text pair to be scored. Specifically, in one embodiment, in step S310, the following steps 41-42 may be included: in step 41, a third image description and a target image are obtained. In step 42, the third image description is used to obtain a fourth image description in the second language through the first language model to obtain a target image-text pair. The process of obtaining the target image-text pair can refer to the aforementioned process of obtaining a sample image-text pair, and will not be described in detail here.

[0137] In yet another embodiment, step S310 may include the following steps 51-52: In step 51, a fourth image description and a target image are obtained. In step 52, the fourth image description is used to obtain a third image description in the first language using a second language model to obtain a target image-text pair. The process of obtaining the target image-text pair can be referred to the process of obtaining the sample image-text pair described above and will not be further described here.

[0138] After obtaining the target image-text pair, in step S320, the third image description is used to generate a third text representation through the first language encoder of the target scoring model. The process of obtaining the third text representation can be referred to the process of obtaining the first text representation, and will not be repeated here.

[0139] Next, in step S330, the fourth image description is passed through the second language encoder of the target scoring model to obtain a fourth text representation. The process of obtaining the fourth text representation can be referred to the process of obtaining the fourth text representation described above and will not be described in detail here.

[0140] Next, in step S340, the target image is used to obtain a target image representation through the image encoder of the target scoring model. The process of obtaining the target image representation can refer to the process of obtaining the sample image representation described above and will not be repeated here.

[0141] Then, in step S350 , the third text representation, the fourth text representation, and the target image representation are used to obtain a target multimodal representation through a multimodal fusion network of a target scoring model.

[0142] Specifically, in one embodiment, step S350 may include the following steps 61-63:

[0143] In step 61, the third text representation and the target image representation are used to obtain a third intermediate representation through the multimodal fusion network of the target scoring model. The process of obtaining the third intermediate representation can be referred to the process of obtaining the first intermediate representation, and will not be repeated here.

[0144] In step 62, the fourth text representation and the target image representation are used to generate a fourth intermediate representation through the multimodal fusion network of the target scoring model. The process of obtaining the fourth intermediate representation can be referred to the process of obtaining the second intermediate representation, and will not be repeated here.

[0145] In step 63, the third intermediate representation and the fourth intermediate representation are fused to obtain the target multimodal representation. The process of obtaining the target multimodal representation can be referred to the process of obtaining the first multimodal representation, and will not be repeated here.

[0146] After obtaining the target multimodal representation, in step S360, the target multimodal representation is used to obtain a target score through the scoring network of the target scoring model. In this step, the first program inputs the target multimodal representation into the scoring network of the target scoring model, and processes the target multimodal representation through the scoring network of the target scoring model to obtain a target score that can represent the quality of the target image-text pair.

[0147] In one embodiment, the scoring network of the target scoring model may include: at least one scoring sub-network, each scoring sub-network corresponds to a different image-text pair quality dimension mentioned in the aforementioned embodiment, and each scoring sub-network can evaluate the quality of the target image-text pair under the image-text pair quality dimension corresponding to the scoring sub-network; in step S360, it may include: using the target multimodal representation to obtain the target sub-score under each image-text pair quality dimension through each scoring sub-network. The target sub-score under each image-text pair quality dimension can represent the quality of the target image-text pair evaluated by the target scoring model under the corresponding image-text pair quality dimension. The value range of the target sub-score can be set to [0, 1], where the larger the target sub-score, the better the quality of the target image-text pair under the image-text pair quality dimension corresponding to the target sub-score.

[0148] In the above process, the target scoring model can be used to obtain a more accurate target score that better characterizes the quality of the target image-text pair. Furthermore, if the scoring network of the target scoring model includes multiple scoring sub-networks, the target scoring model can be used to obtain a more fine-grained target score that characterizes the quality of the target image-text pair across different image-text pair quality dimensions, providing a more intuitive representation of the quality of the target image-text pair.

[0149] The foregoing describes specific embodiments of the present disclosure, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in an order different from that described in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily need to be performed in the specific order shown or in a sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0150] Corresponding to the above method embodiment, an embodiment of the present disclosure provides a scoring model training device 400, whose schematic block diagram is shown in FIG4 , for executing the method shown in FIG2 . The scoring model includes: a first language encoder, an image encoder, a second language encoder, a multimodal fusion network, and a scoring network. The device 400 includes:

[0151] A first acquisition module 410 is configured to acquire a sample image-text pair and acquire label data corresponding to the sample image-text pair, wherein the sample image-text pair includes a first image description in a first language and a sample image corresponding to the first image description and a second image description in a second language;

[0152] A first obtaining module 420 is configured to obtain a first text representation by using the first image description through the first language encoder;

[0153] A second obtaining module 430 is configured to obtain a second text representation by using the second image description through the second language encoder;

[0154] A third obtaining module 440 is configured to obtain a sample image representation by using the sample image through the image encoder;

[0155] a fourth obtaining module 450 configured to obtain a first multimodal representation by using the first text representation, the second text representation, and the sample image representation through the multimodal fusion network;

[0156] a fifth obtaining module 460 configured to obtain a prediction score using the first multimodal representation through the scoring network;

[0157] The adjustment module 470 is configured to adjust the scoring model using the difference between the prediction score and the label data.

[0158] In an optional implementation, the first acquisition module 410 is specifically configured to acquire the first image description and the sample image;

[0159] Obtaining the second image description in the second language using the first image description through a first language model;

[0160] The first image description, the sample image, and the second image description are used as the sample image-text pair.

[0161] In an optional implementation, the first acquisition module 410 is specifically configured to acquire the second image description and the sample image;

[0162] Using the second image description and a second language model, obtain the first image description in the first language;

[0163] The second image description, the sample image, and the first image description are used as the sample image-text pair.

[0164] In an optional implementation, the fourth obtaining module 450 is specifically configured to obtain a first intermediate representation by using the first text representation and the sample image representation through the multimodal fusion network;

[0165] Using the second text representation and the sample image representation, through the multimodal fusion network, to obtain a second intermediate representation;

[0166] The first intermediate representation and the second intermediate representation are fused to obtain the first multimodal representation.

[0167] In an optional embodiment, the scoring network includes: at least one scoring sub-network, the at least one scoring sub-network including: a first scoring sub-network corresponding to a first image-text pair quality dimension; the sample image includes a sample image under the first image-text pair quality dimension, and the label data includes a label value corresponding to the sample image under the first image-text pair quality dimension;

[0168] The fifth obtaining module 460 is specifically configured to use the first multimodal representation to obtain a first prediction sub-score through the first scoring sub-network, and include the first prediction sub-score in the prediction score;

[0169] The adjustment module 470 is specifically configured to adjust the first scoring sub-network using the difference between the first prediction sub-score and the label value corresponding to the sample image under the first image-text pair quality dimension.

[0170] Corresponding to the above method embodiment, an embodiment of the present disclosure provides an apparatus 500 for scoring image-text pairs, a schematic block diagram of which is shown in FIG5 , and is used to execute the method shown in FIG3 . The apparatus 500 includes:

[0171] The second acquisition module 510 is configured to acquire a target image-text pair to be rated, wherein the target image-text pair includes a third image description in a first language, a fourth image description in a second language corresponding to the third image description, and a target image;

[0172] A sixth obtaining module 520 is configured to obtain a third text representation by using the third image description through a first language encoder of a target scoring model;

[0173] a seventh obtaining module 530 configured to obtain a fourth text representation by using the fourth image description through a second language encoder of the target scoring model;

[0174] an eighth obtaining module 540 configured to obtain a target image representation by using the target image through an image encoder of the target scoring model;

[0175] a ninth obtaining module 550 configured to obtain a target multimodal representation by using the third text representation, the fourth text representation, and the target image representation through a multimodal fusion network of the target scoring model;

[0176] The tenth obtaining module 560 is configured to obtain a target score by using the target multimodal representation through the scoring network of the target scoring model.

[0177] In an optional implementation, the second acquisition module 510 is specifically configured to acquire the third image description and the target image;

[0178] The fourth image description in the second language is obtained by using the third image description through a first language model to obtain the target image-text pair.

[0179] In an optional implementation, the second acquisition module 510 is specifically configured to acquire the fourth image description and the target image;

[0180] The fourth image description is used to obtain the third image description in the first language through a second language model to obtain the target image-text pair.

[0181] In an optional implementation, the eighth obtaining module 540 is specifically configured to obtain a third intermediate representation by using the third text representation and the target image representation through a multimodal fusion network of the target scoring model;

[0182] Using the fourth text representation and the target image representation, a fourth intermediate representation is obtained through a multimodal fusion network of the target scoring model;

[0183] The third intermediate representation and the fourth intermediate representation are fused to obtain the target multimodal representation.

[0184] In an optional embodiment, the scoring network of the target scoring model includes: at least one scoring sub-network, each scoring sub-network corresponding to a different image-text pair quality dimension;

[0185] The tenth obtaining module 560 is specifically configured to utilize the target multimodal representation to obtain the target sub-score of each image-text pair quality dimension through each scoring sub-network.

[0186] In an optional implementation, the target scoring model is a scoring model trained by the scoring model training device described in the aforementioned embodiment.

[0187] The above-mentioned device embodiments correspond to the method embodiments. For detailed descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For detailed descriptions, please refer to the corresponding method embodiments.

[0188] An embodiment of the present disclosure provides an electronic device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method shown in FIG2 or FIG3 is implemented. Referring now to FIG6 , a schematic diagram of the structure of an electronic device suitable for implementing an embodiment of the present disclosure is shown. The electronic device shown in FIG6 is merely an example and should not limit the functionality or scope of use of the embodiments of the present disclosure.

[0189] As shown in Figure 6, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0190] Typically, the following devices can be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although FIG6 shows an electronic device with various devices, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have alternatively. Each box shown in FIG6 may represent one device, or may represent multiple devices as needed.

[0191] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0192] An embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute the method shown in FIG. 2 or FIG. 3 provided by the present disclosure.

[0193] It should be noted that the computer-readable medium described in the embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or device. In the embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or convey a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code embodied on the computer-readable medium may be conveyed using any suitable medium, including but not limited to wires, optical cables, RF (Radio Frequency), or any suitable combination thereof.

[0194] The computer-readable medium may be included in the electronic device, or may exist independently and not be incorporated into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: obtain a sample image-text pair and obtain label data corresponding to the sample image-text pair, wherein the sample image-text pair includes a first image description in a first language, a sample image corresponding to the first image description, and a second image description in a second language; obtain a first text representation using the first image description through a first language encoder of a scoring model; obtain a second text representation using the second image description through a second language encoder of the scoring model; obtain a sample image representation using the sample image through an image encoder of the scoring model; obtain a first multimodal representation using the first text representation, the second text representation, and the sample image representation through a multimodal fusion network of the scoring model; obtain a prediction score using the first multimodal representation through a scoring network of the scoring model; and adjust the scoring model using the difference between the prediction score and the label data. Alternatively, the electronic device is configured to: obtain a target image-text pair to be scored, wherein the target image-text pair includes a third image description in a first language and a fourth image description in a second language corresponding to the third image description and a target image; utilize the third image description to obtain a third text representation through a first language encoder of a target scoring model; utilize the fourth image description to obtain a fourth text representation through a second language encoder of the target scoring model; utilize the target image to obtain a target image representation through an image encoder of the target scoring model; utilize the third text representation, the fourth text representation and the target image representation to obtain a target multimodal representation through a multimodal fusion network of the target scoring model; utilize the target multimodal representation to obtain a target score through a scoring network of the target scoring model.

[0195] Computer program code for performing the operations of the disclosed embodiments may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0196] The various embodiments of this disclosure are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences between the other embodiments. In particular, the storage medium and computing device embodiments are described briefly because they are generally similar to the method embodiments. For relevant portions, reference can be made to the description of the method embodiments.

[0197] Those skilled in the art will appreciate that, in one or more of the above examples, the functions described in the embodiments of the present disclosure may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0198] The specific implementation methods described above further illustrate the purpose, technical solutions, and beneficial effects of the embodiments of the present disclosure. It should be understood that the above description is only a specific implementation method of the embodiments of the present disclosure and is not intended to limit the scope of protection of the present disclosure. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the present disclosure shall be included in the scope of protection of the present disclosure.

Claims

1. A scoring model training method, wherein: The scoring model includes a first language encoder, an image encoder, a second language encoder, a multimodal fusion network and a scoring network, and the method includes: Acquire a sample image-text pair, and acquire label data corresponding to the sample image-text pair, wherein the sample image-text pair includes a first image description in a first language, and a sample image corresponding to the first image description and a second image description in a second language; Using the first image description, through the first language encoder, to obtain a first text representation; Using the second image description, through the second language encoder, to obtain a second text representation; Using the sample image, through the image encoder, to obtain a sample image representation; Using the first text representation, the second text representation and the sample image representation, through the multimodal fusion network, a first multimodal representation is obtained; Using the first multimodal representation, through the scoring network, to obtain a prediction score; and The scoring model is adjusted using the difference between the predicted score and the labeled data.

2. The method according to claim 1, wherein obtaining sample image-text pairs comprises: Obtaining the first image description and the sample image; Using the first image description and a first language model, obtain the second image description in the second language; as well as The first image description, the sample image and the second image description are used as the sample image-text pair.

3. The method according to claim 1, wherein obtaining sample image-text pairs comprises: Acquire the second image description and the sample image; Using the second image description and a second language model, obtain the first image description in the first language; as well as The second image description, the sample image and the first image description are used as the sample image-text pair.

4. The method according to any one of claims 1 to 3, wherein: The obtaining of the first multimodal representation comprises: Using the first text representation and the sample image representation, through the multimodal fusion network, to obtain a first intermediate representation; Using the second text representation and the sample image representation, through the multimodal fusion network, to obtain a second intermediate representation; and The first intermediate representation and the second intermediate representation are fused to obtain the first multimodal representation.

5. The method according to any one of claims 1 to 4, wherein: The scoring network includes at least one scoring sub-network, and the at least one scoring sub-network includes a first scoring sub-network corresponding to a first image-text pair quality dimension; the sample image includes a sample image under the first image-text pair quality dimension, and the label data includes a label value corresponding to the sample image under the first image-text pair quality dimension; The obtaining of the prediction score comprises: Using the first multimodal representation, through the first scoring subnetwork, to obtain a first prediction subscore, and including the first prediction subscore in the prediction score; and The adjusting the scoring model comprises: The first scoring subnetwork is adjusted using a difference between the first prediction sub-score and a label value corresponding to the sample image under the first image-text pair quality dimension.

6. A method for scoring a picture-text pair, comprising: Acquire a target image-text pair to be scored, wherein the target image-text pair includes a third image description in a first language, and a fourth image description in a second language corresponding to the third image description and a target image; Using the third image description, through a first language encoder of a target scoring model, to obtain a third text representation; Using the fourth image description, through the second language encoder of the target scoring model, to obtain a fourth text representation; Using the target image, through the image encoder of the target scoring model, to obtain a target image representation; Using the third text representation, the fourth text representation and the target image representation, a target multimodal representation is obtained through a multimodal fusion network of the target scoring model; and The target multimodal representation is used to obtain a target score through a scoring network of the target scoring model.

7. The method according to claim 6, wherein obtaining the target image-text pair to be scored comprises: Acquire the third image description and the target image; as well as The fourth image description in the second language is obtained by using the third image description through a first language model to obtain the target image-text pair.

8. The method according to claim 6, wherein obtaining the target image-text pair to be scored comprises: Acquire the fourth image description and the target image; as well as The third image description in the first language is obtained by using the fourth image description through a second language model to obtain the target image-text pair.

9. The method according to any one of claims 6 to 8, wherein: The obtaining of the target multimodal representation comprises: Using the third text representation and the target image representation, a third intermediate representation is obtained through a multimodal fusion network of the target scoring model; Using the fourth text representation and the target image representation, a fourth intermediate representation is obtained through a multimodal fusion network of the target scoring model; and The third intermediate representation and the fourth intermediate representation are fused to obtain the target multimodal representation.

10. The method according to any one of claims 6 to 8, wherein: The scoring network of the target scoring model includes at least one scoring sub-network, each of which corresponds to a different image-text pair quality dimension; as well as Obtaining the target score includes: By using the target multimodal representation and through each of the scoring sub-networks, the target sub-scores under the quality dimension of each of the image-text pairs are obtained.

11. The method according to any one of claims 6 to 10, wherein: The target scoring model is a scoring model trained by the scoring model training method according to any one of claims 1 to 5.

12. A scoring model training device, wherein: The scoring model includes a first language encoder, an image encoder, a second language encoder, a multimodal fusion network and a scoring network, and the device includes: A first acquisition module is configured to acquire a sample image-text pair and acquire label data corresponding to the sample image-text pair, wherein the sample image-text pair includes a first image description in a first language and a sample image corresponding to the first image description and a second image description in a second language; A first obtaining module configured to obtain a first text representation by using the first image description through the first language encoder; A second obtaining module is configured to obtain a second text representation by using the second image description through the second language encoder; A third obtaining module is configured to obtain a sample image representation by using the sample image through the image encoder; A fourth obtaining module is configured to obtain a first multimodal representation by using the first text representation, the second text representation and the sample image representation through the multimodal fusion network; a fifth obtaining module configured to obtain a prediction score by using the first multimodal representation through the scoring network; and An adjustment module is configured to adjust the scoring model using the difference between the prediction score and the label data.

13. A device for scoring a picture-text pair, comprising: The second acquisition module is configured to acquire a target image-text pair to be scored, wherein the target image-text pair includes a first language a third image description, and a fourth image description in a second language and a target image corresponding to the third image description; a sixth obtaining module configured to obtain a third text representation by using the third image description through a first language encoder of a target scoring model; a seventh obtaining module configured to obtain a fourth text representation by using the fourth image description through a second language encoder of the target scoring model; an eighth obtaining module configured to obtain a target image representation by using the target image through an image encoder of the target scoring model; a ninth obtaining module configured to obtain a target multimodal representation by using the third text representation, the fourth text representation and the target image representation through a multimodal fusion network of the target scoring model; and The tenth obtaining module is configured to obtain a target score by utilizing the target multimodal representation through the scoring network of the target scoring model.

14. An electronic device comprising a memory and a processor, wherein: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 11 is implemented.

15. A computer-readable storage medium, wherein: The computer-readable storage medium stores a computer program, and when the computer program is executed in a computer, the computer is caused to execute the method according to any one of claims 1 to 11.

Images