Model training method, device and equipment in text map scene and medium

Abstract

The invention provides a model training method and device in a text graph scene, equipment and a medium, and belongs to the field of artificial intelligence. The method comprises the following steps: constructing a plurality of image-text sample pairs; performing data perturbation on the original text included in the plurality of image-text sample pairs to obtain an edited text; performing text feature extraction on the edited text and the original text included in the plurality of image-text sample pairs to obtain a first text feature of the edited text and a second text feature of the plurality of image-text sample pairs; performing image feature extraction on images included in the plurality of image-text sample pairs to obtain image features of the plurality of image-text sample pairs; and performing model training based on the first text feature, the second text feature and the image feature by taking learning of the correlation between the text and the image and the correlation between the texts as a target to obtain a model used for executing text coding in the text generation image scene. According to the method and the device, similar images can be generated based on semantically similar texts in the text graph task.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and in particular to a model training method, device, equipment and medium in a cultural graph scenario. Background Art

[0002] With the development of artificial intelligence (AI) technology, AI tools for text graphics have become a powerful assistant for creative people and the general public. This AI tool can not only break through the traditional creative boundaries, but also give text vivid visual expressions.

[0003] Wenshengtu, also known as AI painting, refers to the generation of creative paintings based on descriptive text input by users. In the Wenshengtu process, one of the core technologies used is CLIP (Contrastive Language-Image Pre-Training, a pre-training method or model based on contrastive text-image pairs). For example, in Wenshengtu large models such as DM (DiffusionModel), CLIP is usually used to extract text features from the input text.

[0004] Since the extracted text features will be used as conditional inputs of DM, the text encoder of CLIP controls the image generation process by providing text descriptions, so the performance of CLIP directly affects the quality of the generated images. Therefore, how to train CLIP to ensure the quality of the images generated during the text generation process has become a problem that technicians in this field need to solve. Summary of the invention

[0005] The present disclosure provides a model training method, device, equipment and medium in a cultural graph scenario. The technical solution of the present disclosure is as follows:

[0006] According to a first aspect of an embodiment of the present disclosure, a model training method in a text graph scenario is provided, comprising:

[0007] Constructing a plurality of image-text sample pairs, wherein one of the image-text sample pairs includes a piece of original text and an image;

[0008] Performing data perturbation on the original text included in the plurality of image-text sample pairs to obtain edited text;

[0009] Performing text feature extraction on the edited text to obtain a first text feature; and performing text feature extraction on the original text included in the plurality of image-text sample pairs to obtain a second text feature;

[0010] Extracting image features from the images included in the plurality of image-text sample pairs to obtain image features;

[0011] With the goal of learning the correlation between text and image and the correlation between text and text, model training is performed based on the first text feature, the second text feature and the image feature to obtain a model for performing text encoding in a text-image scenario.

[0012] In some embodiments, the performing of model training based on the first text feature, the second text feature, and the image feature to obtain a model for performing text encoding in a text-image scenario includes:

[0013] Based on the first text feature and the second text feature, obtaining a first loss; wherein the first loss is used to measure the correlation between the original text and the edited text;

[0014] Based on the second text feature and the image feature, obtaining a second loss; wherein the second loss is used to measure the correlation between the original text and the image;

[0015] The model parameters are adjusted based on the first loss and the second loss to obtain the model for performing text encoding.

[0016] In some other embodiments, obtaining the first loss based on the first text feature and the second text feature includes:

[0017] Obtaining the similarity between the first text feature and the second text feature to obtain a first similarity matrix; wherein the first similarity matrix includes first-category elements and second-category elements, the first-category elements are used to indicate the similarity between a pair of text samples with similar semantics, and the second-category elements are used to indicate the similarity between a pair of text samples with different semantics; a pair of text samples includes a section of original text and a section of edited text;

[0018] Obtaining the first loss based on the first label information and the similarity associated with each original text segment in the first similarity matrix; wherein the first label information is used to indicate the edited text that is semantically similar to each original text segment and the edited text that is semantically dissimilar to each original text segment;

[0019] Obtaining a second sub-loss based on the second label information and the similarity associated with each edited text in the first similarity matrix; wherein the second label information is used to indicate the original text that is semantically similar to each edited text and the original text that is semantically not similar to each edited text;

[0020] The first loss is obtained based on the first sub-loss and the second sub-loss.

[0021] In some other embodiments, the first text feature and the second text feature have the same dimension; and obtaining the similarity between the first text feature and the second text feature to obtain a first similarity matrix includes:

[0022] Normalizing the first text feature and the second text feature to obtain a normalized first text feature and a normalized second text feature;

[0023] Performing a dot product on the normalized first text feature and the normalized image feature to obtain the first similarity matrix.

[0024] In some other embodiments, obtaining the second loss based on the second text feature and the image feature includes:

[0025] Acquire the similarity between the second text feature and the image feature to obtain a second similarity matrix; wherein the second similarity matrix includes third-category elements and fourth-category elements, the third-category elements are used to indicate the similarity between the original text and the image that are correctly paired, and the fourth-category elements are used to indicate the similarity between the original text and the image that are incorrectly paired;

[0026] Obtaining a third sub-loss based on the third label information and the similarity associated with each image in the second similarity matrix; wherein the third label information is used to indicate the original text correctly paired with each image and the original text incorrectly paired with each image;

[0027] Obtaining a fourth sub-loss based on fourth label information and the similarity associated with each original text segment in the second similarity matrix; wherein the fourth label information is used to indicate images correctly paired with each original text segment and images incorrectly paired with each original text segment;

[0028] The second loss is obtained based on the third sub-loss and the fourth sub-loss.

[0029] In some other embodiments, the dimensions of the second text feature and the image feature are different; and obtaining the similarity between the second text feature and the image feature to obtain a second similarity matrix includes:

[0030] Aligning the second text feature and the image feature in size to obtain the second text feature and the image feature having the same dimension;

[0031] Normalizing the second text feature and the image feature having the same dimension to obtain a normalized second text feature and a normalized image feature;

[0032] Perform dot product on the normalized second text feature and the normalized image feature to obtain the second similarity matrix.

[0033] In some other embodiments, performing data perturbation on the original text included in the plurality of image-text sample pairs to obtain edited text includes:

[0034] For any original text, delete part of the original text to obtain the edited text corresponding to the original text; or

[0035] For any section of original text, part of the content of the original text is masked to obtain the edited text corresponding to the original text.

[0036] In some other embodiments, deleting part of the original text to obtain the edited text corresponding to the original text includes:

[0037] Splitting the original text into a plurality of basic units for text processing;

[0038] Some basic units are deleted from the multiple basic units obtained by splitting to obtain the edited text, and the edited text includes the basic units that are not deleted.

[0039] In some other embodiments, deleting some basic units from the multiple basic units obtained by splitting includes:

[0040] A preset proportion of basic units is randomly deleted from the plurality of basic units.

[0041] In some other embodiments, masking part of the original text to obtain the edited text corresponding to the original text includes:

[0042] Splitting the original text into a plurality of basic units for text processing;

[0043] Some basic units are masked among the multiple basic units obtained by splitting to obtain the edited text, wherein the edited text includes the basic units that are not masked.

[0044] In some other embodiments, constructing a plurality of image-text sample pairs includes:

[0045] Acquire a sample data set, wherein the sample data set includes a first number of images and texts matching the first number of images;

[0046] Constructing the plurality of image-text sample pairs based on the sample data set;

[0047] The multiple image-text sample pairs include a first number of positive image-text sample pairs and a second number of negative image-text sample pairs; the positive image-text sample pairs include correctly paired original text and images; and the negative image-text sample pairs include incorrectly paired original text and images.

[0048] According to a second aspect of an embodiment of the present disclosure, a model training device in a Wenshengtu scenario is provided, comprising:

[0049] An acquisition module is configured to construct a plurality of image-text sample pairs, wherein one of the image-text sample pairs includes a section of original text and an image;

[0050] An editing module, configured to perform data perturbation on the original text included in the plurality of image-text sample pairs to obtain an edited text;

[0051] A feature extraction module is configured to extract text features from the edited text to obtain a first text feature; and to extract text features from the original text included in the plurality of image-text sample pairs to obtain a second text feature;

[0052] The feature extraction module is further configured to extract image features from the images included in the plurality of image-text sample pairs to obtain image features;

[0053] The training module is configured to learn the correlation between text and image and the correlation between text and text, and to perform model training based on the first text feature, the second text feature and the image feature to obtain a model for performing text encoding in a text-image scenario.

[0054] In some embodiments, the training module is configured to:

[0055] Based on the first text feature and the second text feature, obtaining a first loss; wherein the first loss is used to measure the correlation between the original text and the edited text;

[0056] Based on the second text feature and the image feature, obtaining a second loss; wherein the second loss is used to measure the correlation between the original text and the image;

[0057] The model parameters are adjusted based on the first loss and the second loss to obtain the model for performing text encoding.

[0058] In some other embodiments, the training module is configured to:

[0059] Obtaining the similarity between the first text feature and the second text feature to obtain a first similarity matrix; wherein the first similarity matrix includes first-category elements and second-category elements, the first-category elements are used to indicate the similarity between a pair of text samples with similar semantics, and the second-category elements are used to indicate the similarity between a pair of text samples with different semantics; a pair of text samples includes a section of original text and a section of edited text;

[0060] Obtaining the first loss based on the first label information and the similarity associated with each original text segment in the first similarity matrix; wherein the first label information is used to indicate the edited text that is semantically similar to each original text segment and the edited text that is semantically dissimilar to each original text segment;

[0061] Obtaining a second sub-loss based on the second label information and the similarity associated with each edited text in the first similarity matrix; wherein the second label information is used to indicate the original text that is semantically similar to each edited text and the original text that is semantically not similar to each edited text;

[0062] The first loss is obtained based on the first sub-loss and the second sub-loss.

[0063] In some other embodiments, the first text feature and the second text feature have the same dimension; and the training module is configured to:

[0064] Normalizing the first text feature and the second text feature to obtain a normalized first text feature and a normalized second text feature;

[0065] Performing a dot product on the normalized first text feature and the normalized image feature to obtain the first similarity matrix.

[0066] In some other embodiments, the training module is configured to:

[0067] Acquire the similarity between the second text feature and the image feature to obtain a second similarity matrix; wherein the second similarity matrix includes third-category elements and fourth-category elements, the third-category elements are used to indicate the similarity between the original text and the image that are correctly paired, and the fourth-category elements are used to indicate the similarity between the original text and the image that are incorrectly paired;

[0068] Obtaining a third sub-loss based on the third label information and the similarity associated with each image in the second similarity matrix; wherein the third label information is used to indicate the original text correctly paired with each image and the original text incorrectly paired with each image;

[0069] Obtaining a fourth sub-loss based on fourth label information and the similarity associated with each original text segment in the second similarity matrix; wherein the fourth label information is used to indicate images correctly paired with each original text segment and images incorrectly paired with each original text segment;

[0070] The second loss is obtained based on the third sub-loss and the fourth sub-loss.

[0071] In some other embodiments, the dimension of the second text feature is different from that of the image feature; and the training module is configured to:

[0072] Aligning the second text feature and the image feature in size to obtain the second text feature and the image feature having the same dimension;

[0073] Normalizing the second text feature and the image feature having the same dimension to obtain a normalized second text feature and a normalized image feature;

[0074] Perform dot product on the normalized second text feature and the normalized image feature to obtain the second similarity matrix.

[0075] In some other embodiments, the editing module is configured to:

[0076] For any original text, delete part of the original text to obtain the edited text corresponding to the original text; or

[0077] For any section of original text, part of the content of the original text is masked to obtain the edited text corresponding to the original text.

[0078] In some other embodiments, the editing module is configured to:

[0079] Splitting the original text into a plurality of basic units for text processing;

[0080] Some basic units are deleted from the multiple basic units obtained by splitting to obtain the edited text, and the edited text includes the basic units that are not deleted.

[0081] In some other embodiments, the editing module is configured to:

[0082] A preset proportion of basic units is randomly deleted from the plurality of basic units.

[0083] In some other embodiments, the editing module is configured to:

[0084] Splitting the original text into a plurality of basic units for text processing;

[0085] Some basic units are masked among the multiple basic units obtained by splitting to obtain the edited text, wherein the edited text includes the basic units that are not masked.

[0086] In some other embodiments, the acquisition module is configured to:

[0087] Acquire a sample data set, wherein the sample data set includes a first number of images and texts matching the first number of images;

[0088] Constructing the plurality of image-text sample pairs based on the sample data set;

[0089] The multiple image-text sample pairs include a first number of positive image-text sample pairs and a second number of negative image-text sample pairs; the positive image-text sample pairs include correctly paired original text and images; and the negative image-text sample pairs include incorrectly paired original text and images.

[0090] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device comprising:

[0091] one or more processors;

[0092] A memory for storing program codes executable by the processor;

[0093] Among them, the processor is configured to execute the program code to implement the model training method in the above-mentioned Wenshengtu scenario.

[0094] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, and when a program code in the computer-readable storage medium is executed by a processor of an electronic device, the electronic device is enabled to execute the model training method in the above-mentioned Wenshengtu scenario.

[0095] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which implements the above-mentioned model training method in the Wensheng diagram scenario when executed by a processor of an electronic device.

[0096] For the text-based image scenario, the model training scheme provided by the disclosed embodiment optimizes the training process of CLIP and introduces text-to-text comparative learning into the training process of CLIP. This enables CLIP to learn not only the correlation between images and texts, but also the correlation between texts, significantly improving the text comprehension ability of CLIP. Under the supervision of text-to-text comparative learning, it ensures that the text features corresponding to semantically similar texts are similar, thereby ensuring that the images finally generated in the text-based image task are also similar. That is, similar images can be generated based on semantically similar texts. In summary, this scheme makes the images generated in the text-based image task more accurate and more in line with user needs, thereby improving the human-computer interaction effect.

[0097] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0099] Figure 1 It is a schematic diagram of an implementation environment of a model training method in a cultural graph scenario according to an exemplary embodiment.

[0100] Figure 2 It is a flowchart of a model training method in a text graph scenario according to an exemplary embodiment.

[0101] Figure 3 It is a schematic diagram of a model training process in a cultural graph scenario according to an exemplary embodiment.

[0102] Figure 4 It is a flowchart of another model training method in a text graph scenario according to an exemplary embodiment.

[0103] Figure 5 It is a block diagram of a model training device in a cultural graph scenario according to an exemplary embodiment.

[0104] Figure 6 It is a block diagram of a server according to an exemplary embodiment. DETAILED DESCRIPTION

[0105] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0106] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0107] The information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0108] Some nouns or abbreviations of the embodiments of the present disclosure are first introduced below.

[0109] CLIP: A pre-training method or model based on contrastive text-image pairs. In other words, CLIP is a multimodal pre-training model based on contrastive learning, and the training data of CLIP is text-image pairs. Among them, a text-image pair includes an image and its corresponding text description. Through contrastive learning, the model can learn the matching relationship between text-image pairs. That is, for a pair of text and image input to the model, CLIP uses a text encoder to encode the input text into a text vector, and uses an image encoder to encode the input image into an image vector. For a set of input data (a pair of text and image), the set of data that is correctly paired is called a positive sample, and the set of data that is incorrectly paired is called a negative sample. In addition, when training the model, the optimization goal is to make the distance between the text and image included in the positive sample as close as possible, and the distance between the text and image included in the negative sample as far as possible. This optimization method is called contrastive learning.

[0110] Figure 1 It is a schematic diagram of an implementation environment of a model training method in a cultural graph scenario according to an exemplary embodiment.

[0111] See also Figure 1 , the implementation environment includes: a terminal 101 and a server 102. The terminal 101 is an electronic device used by a user. In the embodiment of the present disclosure, a target application is installed on the terminal 101. The target application can be an instant messaging application, such as a short video application with a text-generating image function, or an application dedicated to image processing, which is not limited in the present disclosure.

[0112] In some embodiments, the terminal 101 is a device such as a smart phone, a desktop computer, or a laptop computer. Figure 1 The terminal 101 is only used as a smartphone for illustration. In addition, those skilled in the art will appreciate that the number of the above terminals may be more or less. For example, the above terminals may be only a few, or the above devices may be dozens or hundreds, or more. The embodiments of the present disclosure do not limit the number and type of terminals.

[0113] In other embodiments, the server 102 provides background services for the target application. In addition, the server 102 is connected to the terminal 101 via a wireless network or a wired network. In addition, the server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server, which is not limited in the present disclosure. In addition, the server involved in the embodiment of the present disclosure may also include other functional servers to provide more comprehensive and diversified services. In addition, the server 102 is responsible for executing the model training method provided in the embodiment of the present disclosure.

[0114] The reason why the training process of CLIP needs to be optimized is that: currently CLIP mainly learns the correlation between the entity concept of the text and the subject of the image, and it is difficult to express the semantic relationship between similar texts. For example, the user inputs two texts that are very similar in semantics, but the text features extracted by CLIP are very different, which will cause the final generated images to be very different, deviating from user needs. For example, "a handsome guy" and "a handsome guy, very handsome", although these two sentences are very similar in semantics, when they are used in the diffusion model to generate two images, the generated images are very different. The fundamental reason for this situation is that CLIP only learns the correlation between text and image, but not the correlation between text and text.

[0115] To this end, for CLIP, this paper adds text-to-text contrastive learning on the basis of text-to-image contrastive learning, so that CLIP can not only learn the correlation between text and image, but also learn the correlation between text and text, in the hope that under the supervision of text-to-text contrastive learning, the text features corresponding to texts with similar semantics are also similar, thereby ensuring that the two images generated in the text-to-image task are also similar.

[0116] In other words, in order to solve the problem of insufficient text understanding ability of CLIP used for text encoding in the text-generated image task, the disclosed embodiment proposes a training process optimization scheme for CLIP. On the basis of maintaining the correlation between CLIP learning images and texts, the text understanding ability of CLIP is improved by learning the correlation between texts, so that similar images can be generated based on semantically similar texts in the text-generated image task.

[0117] In some embodiments, the similarity between images is used to measure the matching degree of the subjects contained in different images. In an image, the subject refers to the main object in the picture. It is the core of the composition, which can focus the audience's attention and is also an important embodiment of the theme of the picture. It can be a single object or a group of objects. It can be a specific object or an abstract concept, which is not limited in this disclosure.

[0118] The following example introduces the model training scheme in the context of the text map provided by the embodiment of the present disclosure.

[0119] Figure 2 is a flowchart of a model training method in a text graph scenario according to an exemplary embodiment. Figure 2 As shown, the model training method in the text image scenario is applied to electronic devices, such as Figure 1 In the server 102 shown. The model training method in the text-based graph scenario includes the following steps:

[0120] In 201, the electronic device constructs a plurality of image-text sample pairs, wherein one image-text sample pair includes a segment of original text and an image.

[0121] In the training task of CLIP, sample pairs of images and texts are constructed. Such sample pairs are also referred to as image-text sample pairs in this article. Among them, an image-text sample pair includes a piece of text and an image. In particular, the text included in the image-text sample pair is also referred to as the original text in this article.

[0122] In some embodiments, the disclosed embodiments construct image-text sample pairs for training CLIP in the following manner:

[0123] First, a sample data set is obtained. The sample data set includes a first number of images and texts matching the first number of images. Taking the value of the first number as N as an example, the sample data set includes N images and text descriptions corresponding to the N images, that is, N segments of original text.

[0124] Next, multiple image-text sample pairs are constructed based on the sample data set. Continuing with the example that the sample data set includes N images, N*N image-text sample pairs can be constructed based on the sample data set. Among them, the N*N image-text sample pairs include N positive image-text sample pairs and N 2 -N negative image-text sample pairs.

[0125] Exemplarily, when constructing image-text sample pairs, for the first image, it and each original text in the N original texts form an image-text sample pair, and then N image-text sample pairs are obtained. For the second image, it and each original text in the N original texts form an image-text sample pair, and N image-text sample pairs are obtained. Similarly, for the Nth image, it and each original text in the N original texts form an image-text sample pair, and N image-text sample pairs are obtained again. At this point, N*N image-text sample pairs are formed.

[0126] It should be noted that the positive image-text sample pair includes the original text and image that are correctly paired, and the negative image-text sample pair includes the original text and image that are incorrectly paired. Among them, the correct pairing means that the text and image included in the image-text sample pair are matched, that is, the text included in the image-text sample pair is used to describe the image included in the image-text sample pair. The incorrect pairing means that the text and image included in the image-text sample pair are not matched, that is, the text included in the image-text sample pair is not used to describe the image included in the image-text sample pair.

[0127] In summary, the disclosed embodiment constructs a variety of image-text sample pairs, so that during the training process, based on the constructed image-text sample pairs, CLIP can learn the correlation between text and image, so that the distance between the text and image included in the positive sample is as close as possible, and the distance between the text and image included in the negative sample is as far as possible. This ensures that when the trained CLIP is subsequently applied to the text-based image task, it can help the large text-based image model generate an image that is more consistent with the text description input by the user. In addition, based on the limited number of images collected and the corresponding text descriptions, the disclosed embodiment can construct a larger number of image-text sample pairs, enrich the number of samples, and thus ensure the accuracy of the model.

[0128] In 202, the electronic device performs data perturbation on the original text included in the plurality of image-text sample pairs to obtain edited text.

[0129] In the disclosed embodiment, the edited text is also referred to as Mask Tokens. Continuing with the above step 201 as an example, since the original text used to construct the above multiple image-text sample pairs is N segments in total, this step is used to perform data perturbation on the N segments of original text, thereby obtaining N segments of edited text.

[0130] Among them, a section of original text and a section of edited text constitute another type of sample pair, which is also referred to as a text sample pair in this article. Among them, the positive text sample pair includes a section of original text and a section of edited text that are correctly paired. In this text sample pair, the edited text is obtained by perturbing the original text. On the contrary, the negative text sample pair includes a section of original text and a section of edited text that are incorrectly paired. In this text sample pair, the edited text is not obtained by perturbing the original text.

[0131] In the embodiments of the present disclosure, the data disturbance methods include but are not limited to deletion, masking, etc., which are not limited in the present disclosure. Taking content masking of the original text as an example, the method provided in the embodiments of the present disclosure includes: for any section of the original text, part of the content of the section of the original text is deleted to obtain the edited text corresponding to the section of the original text.

[0132] Exemplarily, before content masking is performed on a section of text, the section of text is first split into multiple basic units for text processing by a tokenizer. The basic unit for text processing is called a token, and a token can be a character, a word, or a phrase, etc., which is not limited in the present disclosure. Afterwards, some basic units are masked among the multiple basic units obtained by the splitting to obtain an edited text, wherein the edited text includes the basic units that are not masked.

[0133] In addition, when masking the content of a text, a special mark may be selected for masking, which is not limited in the present disclosure. In addition, when masking the content, some basic units may be randomly selected for masking, such as randomly masking a preset proportion of basic units. The value of the preset proportion may be 25%, which is also not limited in the present disclosure.

[0134] Taking deleting the content of the original text as an example, the method provided by the embodiment of the present disclosure includes: for any section of the original text, deleting part of the content of the section of the original text to obtain the edited text corresponding to the section of the original text.

[0135] Similarly, before deleting content from a text, the text is first split into multiple basic units for text processing by a word segmenter. Afterwards, some basic units are deleted from the multiple basic units obtained by splitting to obtain an edited text, wherein the edited text includes the basic units that have not been deleted. In addition, when deleting content, some basic units can be randomly selected for deletion, such as randomly deleting a preset proportion of basic units, which is not limited in the present disclosure.

[0136] In summary, the embodiments of the present disclosure provide a variety of data perturbation methods, enriching the processing methods for the original text included in the image-text sample pairs. In addition, when deleting or masking the content of the original text included in the image-text sample pairs, the embodiments of the present disclosure adopt random deletion or random masking to ensure the diversity and randomness of the Mask Tokens input to the model, thereby improving the generalization ability of the model. In addition, when randomly deleting or randomly masking the original text included in the image-text sample pairs, by setting a preset ratio, the generalization ability and robustness of the model can be improved, and potential catastrophic consequences can be avoided.

[0137] In 203, the electronic device performs text feature extraction on the edited text to obtain a first text feature; and performs text feature extraction on the original text included in the multiple image-text sample pairs to obtain a second text feature; and performs image feature extraction on the images included in the multiple image-text sample pairs to obtain image features.

[0138] See also Figure 3 , CLIP includes an image encoder and a text encoder. Among them, the image encoder usually uses a convolutional neural network or is based on the Vision Transformer (ViT) architecture to process the input image and extract important image features, that is, convert the input image into a vector representation. The text encoder is usually based on the Transformer architecture to process the input text and extract important text features, that is, convert the input text into a vector representation.

[0139] In some embodiments, Figure 3 The text encoder for processing the original text shown in the figure and the text encoder for processing MaskTokens can be the same text encoder or different text encoders, which is not limited in the present disclosure. That is, the number of text encoders included in CLIP can be 1 or 2.

[0140] In the disclosed embodiment, after N segments of edited text are encoded by the text encoder, a vector representation of size N*T can be obtained, which is also referred to as the first text feature in this article. After N segments of original text are encoded by the text encoder, a vector representation of size N*T can be obtained, which is also referred to as the second text feature in this article. After N images are encoded by the image encoder, a vector representation of size N*I can be obtained, which is also referred to as the image feature in this article.

[0141] In other words, during the training process of CLIP, N segments of original text, N images matching the N segments of original text, and N segments of edited text are input to CLIP at one time.

[0142] It should be noted that the values ​​of N, I and T are all positive integers.

[0143] In 204, the electronic device performs model training based on the first text feature, the second text feature and the image feature with the goal of learning the correlation between text and image and the correlation between text and text, and obtains a model for performing text encoding in a text-image scenario.

[0144] In the disclosed embodiments, the correlation between text and image refers to the correlation between the entity concept of the text and the subject of the image, which is also called the alignment relationship between the text and the image. The correlation between text and text refers to the semantic correlation between text and text, which is also called the alignment relationship between text and text.

[0145] In addition, if Figure 3 As shown, the embodiment of the present disclosure performs model training based on the idea of ​​contrastive learning, that is, the correlation between images and texts and the correlation between texts are measured by the distance between vector representations.

[0146] For example, the distance (such as cosine distance) between the image and text included in the positive image-text sample pair in the feature space is made smaller, and the distance between the two texts included in the positive text sample pair in the feature space is made smaller, that is, the correlation is higher.

[0147] For another example, the distance between the image and the text included in the negative image-text sample pair in the feature space is larger, and the distance between the two texts included in the negative text sample pair in the feature space is larger, that is, the correlation is lower.

[0148] For the text-based image scenario, the model training scheme provided by the disclosed embodiment optimizes the training process of CLIP and introduces text-to-text comparative learning into the training process of CLIP. This enables CLIP to learn not only the correlation between images and texts, but also the correlation between texts, significantly improving the text comprehension ability of CLIP. Under the supervision of text-to-text comparative learning, it ensures that the text features corresponding to semantically similar texts are similar, thereby ensuring that the images finally generated in the text-based image task are also similar. That is, similar images can be generated based on semantically similar texts. In summary, this scheme makes the images generated in the text-based image task more accurate and more in line with user needs, thereby improving the human-computer interaction effect.

[0149] Figure 4 is a flowchart of another model training method in a text graph scenario according to an exemplary embodiment. Figure 4 As shown, the model training method in the text image scenario is applied to electronic devices, such as Figure 1In the server 102 shown. The model training method in the text-based graph scenario includes the following steps:

[0150] In 401, the electronic device constructs a plurality of image-text sample pairs, wherein one image-text sample pair includes a segment of original text and an image.

[0151] This step is the same as the above step 201 and will not be repeated here.

[0152] In 402, for any segment of original text, the electronic device splits the segment of original text into multiple basic units for text processing; deletes some basic units from the multiple basic units obtained by splitting, and obtains an edited text corresponding to the segment of original text, wherein the segment of edited text includes the basic units that have not been deleted.

[0153] This step is the same as the above step 202 and will not be described again here.

[0154] In 403, the electronic device performs text feature extraction on the edited text to obtain a first text feature; and performs text feature extraction on the original text included in the multiple image-text sample pairs to obtain a second text feature; and performs image feature extraction on the images included in the multiple image-text sample pairs to obtain image features.

[0155] This step is the same as the above step 203 and will not be described again here.

[0156] In 404, the electronic device performs model training based on the first text feature, the second text feature and the image feature with the goal of learning the correlation between text and image and the correlation between text and text, and obtains a model for performing text encoding in the text-image scenario.

[0157] In some embodiments, the embodiments of the present disclosure perform model training in the following manner based on the first text feature, the second text feature and the image feature, thereby obtaining a model for performing text encoding in a text-image scenario, wherein the model is a multimodal pre-trained model, namely CLIP.

[0158] 4041. Obtain a first loss based on the first text feature and the second text feature, wherein the first loss is used to measure the correlation between the original text and the edited text.

[0159] For this step, first obtain the similarity between the first text feature and the second text feature to obtain a first similarity matrix. In the embodiment of the present disclosure, the first text feature and the second text feature have the same dimension, both of which are T dimensions. Therefore, the similarity between the first text feature and the second text feature can be obtained in the following manner:

[0160] First, the first text feature and the second text feature are normalized to obtain a normalized first text feature and a normalized second text feature; then, a dot product is performed on the normalized first text feature and the normalized image feature to obtain a first similarity matrix.

[0161] The size of the first similarity matrix is ​​N*N. The value of each element in the first similarity matrix represents the similarity between the text embedding and the mask tokens embedding. Exemplarily, the similarity mentioned in this article is cosine similarity, where cosine distance = 1-cosine similarity.

[0162] Among them, the first similarity matrix includes first-category elements and second-category elements, the first-category elements are used to indicate the similarity between a pair of text samples with similar semantics, and the second-category elements are used to indicate the similarity between a pair of text samples with different semantics. Usually, the first-category elements are the elements on the diagonal line of the first similarity matrix, and the second-category elements are the elements on the off-diagonal line of the first similarity matrix. Accordingly, in the first similarity matrix, the elements on the diagonal line are used to indicate the similarity between a pair of text samples with similar semantics, and the elements on the off-diagonal line are used to indicate the similarity between a pair of text samples with different semantics.

[0163] Next, a first loss is obtained based on the first label information and the similarity associated with each original text in the first similarity matrix, and a second sub-loss is obtained based on the second label information and the similarity associated with each edited text in the first similarity matrix.

[0164] The first label information is used to indicate the edited texts that are semantically similar to each original text segment, and the edited texts that are semantically dissimilar to each original text segment. For example, the columns in the first similarity matrix are used to indicate the original texts, and the rows are used to indicate the edited texts. For any original text segment, the similarity in the first similarity matrix related to the original text segment refers to the value of the column element corresponding to the original text segment.

[0165] The second label information is used to indicate the original text that is semantically similar to each edited text, and the original text that is semantically not similar to each edited text. Continuing with the example that the columns in the first similarity matrix are used to indicate the original text and the rows are used to indicate the edited text, for any edited text, the similarity associated with the edited text in the first similarity matrix refers to the value of the element in the row corresponding to the edited text.

[0166] During training, CLIP uses a contrastive loss function that encourages the model to keep the vector representation of each piece of original text close to the vector representation of the edited text that is semantically close to it, and away from the vector representation of the edited text that is semantically dissimilar to it. Exemplarily, the model is trained to maximize the dot product between each piece of original text and the edited text that is correctly paired with it, while minimizing the dot product between each piece of original text and the edited text that is incorrectly paired with it. This means that the dot product must be as close to 1 as possible (the diagonal element in the first similarity matrix). For everything else, it needs to be pushed towards 0.

[0167] Finally, based on the first sub-loss and the second sub-loss, the first loss is obtained. For example, the weighted average of the first sub-loss and the second sub-loss is taken as the first loss, which is not limited in the present disclosure.

[0168] 4042. Obtain a second loss based on the second text feature and the image feature, wherein the second loss is used to measure the correlation between the original text and the image.

[0169] For this step, first obtain the similarity between the second text feature and the image feature to obtain a second similarity matrix. Since the dimensions of the second text feature and the image feature are different, namely, I dimension and T dimension respectively, the similarity between the second text feature and the image feature can be obtained in the following manner:

[0170] First, the second text feature and the image feature are size-aligned to obtain the second text feature and the image feature with the same dimension; next, the second text feature and the image feature with the same dimension are normalized to obtain the normalized second text feature and the normalized image feature; finally, the dot product is performed on the normalized second text feature and the normalized image feature to obtain the second similarity matrix.

[0171] Exemplarily, in order to make the second text feature and the image feature reach the same dimension, two projection layers can be introduced, one for processing the image feature and the other for processing the second text feature, so that they reach the same dimension D, where the value of D is a positive integer. After the projection layer, two matrices of size N*D are obtained. Next, the two matrices are multiplied to obtain a matrix of size N*N, namely the second similarity matrix. Among them, the value of each element in the second similarity matrix represents the similarity between the text embedding and the image embedding.

[0172] Among them, the second similarity matrix includes third-category elements and fourth-category elements, the third-category elements are used to indicate the similarity between the original text and the image that are correctly paired, and the fourth-category elements are used to indicate the similarity between the original text and the image that are incorrectly paired. Usually, the third-category elements are the elements on the diagonal line of the second similarity matrix, and the fourth-category elements are the elements on the non-diagonal line of the second similarity matrix. Correspondingly, in the second similarity matrix, the elements on the diagonal line are used to indicate the similarity between the original text and the image that are correctly paired, and the elements on the non-diagonal line are used to indicate the similarity between the original text and the image that are incorrectly paired.

[0173] Next, based on the third label information and the similarity associated with each image in the second similarity matrix, a third sub-loss is obtained. And based on the fourth label information and the similarity associated with each original text in the second similarity matrix, a fourth sub-loss is obtained.

[0174] The third label information is used to indicate the original text that is correctly matched with each image, and the original text that is incorrectly matched with each image. Taking the example that the rows of the second similarity matrix are used to indicate images and the columns are used to indicate texts, for any piece of original text, the similarity associated with the piece of original text in the second similarity matrix refers to the value of the element in the column corresponding to the piece of original text.

[0175] Among them, the fourth label information is used to indicate the image that is correctly matched with each original text, and the image that is incorrectly matched with each original text; continuing to take the example that the rows of the second similarity matrix are used to indicate images and the columns are used to indicate texts, then for any image, the similarity associated with the image in the second similarity matrix refers to the value of the element in the row corresponding to the image.

[0176] Finally, based on the third sub-loss and the fourth sub-loss, the second loss is obtained. For example, the weighted average of the third sub-loss and the fourth sub-loss is taken as the second loss, which is not limited in the present disclosure.

[0177] It should be noted that the first label information, the second label information, the third label information and the fourth label information mentioned above can be obtained by constructing the image-text sample pair and the text sample pair. In addition, when constructing the sub-loss, for a given original text, an edited text or an image, the similarity related to it can be softmaxed, and then the cross entropy loss can be obtained based on the corresponding label information, which is not limited in the present disclosure.

[0178] 4043. Adjust the model parameters based on the first loss and the second loss to obtain a CLIP for performing text encoding in a text image scenario.

[0179] Exemplarily, the model parameters can be adjusted by minimizing the first loss and the second loss, thereby obtaining a CLIP for performing text encoding in a text image scenario.

[0180] In addition, if the first loss and the second loss cannot reach the minimum at the same time, the training can be terminated when one of the losses reaches the minimum value and the other loss does not exceed the preset threshold. This is not limited in the present disclosure.

[0181] For the text-based image scenario, the model training scheme provided by the disclosed embodiment optimizes the training process of CLIP and introduces text-to-text comparison learning into the training process of CLIP. This enables CLIP to learn not only the correlation between images and texts, but also the correlation between texts, significantly improving the text comprehension ability of CLIP. Under the supervision of text-to-text comparison learning, it is ensured that the text features corresponding to semantically similar texts are similar, thereby ensuring that the images finally generated in the text-based image task are also similar. That is, similar images can be generated based on semantically similar texts.

[0182] In addition, when learning the correlation between images and texts, by constructing image-text sample pairs and determining the similarity between the vector representation of the original text and the vector representation of the image, the distance between the correctly paired original text and the image in the feature space is smaller, that is, the correlation is higher, and the distance between the incorrectly paired original text and the image in the feature space is larger, that is, the correlation is lower. This enables CLIP to better learn the pairing relationship between images and texts.

[0183] In addition, when learning the correlation between texts, by constructing text sample pairs and determining the similarity between the vector representation of the original text and the vector representation of Mask Tokens, the original text with similar semantics and MaskTokens have a smaller distance in the feature space, that is, a higher correlation, and the original text with different semantics and MaskTokens have a larger distance in the feature space, that is, a lower correlation. This enables CLIP to better learn the pairing relationship between texts and significantly improves CLIP's text comprehension ability.

[0184] In addition, when determining the similarity between the vector representation of the original text and the vector representation of the Mask Tokens, a normalization operation is performed so that the cosine similarity between the vector representations of different texts can be directly obtained after the dot product operation, thereby improving computational efficiency.

[0185] In addition, when determining the similarity between the vector representation of the original text and the vector representation of the image, by performing size alignment and normalization operations, the cosine similarity between the vector representation on the text dimension and the vector representation on the image dimension can be directly obtained after the dot product operation, which has high computational efficiency.

[0186] In summary, this scheme can better learn the pairing relationship between images and texts, as well as the pairing relationship between texts, which makes the images generated in the text-to-text task more accurate and more in line with user needs. That is, the generated images are more consistent with the text descriptions input by the users, which improves the human-computer interaction effect in the text-to-text scenario.

[0187] Figure 5 is a block diagram of a model training device in a text graph scenario according to an exemplary embodiment. Figure 5 , the device comprises:

[0188] The acquisition module 501 is configured to construct a plurality of image-text sample pairs, wherein one of the image-text sample pairs includes a section of original text and an image;

[0189] The editing module 502 is configured to perform data perturbation on the original text included in the plurality of image-text sample pairs to obtain an edited text;

[0190] A feature extraction module 503 is configured to extract text features from the edited text and the original text included in the plurality of image-text sample pairs, to obtain a first text feature of the edited text and a second text feature of the plurality of image-text sample pairs;

[0191] The feature extraction module 503 is further configured to extract image features from the images included in the plurality of image-text sample pairs to obtain image features;

[0192] The training module 504 is configured to perform model training based on the first text feature, the second text feature and the image feature with the goal of learning the correlation between text and image and the correlation between text and text, so as to obtain a model for performing text encoding in the text-image scenario.

[0193] For the text-based image scenario, the model training scheme provided by the disclosed embodiment optimizes the training process of CLIP and introduces text-to-text comparative learning into the training process of CLIP. This enables CLIP to learn not only the correlation between images and texts, but also the correlation between texts, significantly improving the text comprehension ability of CLIP. Under the supervision of text-to-text comparative learning, it ensures that the text features corresponding to semantically similar texts are similar, thereby ensuring that the images finally generated in the text-based image task are also similar. That is, similar images can be generated based on semantically similar texts. In summary, this scheme makes the images generated in the text-based image task more accurate and more in line with user needs, thereby improving the human-computer interaction effect.

[0194] In some embodiments, the training module 504 is configured to:

[0195] Based on the first text feature and the second text feature, obtaining a first loss; wherein the first loss is used to measure the correlation between the original text and the edited text;

[0196] Based on the second text feature and the image feature, obtaining a second loss; wherein the second loss is used to measure the correlation between the original text and the image;

[0197] The model parameters are adjusted based on the first loss and the second loss to obtain the model for performing text encoding.

[0198] In some other embodiments, the training module 504 is configured to:

[0199] Obtaining the similarity between the first text feature and the second text feature to obtain a first similarity matrix; wherein the first similarity matrix includes first-category elements and second-category elements, the first-category elements are used to indicate the similarity between a pair of text samples with similar semantics, and the second-category elements are used to indicate the similarity between a pair of text samples with different semantics; a pair of text samples includes a section of original text and a section of edited text;

[0200] Obtaining the first loss based on the first label information and the similarity associated with each original text segment in the first similarity matrix; wherein the first label information is used to indicate the edited text that is semantically similar to each original text segment and the edited text that is semantically dissimilar to each original text segment;

[0201] Obtaining a second sub-loss based on the second label information and the similarity associated with each edited text in the first similarity matrix; wherein the second label information is used to indicate the original text that is semantically similar to each edited text and the original text that is semantically not similar to each edited text;

[0202] The first loss is obtained based on the first sub-loss and the second sub-loss.

[0203] In some other embodiments, the first text feature and the second text feature have the same dimension; the training module 504 is configured to:

[0204] Normalizing the first text feature and the second text feature to obtain a normalized first text feature and a normalized second text feature;

[0205] Performing a dot product on the normalized first text feature and the normalized image feature to obtain the first similarity matrix.

[0206] In some other embodiments, the training module 504 is configured to:

[0207] Acquire the similarity between the second text feature and the image feature to obtain a second similarity matrix; wherein the second similarity matrix includes third-category elements and fourth-category elements, the third-category elements are used to indicate the similarity between the original text and the image that are correctly paired, and the fourth-category elements are used to indicate the similarity between the original text and the image that are incorrectly paired;

[0208] Obtaining a third sub-loss based on the third label information and the similarity associated with each image in the second similarity matrix; wherein the third label information is used to indicate the original text correctly paired with each image and the original text incorrectly paired with each image;

[0209] Obtaining a fourth sub-loss based on fourth label information and the similarity associated with each original text segment in the second similarity matrix; wherein the fourth label information is used to indicate images correctly paired with each original text segment and images incorrectly paired with each original text segment;

[0210] The second loss is obtained based on the third sub-loss and the fourth sub-loss.

[0211] In some other embodiments, the dimension of the second text feature is different from that of the image feature; the training module 504 is configured to:

[0212] Aligning the second text feature and the image feature in size to obtain the second text feature and the image feature having the same dimension;

[0213] Normalizing the second text feature and the image feature having the same dimension to obtain a normalized second text feature and a normalized image feature;

[0214] Perform dot product on the normalized second text feature and the normalized image feature to obtain the second similarity matrix.

[0215] In some other embodiments, the editing module 502 is configured to:

[0216] For any original text, delete part of the original text to obtain the edited text corresponding to the original text; or

[0217] For any section of original text, part of the content of the original text is masked to obtain the edited text corresponding to the original text.

[0218] In other embodiments, the editing module 502 is configured to split the original text into multiple basic units for text processing; delete some basic units from the multiple basic units obtained by splitting to obtain the edited text, and the edited text includes the basic units that are not deleted.

[0219] In some other embodiments, the editing module 502 is configured to randomly delete a preset proportion of basic units from the multiple basic units.

[0220] In other embodiments, the editing module 502 is configured to split the original text into multiple basic units for text processing; mask some basic units among the multiple basic units obtained by the splitting to obtain the edited text, and the edited text includes the unmasked basic units.

[0221] In some other embodiments, the acquisition module 501 is configured to:

[0222] Acquire a sample data set, wherein the sample data set includes a first number of images and texts matching the first number of images;

[0223] Constructing the plurality of image-text sample pairs based on the sample data set;

[0224] The multiple image-text sample pairs include a first number of positive image-text sample pairs and a second number of negative image-text sample pairs; the positive image-text sample pairs include correctly paired original text and images; and the negative image-text sample pairs include incorrectly paired original text and images.

[0225] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, which will not be described in detail here.

[0226] It should be noted that the model training device in the Wensheng graph scenario provided in the above embodiment only uses the division of the above functional units as an example when performing model training in the Wensheng graph scenario. In actual applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device is divided into different functional units to complete all or part of the functions described above. In addition, the model training device in the Wensheng graph scenario provided in the above embodiment and the model training method embodiment in the Wensheng graph scenario belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0227] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0228] In some embodiments, when the electronic device is provided as a server, Figure 6 6 is a block diagram of a server 600 according to an exemplary embodiment. The server 600 may have relatively large differences due to different configurations or performances, and includes one or more processors (Central Processing Units, CPU) 601 and one or more memories 602, wherein the memory 602 stores at least one program code, and the at least one program code is loaded and executed by the processor 601 to implement the image processing method provided by the above-mentioned various method embodiments. Of course, the server also has components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server 600 also includes other components for realizing device functions, which will not be repeated here.

[0229] In some embodiments, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, and the instructions are executed by a processor of an electronic device to implement the model training method in the above-mentioned Vincent map scenario. In other embodiments, the computer-readable storage medium is a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device.

[0230] In some embodiments, a computer program product is also provided, including a computer program, which, when executed by a processor of an electronic device, implements the model training method in the above-mentioned Wenshengtu scenario.

[0231] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0232] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A model training method in a cultural image scenario, characterized in that: The method comprises: Constructing a plurality of image-text sample pairs, wherein one of the image-text sample pairs includes a piece of original text and an image; Performing data perturbation on the original text included in the plurality of image-text sample pairs to obtain edited text; Performing text feature extraction on the edited text to obtain a first text feature; and performing text feature extraction on the original text included in the plurality of image-text sample pairs to obtain a second text feature; Extracting image features from the images included in the plurality of image-text sample pairs to obtain image features; With the goal of learning the correlation between text and image and the correlation between text and text, model training is performed based on the first text feature, the second text feature and the image feature to obtain a model for performing text encoding in a text-image scenario.

2. The model training method in the context of a cultural image according to claim 1, characterized in that: The performing of model training based on the first text feature, the second text feature and the image feature to obtain a model for performing text encoding in a text-image scenario includes: Based on the first text feature and the second text feature, obtaining a first loss; wherein the first loss is used to measure the correlation between the original text and the edited text; Based on the second text feature and the image feature, obtaining a second loss; wherein the second loss is used to measure the correlation between the original text and the image; The model parameters are adjusted based on the first loss and the second loss to obtain the model for performing text encoding.

3. The model training method in the context of a cultural graph according to claim 2, characterized in that: The obtaining a first loss based on the first text feature and the second text feature includes: Obtaining the similarity between the first text feature and the second text feature to obtain a first similarity matrix; wherein the first similarity matrix includes first-category elements and second-category elements, the first-category elements are used to indicate the similarity between a pair of text samples with similar semantics, and the second-category elements are used to indicate the similarity between a pair of text samples with different semantics; a pair of text samples includes a section of original text and a section of edited text; Obtaining a first sub-loss based on the first label information and the similarity associated with each original text segment in the first similarity matrix; wherein the first label information is used to indicate edited texts that are semantically similar to each original text segment and edited texts that are semantically dissimilar to each original text segment; Obtaining a second sub-loss based on the second label information and the similarity associated with each edited text in the first similarity matrix; wherein the second label information is used to indicate the original text that is semantically similar to each edited text and the original text that is semantically not similar to each edited text; The first loss is obtained based on the first sub-loss and the second sub-loss.

4. The model training method in the context of a cultural image according to claim 3, characterized in that: The first text feature and the second text feature have the same dimension; and the obtaining of the similarity between the first text feature and the second text feature to obtain a first similarity matrix includes: Normalizing the first text feature and the second text feature to obtain a normalized first text feature and a normalized second text feature; Performing a dot product on the normalized first text feature and the normalized image feature to obtain the first similarity matrix.

5. The model training method in the context of a cultural graph according to claim 2, characterized in that: The obtaining a second loss based on the second text feature and the image feature includes: Acquire the similarity between the second text feature and the image feature to obtain a second similarity matrix; wherein the second similarity matrix includes third-category elements and fourth-category elements, the third-category elements are used to indicate the similarity between the original text and the image that are correctly paired, and the fourth-category elements are used to indicate the similarity between the original text and the image that are incorrectly paired; Obtaining a third sub-loss based on the third label information and the similarity associated with each image in the second similarity matrix; wherein the third label information is used to indicate the original text correctly paired with each image and the original text incorrectly paired with each image; Obtaining a fourth sub-loss based on fourth label information and the similarity associated with each original text segment in the second similarity matrix; wherein the fourth label information is used to indicate images correctly paired with each original text segment and images incorrectly paired with each original text segment; The second loss is obtained based on the third sub-loss and the fourth sub-loss.

6. The model training method in the context of a cultural graph according to claim 5, characterized in that: The second text feature and the image feature have different dimensions; and the obtaining of the similarity between the second text feature and the image feature to obtain a second similarity matrix includes: Aligning the second text feature and the image feature in size to obtain the second text feature and the image feature having the same dimension; Normalizing the second text feature and the image feature having the same dimension to obtain a normalized second text feature and a normalized image feature; Perform dot product on the normalized second text feature and the normalized image feature to obtain the second similarity matrix.

7. The model training method in the context of a cultural image according to claim 1, characterized in that: The step of performing data perturbation on the original text included in the plurality of image-text sample pairs to obtain edited text includes: For any original text, delete part of the original text to obtain the edited text corresponding to the original text; or For any section of original text, part of the content of the original text is masked to obtain the edited text corresponding to the original text.

8. The model training method in the context of a cultural image according to claim 7, characterized in that: The step of deleting part of the original text to obtain an edited text corresponding to the original text includes: Splitting the original text into a plurality of basic units for text processing; Some basic units are deleted from the multiple basic units obtained by splitting to obtain the edited text, and the edited text includes the basic units that are not deleted.

9. The model training method in the context of a cultural graph according to claim 8, characterized in that: The deleting some basic units from the multiple basic units obtained by splitting includes: A preset proportion of basic units is randomly deleted from the plurality of basic units.

10. The model training method in the context of a cultural image according to claim 7, characterized in that: The step of masking part of the original text to obtain an edited text corresponding to the original text includes: Splitting the original text into a plurality of basic units for text processing; Some basic units are masked among the multiple basic units obtained by splitting to obtain the edited text, wherein the edited text includes the basic units that are not masked.

11. The model training method in the Vincent diagram scenario according to any one of claims 1 to 10, characterized in that: The step of constructing a plurality of image-text sample pairs includes: Acquire a sample data set, wherein the sample data set includes a first number of images and texts matching the first number of images; Constructing the plurality of image-text sample pairs based on the sample data set; The multiple image-text sample pairs include a first number of positive image-text sample pairs and a second number of negative image-text sample pairs; the positive image-text sample pairs include correctly paired original text and images; and the negative image-text sample pairs include incorrectly paired original text and images.

12. A model training device in a Wenshengtu scenario, characterized in that: The device comprises: An acquisition module is configured to construct a plurality of image-text sample pairs, wherein one of the image-text sample pairs includes a section of original text and an image; An editing module, configured to perform data perturbation on the original text included in the plurality of image-text sample pairs to obtain an edited text; A feature extraction module is configured to extract text features from the edited text to obtain a first text feature; and to extract text features from the original text included in the plurality of image-text sample pairs to obtain a second text feature; The feature extraction module is further configured to extract image features from the images included in the plurality of image-text sample pairs to obtain image features; The training module is configured to perform model training based on the first text feature, the second text feature and the image feature with the goal of learning the correlation between text and image and the correlation between text and text, so as to obtain a model for performing text encoding in a text-image scenario.

13. An electronic device, characterized in that: The electronic device comprises: one or more processors; a memory for storing program code executable by the processor; The processor is configured to execute the program code to implement the model training method in the Wenshengtu scenario as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the model training method in the Wenshengtu scenario as described in any one of claims 1 to 11.

15. A computer program product, characterized in that It includes a computer program, which, when executed by a processor of an electronic device, implements the model training method in the Wenshengtu scenario as described in any one of claims 1 to 11.

Images