Data processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the application discloses a data processing method and device, electronic equipment and storage medium. If the first similarity of the first image and the first text in the target data set is greater than the target similarity, the second similarity of the enhanced text of the first image and the first text is obtained. If the second similarity of the enhanced text of the first text and the first image is greater than the first similarity, the image-text pair composed of the first image and the enhanced text of the first text is added to the target data set as a positive sample. If the similarity of the enhanced text of the first text and the first image is less than the first similarity, the image-text pair composed of the first image and the enhanced text of the first text is added to the target data set as a negative sample.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, and more particularly, to a data processing method and device, electronic equipment and storage medium. BACKGROUND

[0002] Some popular applications at present, such as image-text retrieval, generating images through text, etc., are usually implemented based on machine models, and the machine models are usually trained through one-to-one image-text pairs. However, in actual applications, different language descriptions often match one image, or different images match one text, so the machine model trained based on one-to-one image-text pairs increases the limitation on the flexibility of text or image, thereby losing certain feature extraction accuracy and resulting in weak generalization ability of the machine model. SUMMARY

[0003] The present application aims to provide a data processing method and device, electronic equipment and storage medium, comprising the following technical solutions:

[0004] A data processing method, the method comprising:

[0005] If the first similarity of the first image and the first text in the target data set is greater than the target similarity, obtaining the second similarity of the enhanced text of the first image and the first text;

[0006] If the second similarity is greater than the first similarity, adding the image-text pair composed of the first image and the enhanced text as a positive sample to the target data set, and if the second similarity is less than the first similarity, adding the image-text pair composed of the first image and the enhanced text as a negative sample to the target data set, to obtain an expanded data set.

[0007] The above method, optionally, further comprises:

[0008] Obtaining the similarity of any image and any text in the target data set, and determining whether the similarity is greater than the target similarity;

[0009] Or,

[0010] Determining whether the image-text pair composed of any image and any text in the target data set is a positive sample, and if it is a positive sample, determining that the similarity of the any image and the any text is greater than the target similarity.

[0011] The above method, optionally, the target data set is used to train a first image-text processing model, and the first image-text processing model is trained in the following manner:

[0012] extracting features of the first text through the first image-text processing model to obtain features of the first text; the first image and the first text constitute a positive sample or a negative sample;

[0013] extracting features of the first text through the first image-text processing model to obtain features of the first text; the first image and the first text constitute a positive sample or a negative sample;

[0014] if the first similarity of the first image and the first text obtained based on the features of the first image and the features of the first text is greater than the target similarity, extracting features of the enhanced text of the first text through the first image-text processing model to obtain features of the enhanced text of the first text;

[0015] if the second similarity of the first image and the enhanced text of the first text obtained based on the features of the first image and the features of the enhanced text of the first text is greater than the first similarity of the first image and the first text, determining the image-text pair constituted by the first image and the enhanced text of the first text as a positive sample, if the second similarity of the first image and the enhanced text of the first text is less than the first similarity of the first image and the first text, determining the image-text pair constituted by the first image and the enhanced text of the first text as a negative sample;

[0016] updating parameters of the first image-text processing model, with the similarity between the image and the text in the positive sample becoming greater and greater, and the similarity between the image and the text in the negative sample becoming smaller and smaller as the target.

[0017] Optionally, the expanded data set is used to train a second image-text processing model or a third image-text processing model,

[0018] the second image-text processing model is trained in the following manner:

[0019] extracting features of the second text in the expanded data set through the second image-text processing model to obtain features of the second text; processing the features of the second text to obtain a predicted image corresponding to the second text; updating parameters of the second image-text processing model, with the predicted image approaching the second image and the predicted image moving away from the third image as the target; the second image is an image in the expanded data set that constitutes a positive sample with the second text, and the third image is an image in the expanded data set that constitutes a negative sample with the second text;

[0020] the third image-text processing model is trained in the following manner:

[0021] extracting features of a fourth image in the augmented data set by the third image-text processing model to obtain features of the fourth image; processing the features of the fourth image to obtain a predicted text corresponding to the fourth image; updating parameters of the third image-text processing model, with the predicted text tending to be close to a third text and the predicted text tending to be far away from a fourth text as a target; the third text being a text in the augmented data set that forms a positive sample with the fourth image, and the fourth text being a text in the augmented data set that forms a negative sample with the fourth image.

[0022] Optionally, in the method, the target data set is composed of images and texts in a plurality of positive samples, each positive sample containing one image and one text, and different positive samples containing different images and different texts.

[0023] Optionally, in the method, the target data set is composed of images and texts in a plurality of positive samples, each positive sample containing one image and one text, and different positive samples containing different images and different texts.

[0024] For any test image in a test set, there are multiple extreme values in similarity of the any test image to each test text in the test set obtained based on a first model; the first model being any one of the first image-text processing model and a target image-text processing model; the test set being composed of images and texts in a plurality of one-to-one positive samples;

[0025] A difference between a maximum value and a minimum value in each extreme value corresponding to the first image-text processing model is less than a difference between the maximum value and the minimum value in each extreme value corresponding to the target image-text processing model.

[0026] The target image-text processing model is obtained in the following manner:

[0027] extracting features of an image sample and a text sample by the target image-text processing model to obtain a first sample feature of the image sample and a second sample feature of the text sample; updating parameters of the target image-text processing model based on a preset target; the preset target including: if the image sample and the text sample form a one-to-one positive sample, a similarity of the image sample and the text sample obtained based on the first sample feature and the second sample feature is getting larger and larger; and if the image sample and the text sample form a negative sample, the similarity of the image sample and the text sample obtained based on the first sample feature and the second sample feature is getting smaller and smaller.

[0028] Optionally, the method further includes:

[0029] reading the second similarity from a target storage area;

[0030] Alternatively,

[0031] obtaining an enhanced text of the first text; processing the first image and the enhanced text of the first text to obtain the second similarity.

[0032] A data processing apparatus, the apparatus comprising:

[0033] an obtaining module, configured to, if a first similarity of a first image and a first text in a target data set is greater than a target similarity, obtain a second similarity of an enhanced text of the first image and the first text;

[0034] an expanding module, configured to, if the second similarity is greater than the first similarity, add an image-text pair composed of the first image and the enhanced text to the target data set as a positive sample, and if the second similarity is less than the first similarity, add the image-text pair composed of the first image and the enhanced text to the target data set as a negative sample, to obtain an expanded data set.

[0035] An electronic device, comprising:

[0036] a memory, configured to store a program;

[0037] a processor, configured to invoke and execute the program in the memory, and realize each step of the data processing method according to any one of the preceding embodiments by executing the program.

[0038] A readable storage medium, having a computer program stored thereon, the computer program being executed by a processor to realize each step of the data processing method according to any one of the preceding embodiments.

[0039] According to the above scheme, the data processing method, apparatus, electronic device and storage medium provided by the present application can be used to, if a first similarity of a first image and a first text in a target data set is greater than a target similarity, obtain a second similarity of an enhanced text of the first image and the first text, if the second similarity of the enhanced text of the first text and the first image is greater than the first similarity, add an image-text pair composed of the first image and the enhanced text of the first text to the target data set as a positive sample, if the similarity of the enhanced text of the first text and the first image is less than the first similarity, add the image-text pair composed of the first image and the enhanced text of the first text to the target data set as a negative sample, realize expansion of the target data set, and in the expanded data set, different positive samples can contain the same image, or different positive samples contain the same text, increase the diversity of samples in the data set, and thus improve the generalization ability of a machine model trained based on the expanded target data set. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed by the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description only represent some of the embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings.

[0041] Figure 1 An implementation flowchart of the data processing method provided by the embodiments of the present application is provided.

[0042] Figure 2 An implementation flowchart of training the first image-text processing model provided by the embodiments of the present application is provided.

[0043] Figure 3 An implementation flowchart of training the second image-text processing model by expanding the data set provided by the embodiments of the present application is provided.

[0044] Figure 4 An implementation flowchart of training the third image-text processing model by expanding the data set provided by the embodiments of the present application is provided.

[0045] Figure 5 A principle example diagram of the data processing method provided by the embodiments of the present application is provided.

[0046] Figure 6 An architecture schematic diagram of the data processing method provided by the embodiments of the present application is provided.

[0047] Figure 7 A structure schematic diagram of the data processing device provided by the embodiments of the present application is provided.

[0048] Figure 8 A structure schematic diagram of the electronic device provided by the embodiments of the present application is provided.

[0049] The terms "first", "second", "third", "fourth" and the like (if any) in the description, claims and above drawings are used to distinguish similar parts, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated herein. DETAILED DESCRIPTION

[0050] The technical solutions of the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0051] The data processing method provided by the embodiments of the present application can be used in an electronic device, which can be a terminal device, such as a desktop computer, a notebook computer, or the like, or a server, such as a single server or a server cluster.

[0052] As shown in the figure, the flowchart of one implementation of the data processing method provided by the embodiments of the present application can include the following steps. Figure 1

[0053] Step S101: If the similarity between the first image and the first text in the target data set (denoted as the first similarity for the sake of description and distinction) is greater than the target similarity, obtaining the similarity between the first image and the enhanced text of the first text (denoted as the second similarity for the sake of description and distinction).

[0054] The target data set is composed of images and texts in image-text pairs, wherein different image-text pairs can contain different images and texts, or different image-text pairs contain the same image and different texts, or different image-text pairs contain different images and the same text.

[0055] The first image can be any image in the target data set, and the first text can be any text in the target data set.

[0056] The enhanced text of the first text is generated by performing target processing on the first text.

[0057] Optionally, the similarity between the image and the text can be represented by distance measurement or similarity measurement.

[0058] Taking the first image and the first text as an example, the features of the first image and the features of the first text can be obtained, and the distance measurement (such as Euclidean distance, Minkowski distance, Manhattan distance, or the like) is calculated based on the features of the first image and the features of the first text, so as to represent the similarity between the first image and the first text; or the similarity measurement (such as cosine similarity, Pearson correlation coefficient, or the like) is calculated based on the features of the first image and the features of the first text, so as to represent the similarity between the first image and the first text.

[0059] Step S102: If the second similarity is greater than the first similarity, adding the image-text pair composed of the first image and the enhanced text of the first text as a positive sample to the target data set, and if the second similarity is less than the first similarity, adding the image-text pair composed of the first image and the enhanced text of the first text as a negative sample to the target data set, so as to obtain an expanded data set.

[0060] ​The second similarity is greater than the first similarity, indicating that the enhanced text of the first text matches the first image more than the first text does; the second similarity is less than the first similarity, indicating that the matching degree of the enhanced text of the first text with the first image is reduced compared with the first text, that is, the enhanced text of the first text does not match the first image as well as the first text does.

[0061] If the second similarity is equal to the first similarity, the enhanced text of the first text is discarded, that is, the image-text pair formed by the enhanced text of the first text and the first image is not added to the target data set.

[0062] The data processing method provided in the application, if the first similarity of the first image and the first text in the target data set is greater than the target similarity, obtains the second similarity of the enhanced text of the first image and the first text, if the second similarity of the enhanced text of the first text and the first image is greater than the first similarity, adds the image-text pair formed by the first image and the enhanced text of the first text as a positive sample to the target data set, if the similarity of the enhanced text of the first text and the first image is less than the first similarity, adds the image-text pair formed by the first image and the enhanced text of the first text as a negative sample to the target data set, realizes the expansion of the target data set, and in the expanded data set, different positive samples can contain the same image, or different positive samples contain the same text, increasing the diversity of samples in the data set, thereby improving the generalization ability of the machine model trained based on the expanded target data (i.e., the expanded data set).

[0063] In an optional embodiment, before step S101, the data processing method of the application can further include:

[0064] Obtaining the similarity of any image and any text in the target data set, and determining whether the similarity is greater than the target similarity.

[0065] The similarity of any image and any text can be represented by distance measurement or similarity measurement. The specific way of obtaining the distance measurement and the similarity measurement of any image and any text can refer to the foregoing process of obtaining the similarity of the first image and the first text, which will not be described here.

[0066] In an optional embodiment, before step S101, the data processing method of the application can further include:

[0067] Determining whether the image-text pair formed by any image and any text in the target data set is a positive sample, if it is a positive sample, determining that the similarity of any image and any text is greater than the target similarity, otherwise, determining that the similarity of any image and any text is less than or equal to the target similarity.

[0068] Wherein, whether the image-text pair composed of any image and any text is a positive sample can be determined in advance according to the similarity of any image and any text. Optionally, if the similarity of any image and any text is greater than a specified similarity, it is determined that the image-text pair composed of the any image and the any text is a positive sample, otherwise, it is determined that the image-text pair composed of the any image and the any text is a negative sample. Based on this, the application can save the similarity of the any image and the any text when determining positive / negative samples, so that the first similarity of the first image and the first text can be directly read when comparing the first similarity and the second similarity subsequently.

[0069] In an optional embodiment, the target data set described above is used to train to obtain a first image-text processing model. As an example, the first image-text processing model can be an image-text matching model, which is used to judge whether an image and a text match, based on which the image-text matching model can be used to find an image matching a text in a plurality of images; or the image-text matching model can be used to find a text matching an image in a plurality of texts.

[0070] As an example, for the to-be-processed image and the to-be-processed text, the to-be-processed image can be feature-extracted by the trained first image-text processing model to obtain the features of the to-be-processed image, the to-be-processed text can be feature-extracted by the trained first image-text processing model to obtain the features of the to-be-processed text; the similarity of the to-be-processed image and the to-be-processed text is determined based on the features of the to-be-processed image and the features of the to-be-processed text; if the similarity is greater than a threshold, it is determined that the to-be-processed image and the to-be-processed text match, otherwise, it is determined that the to-be-processed image and the to-be-processed text do not match.

[0071] Optionally, an implementation flowchart for training the first image-text processing model provided by the embodiments of the application is as shown in Figure 2 , which can include:

[0072] Step S201: feature-extracting the first image and the first text by the first image-text processing model, specifically including: feature-extracting the first image by the first image-text processing model to obtain the features of the first image; feature-extracting the first text by the first image-text processing model to obtain the features of the first text.

[0073] As an example, the first image-text processing model can include a first image encoder and a first text encoder; wherein the first image encoder is used to encode the first image to obtain the features of the first image, and the first text encoder is used to encode the first text to obtain the features of the first text.

[0074] The first image and the first text constitute a positive sample or a negative sample in the target data set.

[0075] Optionally, the first image encoder and the first text encoder can be pre-trained encoders. As an example, the first image encoder and the first text encoder can be pre-trained based on a text-image matching task, which can refer to a process of training the target text-image processing model, which will not be repeated here. As an example, the first image encoder and the first text encoder can be pre-trained based on different tasks, respectively. For example, the first text encoder can be pre-trained based on an entity recognition task or a text-image generation task, and the first image encoder can be pre-trained based on a target recognition task or a text-image generation task.

[0076] The process of pre-training the first text encoder based on the entity recognition task can include: encoding the sample text by the first text encoder to obtain the feature of the sample text, decoding the feature of the sample text by the entity recognition decoder to obtain the predicted entity, updating the parameters of the first text encoder and the entity recognition decoder until the training end condition is met, with the predicted entity approaching the label (i.e. entity) of the sample text as the goal.

[0077] The process of pre-training the first text encoder based on the text-image generation task can include: encoding the sample text by the first text encoder to obtain the feature of the sample text, decoding the feature of the sample text by the text-image decoder to obtain the predicted image, updating the parameters of the first text encoder and the text-image decoder until the training end condition is met, with the predicted image approaching the label (i.e. image) of the sample text as the goal.

[0078] The process of pre-training the first image encoder based on the target recognition task can include: encoding the sample image by the first image encoder to obtain the feature of the sample image, decoding the feature of the sample image by the target recognition decoder to obtain the predicted target, updating the parameters of the first image encoder and the target recognition decoder until the training end condition is met, with the predicted target approaching the label (i.e. target in the image) of the sample image as the goal.

[0079] The process of pre-training the first image encoder based on the text-image generation task can include: encoding the sample image by the first image encoder to obtain the feature of the sample image, decoding the feature of the sample image by the text-image decoder to obtain the predicted text, updating the parameters of the first image encoder and the text-image decoder until the training end condition is met, with the predicted text approaching the label (i.e. text) of the sample image as the goal.

[0080] Step S202: If the condition is met, obtain the feature of the enhanced text of the first text, including: if the first similarity of the first image and the first text obtained based on the feature of the first image and the feature of the first text is greater than the target similarity, obtaining the feature of the enhanced text of the first text by performing feature extraction on the enhanced text of the first text through the first image-text processing model.

[0081] Optionally, the distance between the first image and the first text can be calculated based on the feature of the first image and the feature of the first text, and the distance is used to represent the first similarity of the first image and the first text. Based on this, the target similarity can be represented by a target distance.

[0082] Optionally, the cosine similarity (or other similarity measure) between the first image and the first text can be calculated based on the feature of the first image and the feature of the first text, as the first similarity of the first image and the first text.

[0083] The feature of the enhanced text of the first text can be extracted through the first image-text processing model before step S202; or, the feature of the enhanced text of the first text can be extracted through the first image-text processing model again when it is determined that the first similarity of the first image and the first text is greater than the target similarity.

[0084] Step S203: Determine the added positive / negative sample, including: if the second similarity of the first image and the enhanced text of the first text based on the feature of the first image and the feature of the enhanced text of the first text is greater than the first similarity of the first image and the first text, the image-text pair composed of the first image and the enhanced text of the first text is determined as a positive sample, and if the second similarity of the first image and the enhanced text of the first text is less than the first similarity of the first image and the first text, the image-text pair composed of the first image and the enhanced text of the first text is determined as a negative sample.

[0085] If the second similarity of the first image and the enhanced text of the first text is equal to the first similarity of the first image and the first text, the enhanced text of the first text is discarded, that is, the image-text pair composed of the first image and the enhanced text of the first text is not formed.

[0086] Further, if the image-text pair composed of the first image and the enhanced text of the first text is determined as a positive sample, the positive sample can also be added to the target data set, and similarly, if the image-text pair composed of the first image and the enhanced text of the first text is determined as a negative sample, the negative sample can also be added to the target data set, that is, the target data set can be expanded in the process of training the first image-text processing model, that is, the target data set is expanded while the first image-text processing model is trained.

[0087] Since the training process includes multiple iteration processes, if the enhanced text of the same text needs to be obtained in different iteration processes, the enhanced text of the same text obtained in different iteration processes is different.

[0088] Step S204: Update the parameters of the first image-text processing model, aiming at the similarity between the image and the text in the positive sample being more and more large, and the similarity between the image and the text in the negative sample being more and more small.

[0089] Here, the positive sample (which can be denoted as a target positive sample) used to update the parameters of the first image-text processing model includes: the positive sample input into the first image-text processing model, and / or the positive sample determined based on the positive sample or the negative sample input into the first image-text processing model.

[0090] The negative sample (which can be denoted as a target negative sample) used to update the parameters of the first image-text processing model includes: the negative sample input into the first image-text processing model, and / or the negative sample determined based on the positive sample or the negative sample input into the first image-text processing model.

[0091] Specifically:

[0092] If the first image and the first text constitute a positive sample, and the image-text pair constituted by the first image and the enhanced text of the first text is determined as a positive sample, then the parameters of the first image-text processing model are updated, aiming at the similarity between the first image and the first text being more and more large, and the similarity between the first image and the enhanced text of the first text being more and more large.

[0093] If the first image and the first text constitute a positive sample, and the image-text pair constituted by the first image and the enhanced text of the first text is determined as a negative sample, then the parameters of the first image-text processing model are updated, aiming at the similarity between the first image and the first text being more and more large, and the similarity between the first image and the enhanced text of the first text being more and more small.

[0094] If the first image and the first text constitute a positive sample, and the image-text pair constituted by the first image and the enhanced text of the first text is not determined as a positive sample and a negative sample, then the parameters of the first image-text processing model are updated, aiming at the similarity between the first image and the first text being more and more large.

[0095] If the first image and the first text constitute a negative sample, and the image-text pair constituted by the first image and the enhanced text of the first text is determined as a positive sample, then the parameters of the first image-text processing model are updated, aiming at the similarity between the first image and the first text being more and more small, and the similarity between the first image and the enhanced text of the first text being more and more large.

[0096] If the first image and the first text constitute a negative sample, and the image-text pair constituted by the first image and the enhanced text of the first text is determined as a negative sample, the similarity between the first image and the first text is made smaller and smaller, and the similarity between the first image and the enhanced text of the first text is made smaller and smaller as a target, and the parameters of the first image-text processing model are updated.

[0097] If the first image and the first text constitute a negative sample, and the image-text pair constituted by the first image and the enhanced text of the first text is not determined as a positive sample and a negative sample, the similarity between the first image and the first text is made smaller and smaller as a target, and the parameters of the first image-text processing model are updated.

[0098] In a case where the first image-text processing model comprises a first image encoder and a first text encoder, the parameters of the first image-text processing model are updated, comprising updating the parameters of the first image encoder and the first text encoder.

[0099] Optionally, the parameters of the first image-text processing model are updated based on a loss function as follows:

[0100]

[0101]

[0102]

[0103]

[0104] wherein N represents the number of samples input into the first image-text processing model, 2g-1 represents an image in the gth sample input into the first image-text processing model, and 2g represents a text in the gth sample input into the first image-text processing model.

[0105] The value of an element M i，j in the matrix M is determined according to formula (2), wherein s i，j represents the similarity between the ith image and the jth text input into the first image-text processing model; s i，j′ represents the similarity between the ith image and the jth text input into the first image-text processing model; s i，j = 1 indicates that the similarity between the ith image and the jth text is greater than the target similarity, and the enhanced text of the ith image and the jth text is determined as a positive sample; M i，j = 0 indicates that the similarity between the ith image and the jth text is greater than the target similarity, and the enhanced text of the ith image and the jth text is determined as a negative sample; M i，j = -1 indicates that the similarity between the ith image and the jth text is less than or equal to the target similarity, and the enhanced text of the ith image and the jth text is not determined as a positive sample or a negative sample.

[0106] numel(M|M i，j represents the number of elements with value 1 in the statistical matrix M.

[0107] s i，k represents the similarity between the ith image and the kth text; s i，k′ represents the similarity between the ith image and the kth text.

[0108] z i represents the feature of the ith image, which is a vector; z j represents the feature of the jth text, which is a vector. T represents the transpose operation.

[0109] The calculation process of l(2g-1, 2g) and l(i, j') is described with reference to formulas (3)-(4), specifically,

[0110]

[0111]

[0112] In the above embodiment, the target data set can be augmented in the process of training the first image-text processing model. In another optional embodiment, the target data set can be augmented by using the trained first image-text processing model after the training of the first image-text processing model is completed, or other data sets can be augmented by using the trained first image-text processing model.

[0113] The process of augmenting the target data set or other data sets by using the trained first image-text processing model can be referred to the embodiment shown in Figure 1 , which will not be described in detail here.

[0114] In an optional embodiment, the augmented data set can be used to train a second image-text processing model, which is a text-to-image generation model, i.e., a model for generating an image corresponding to a text according to an input text.

[0115] An implementation flowchart for training the second image-text processing model by augmenting the data set provided in the embodiments of the present application is shown in Figure 3 , which can include:

[0116] Step S301: performing feature extraction on the second text in the augmented data set by using the second image-text processing model to obtain the feature of the second text; and processing the feature of the second text to obtain an image corresponding to the second text (denoted as a predicted image).

[0117] The second text is any text in the augmented data set.

[0118] Optionally, the second image-text processing model can include a second text encoder and a first decoder. The second text encoder is configured to encode the second text to obtain a feature of the second text, and the first decoder is configured to decode the feature of the second text to obtain a predicted image corresponding to the second text.

[0119] The second text encoder can be a first text encoder in the first image-text processing model, or can be a text encoder different from the first text encoder.

[0120] In step S302, parameters of the second image-text processing model are updated, with the goal that the predicted image approaches the second image and is away from the third image.

[0121] The second image is an image in the augmented dataset that constitutes a positive sample with the second text, and the third image is an image in the augmented dataset that constitutes a negative sample with the second text.

[0122] In the case where the second text encoder is the first text encoder in the first image-text processing model, updating the parameters of the second image-text processing model can include updating parameters of the first decoder, but not updating parameters of the second text encoder.

[0123] In the case where the second text encoder is not the first text encoder in the first image-text processing model, updating the parameters of the second image-text processing model includes updating parameters of the first decoder and updating parameters of the second text encoder.

[0124] In an optional embodiment, the augmented dataset can be used to train a third image-text processing model, which is a text generation model, i.e., a model for generating text corresponding to an input image.

[0125] An implementation flowchart for training the third image-text processing model by using the augmented dataset according to an embodiment of the present application is shown in Figure 4 The implementation flowchart can include the following steps.

[0126] In step S401, a fourth image in the augmented dataset is subjected to feature extraction by the third image-text processing model to obtain a feature of the fourth image, and the feature of the fourth image is processed to obtain text (denoted as predicted text) corresponding to the fourth image.

[0127] The fourth image is any image in the augmented dataset.

[0128] Optionally, the third image-text processing model can include a second image encoder and a second decoder. The second image encoder is configured to encode the fourth image to obtain the feature of the fourth image, and the second decoder is configured to decode the feature of the fourth image to obtain the predicted text corresponding to the fourth image.

[0129] The second image encoder can be the first image encoder in the first image-text processing model, or can be an image encoder different from the first image encoder.

[0130] Step S402: updating the parameters of the third image-text processing model, aiming at the prediction text approaching the third text and the prediction text moving away from the fourth text.

[0131] The third text is a text in the expansion data set that constitutes a positive sample with the fourth image, and the fourth text is a text in the expansion data set that constitutes a negative sample with the fourth image.

[0132] In the case where the second image encoder is the first image encoder in the first image-text processing model, updating the parameters of the third image-text processing model can include updating the parameters of the second decoder, and does not include updating the parameters of the second image encoder.

[0133] In the case where the second image encoder is not the first image encoder in the first image-text processing model, updating the parameters of the second image-text processing model includes updating the parameters of the second decoder, and also includes updating the parameters of the second image encoder.

[0134] In an optional embodiment, the target data set is composed of images and texts in a plurality of positive samples; wherein each positive sample contains an image and a text, and different positive samples contain different images and different texts. That is, the target data set is composed of one-to-one image-text pairs.

[0135] In an optional embodiment, for any test image in the test set, there are multiple extreme values in the similarity between the test image obtained based on the first model and each test text in the test set; the first model is any one of the first image-text processing model and the target image-text processing model; and the test set is composed of images and texts in a plurality of one-to-one positive samples.

[0136] In this application, the test set is composed of images and texts in one-to-one positive samples, that is, each positive sample includes an image and a text, and different positive samples include different images and different texts.

[0137] Suppose the test set contains n one-to-one positive samples, then the test set contains n test images and n test texts; for any test image in the test set, denoted as the rth(r=1, 2, 3, …, n) test image, the similarity between the rth test image and each test text in the n test texts can be obtained through the first image-text processing model, and n similarities corresponding to the first image-text processing model are obtained, denoted as S 11 1n ​The similarity between the r-th test image and each of the n test texts can be obtained through the target image processing model, resulting in n similarity scores corresponding to the target image processing model, denoted as S. 21 ~S 2n .

[0138] Where the similarity S 11 ~S 1n There are multiple extreme values ​​in it, and the similarity S 21 ~S 2n There are also multiple extreme values ​​in it.

[0139] The difference between the maximum and minimum values ​​among the extreme values ​​corresponding to the first image and text processing model is less than the difference between the maximum and minimum values ​​among the extreme values ​​corresponding to the target image and text processing model.

[0140] Following the example above, suppose S 11 ~S 1n The maximum and minimum extrema in the value are O. 1max and O 1min Assume S 21 ~S 2n The maximum and minimum extrema in the value are O. 2max and O 2min Then O 1max -O 1min <O 2max -O 2min .

[0141] The target image processing model is obtained in the following way:

[0142] The target image-text processing model is used to extract features from image samples and text samples respectively, resulting in the first sample features of the image samples and the second sample features of the text samples.

[0143] The parameters of the target image and text processing model are updated based on preset targets. The preset targets include: if the image sample and the text sample constitute a one-to-one positive sample, the similarity between the image sample and the text sample obtained based on the first sample feature and the second sample feature increases; if the image sample and the text sample constitute a negative sample, the similarity between the image sample and the text sample obtained based on the first sample feature and the second sample feature decreases.

[0144] In this application, the target image-text processing model is trained using a training dataset, which consists of image samples and text samples from a one-to-one positive sample set. Assuming that image sample I1 and text sample T1 in the training dataset constitute positive samples, then image sample I1 and text sample T1 in the training dataset... h (h ≠ 1) constitutes a negative sample.

[0145] In an optional embodiment, when the first similarity of the first image and the first text in the target data set is greater than the target similarity, one implementation of obtaining the second similarity of the enhanced text of the first image and the first text can be as follows:

[0146] The second similarity is read from a target storage area (for ease of description and differentiation, referred to as a first target storage area). That is, the second similarity of the first image and the enhanced text of the first text is pre-determined and stored in the first target storage area, and is directly read when needed.

[0147] In an optional embodiment, when the first similarity of the first image and the first text in the target data set is greater than the target similarity, one implementation of obtaining the second similarity of the enhanced text of the first image and the first text can be as follows:

[0148] The enhanced text of the first text is obtained, and the first image and the enhanced text of the first text are processed to obtain the second similarity.

[0149] That is, the second similarity of the first image and the enhanced text of the first text is calculated in real time when the second similarity is needed.

[0150] Optionally, one implementation of obtaining the enhanced text of the first text can be as follows: the enhanced text of the first text is read from a target storage area (for ease of description and differentiation, referred to as a second target storage area). That is, the enhanced text of each text in the target data set is pre-obtained in the present application, and the enhanced text of each text is stored in the second target storage area. When the enhanced text of the first text is needed, the first text does not need to be enhanced, and the enhanced text of the first text can be directly read from the second target storage area.

[0151] As an example, the second target storage area can store the enhanced text of the first text, or the features of the enhanced text of the first text.

[0152] The enhanced text of the first text is obtained by target processing of the first text, and the target processing can include but is not limited to at least one of the following: randomly inserting text, randomly deleting part of the text, performing synonym replacement, etc.

[0153] Optionally, another implementation of obtaining the enhanced text of the first text can be as follows: the first text is target processed to obtain the enhanced text of the first text. The target processing includes at least one of the following: randomly inserting text, randomly deleting part of the text, performing synonym replacement, etc.

[0154] In other words, this application can obtain the enhanced text of the first text without obtaining the enhanced text of each text in the target dataset in advance, but can perform target processing on the first text as needed to obtain the enhanced text of the first text.

[0155] like Figure 5 The diagram shown is a schematic representation of a data processing method provided in an embodiment of this application. In this diagram, x1 to x N This represents N images in the target dataset, where each image uniquely corresponds to a text. In other words, the target dataset consists of N positive samples, each of which consists of one image and one text. Different positive samples include different images and different texts.

[0156] T1~T N Let T1′~T be the features of N texts in the target dataset. N ′ represents the features of the enhanced text of N texts, I1~I N This represents the features of N images in the target dataset.

[0157] For any image among N images ( Figure 5 Taking the Nth image as an example, based on the features I of the Nth image... N The features T of the h-th (h = 1, 2, 3, ..., N) text h Calculate the similarity between the Nth image and the hth text, denoted as Sim(I N T h If Sim(I) N T h ) greater than the target similarity (i.e. Figure 5 If the threshold is calculated, then the features of the enhanced text of the h-th text are obtained, and the elements M in matrix M are... N,h The value of Sim(I) is set to 0. N T h If the similarity is less than or equal to the target similarity, then obtaining the features of the enhanced text of the h-th text is prohibited, and the elements M in matrix M are removed. N,h The value is set to -1.

[0158] Figure 5 The similarity between the Nth image and the 3rd text (denoted as Sim(I)) is used to measure the similarity between the Nth image and the 3rd text. N Taking the example that T3)) is greater than the target similarity, let's take the elements M in matrix M as... N,3 The value of is set to 0, and the feature T3′ of the enhanced text of the third text is obtained.

[0159] Then, calculate the similarity between the enhanced text of the Nth image and the 3rd text, denoted as Sim(I N T3′), compare Sim(I)N T3′) and Sim(I N (T3).

[0160] If Sim(I) N ,T3′) <Sim(I N T3), the augmented text of the Nth image and the 3rd text is determined as the negative sample (i.e., T3). Figure 5 (in the Negative pair), M N,3 The value of Sim(I) remains 0; N ,T3′)>Sim(I N T3), the augmented text of the Nth image and the 3rd text is determined as a positive sample (i.e., T3). Figure 5 Positive pair in M N,3 The value is updated to 1.

[0161] like Figure 6 The diagram shown is an architectural schematic of a data processing method provided in an embodiment of this application. In this schematic, a text encoder is used to extract features from text or enhanced text to obtain the features of the text or enhanced text, and an image encoder is used to extract features from an image to obtain the features of the image.

[0162] Optionally, a first similarity between the image and text can be calculated based on the features of the image and the text. If the first similarity is greater than the target similarity, the enhanced text is then input into the text encoder, which extracts features from the enhanced text. If the features of the enhanced text are obtained, a second similarity between the image and the enhanced text is calculated based on the features of the image and the enhanced text. If the second similarity is greater than the first similarity, the image and enhanced text are identified as a positive image-text pair (i.e., a positive sample). If the second similarity is less than the first similarity, the image and enhanced text are identified as a negative image-text pair (i.e., a negative sample). Then, the parameters of the first image-text processing model are updated using a loss function.

[0163] Optionally, the text and the enhanced text can be simultaneously input into the first image-text processing model to obtain the features of the text and the features of the enhanced text. If the first similarity between the image and the text is greater than the target similarity, then the second similarity between the image and the enhanced text is calculated based on the features of the image and the features of the enhanced text. Otherwise, the calculation of the second similarity between the image and the enhanced text is prohibited based on the features of the image and the features of the enhanced text.

[0164] Corresponding to the method embodiments, this application also provides a data processing apparatus. A schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application is shown below. Figure 7 As shown, it may include:

[0165] The obtaining module 701 and the expanding module 702; wherein

[0166] The obtaining module 701 is configured to obtain a second similarity between the first image and the enhanced text of the first text if the first similarity between the first image and the first text in the target data set is greater than the target similarity.

[0167] The expanding module 702 is configured to add the image-text pair composed of the first image and the enhanced text of the first text to the target data set as a positive sample if the second similarity is greater than the first similarity, or add the image-text pair composed of the first image and the enhanced text of the first text to the target data set as a negative sample, so as to obtain an expanded data set.

[0168] The data processing apparatus provided by the embodiments of the present application can obtain a second similarity between a first image and an enhanced text of a first text if a first similarity between the first image and the first text in a target data set is greater than a target similarity, add an image-text pair composed of the first image and the enhanced text of the first text to the target data set as a positive sample if the second similarity between the enhanced text of the first text and the first image is greater than the first similarity, or add the image-text pair composed of the first image and the enhanced text of the first text to the target data set as a negative sample, so as to expand the target data set, and the expanded data set can contain the same image in different positive samples, or contain the same text in different positive samples, thereby increasing the diversity of samples in the data set and improving the generalization ability of a machine model trained based on the expanded target data (i.e., the expanded data set).

[0169] In an optional embodiment, the apparatus further includes a determining module configured to:

[0170] Obtain a similarity between any image and any text in the target data set, and determine whether the similarity is greater than the target similarity.

[0171] Or,

[0172] Determine whether an image-text pair composed of any image and any text in the target data set is a positive sample, and if so, determine whether the similarity between the any image and the any text is greater than the target similarity.

[0173] In an optional embodiment, the target data set is used to train a first image-text processing model, and the apparatus further includes a first training module configured to:

[0174] Extract features of the first image by the first image-text processing model to obtain features of the first image.

[0175] extracting features of the first text by the first image-text processing model, to obtain features of the first text;

[0176] If the first similarity of the first image and the first text obtained based on the features of the first image and the features of the first text is greater than the target similarity, features of the enhanced text of the first text obtained by extracting features of the enhanced text of the first text by the first image-text processing model are obtained.

[0177] If the second similarity of the first image and the enhanced text of the first text obtained based on the features of the first image and the features of the enhanced text of the first text is greater than the first similarity of the first image and the first text, the image-text pair composed of the first image and the enhanced text of the first text is determined as a positive sample, and if the second similarity of the first image and the enhanced text of the first text is less than the first similarity of the first image and the first text, the image-text pair composed of the first image and the enhanced text of the first text is determined as a negative sample.

[0178] The similarity of the image and the text in the positive sample is increasingly greater, and the similarity of the image and the text in the negative sample is increasingly smaller, and the parameters of the first image-text processing model are updated.

[0179] In an optional embodiment, the expanded data set is used to train a second image-text processing model or a third image-text processing model, and the device further comprises a second training module and a third training module, wherein,

[0180] The second training module is configured to:

[0181] extracting features of the second text in the expanded data set by the second image-text processing model, to obtain features of the second text; processing the features of the second text to obtain a predicted image corresponding to the second text; updating the parameters of the second image-text processing model, with the predicted image approaching a second image and the predicted image moving away from a third image as the target; the second image is an image in the expanded data set that constitutes a positive sample with the second text, and the third image is an image in the expanded data set that constitutes a negative sample with the second text;

[0182] The third training module is configured to:

[0183] extracting features of a fourth image in the augmented dataset through the third image-text processing model to obtain features of the fourth image; processing the features of the fourth image to obtain predicted text corresponding to the fourth image; updating parameters of the third image-text processing model, with the predicted text approaching third text and the predicted text moving away from fourth text as the goal; the third text being text in the augmented dataset that forms a positive sample with the fourth image, and the fourth text being text in the augmented dataset that forms a negative sample with the fourth image.

[0184] In an optional embodiment, the target dataset is composed of images and texts in a plurality of positive samples, each positive sample containing an image and a text; different positive samples contain different images and different texts.

[0185] In an optional embodiment, the target dataset is composed of images and texts in a plurality of positive samples, each positive sample containing an image and a text; different positive samples contain different images and different texts.

[0186] For any test image in a test set, there are multiple extreme values in similarity of the any test image obtained based on a first model and each test text in the test set; the first model being any one of the first image-text processing model and the target image-text processing model; the test set being composed of images and texts in a plurality of one-to-one positive samples;

[0187] A difference between a maximum value and a minimum value in each extreme value corresponding to the first image-text processing model is less than a difference between a maximum value and a minimum value in each extreme value corresponding to the target image-text processing model;

[0188] The target image-text processing model is obtained in the following manner:

[0189] extracting features of an image sample and a text sample through the target image-text processing model to obtain first sample features of the image sample and second sample features of the text sample; updating parameters of the target image-text processing model based on a preset target; the preset target including: if the image sample and the text sample form a one-to-one positive sample, similarity of the image sample and the text sample obtained based on the first sample features and the second sample features is getting larger and larger; if the image sample and the text sample form a negative sample, similarity of the image sample and the text sample obtained based on the first sample features and the second sample features is getting smaller and smaller.

[0190] In an optional embodiment, when the first obtaining module 701 obtains the second similarity of the first image and the enhanced text, it is used for:

[0191] reading the second similarity from a target storage area;

[0192] Or,

[0193] obtaining an enhanced text of the first text; processing the first image and the enhanced text of the first text to obtain the second similarity.

[0194] Corresponding to the method embodiments, the present application also provides an electronic device, a structural schematic diagram of which is shown in Figure 8 which can include at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4.

[0195] In the embodiments of the present application, the number of the processor 1, the communication interface 2, the memory 3 and the communication bus 4 is at least one, and the processor 1, the communication interface 2 and the memory 3 complete the communication with each other through the communication bus 4.

[0196] The processor 1 can be a central processing unit CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application, etc.

[0197] The memory 3 can include a high-speed RAM memory, and can also include a non-volatile memory, etc., such as at least one disk memory.

[0198] The memory 3 stores a program, and the processor 1 can call the program stored in the memory 3, and the program is used for:

[0199] If the first similarity of the first image and the first text in the target data set is greater than the target similarity, obtaining the second similarity of the enhanced text of the first image and the first text;

[0200] If the second similarity is greater than the first similarity, adding the image-text pair composed of the first image and the enhanced text to the target data set as a positive sample, and if the second similarity is less than the first similarity, adding the image-text pair composed of the first image and the enhanced text to the target data set as a negative sample, to obtain an expanded data set.

[0201] Optionally, the refinement function and the extension function of the program can refer to the description above.

[0202] The embodiments of the present application also provide a storage medium, which can store a program suitable for the processor to execute, and the program is used for:

[0203] If the first similarity of the first image and the first text in the target dataset is greater than the target similarity, a second similarity of the first image and the enhanced text of the first text is obtained;

[0204] If the second similarity is greater than the first similarity, the image-text pair composed of the first image and the enhanced text is added to the target dataset as a positive sample, and if the second similarity is less than the first similarity, the image-text pair composed of the first image and the enhanced text is added to the target dataset as a negative sample, so as to obtain an expanded dataset.

[0205] Optionally, the refinement function and the expansion function of the program can refer to the description above.

[0206] Those skilled in the art can appreciate that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solutions. Those skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0207] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be realized by other ways. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0208] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0209] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0210] It should be understood that the features in the embodiments of the present application, each embodiment, and the features can be combined with each other, and all can achieve the purpose of solving the above technical problems.

[0211] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0212] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data processing method, the method comprising: If the first similarity between the first image and the first text in the target dataset is greater than the target similarity, then the second similarity between the enhanced text of the first image and the first text is obtained. If the second similarity is greater than the first similarity, the image-text pair consisting of the first image and the enhanced text is added as a positive sample to the target dataset; if the second similarity is less than the first similarity, the image-text pair consisting of the first image and the enhanced text is added as a negative sample to the target dataset to obtain an expanded dataset.

2. The method according to claim 1, further comprising: Obtain the similarity between any image and any text in the target dataset, and determine whether the similarity is greater than the target similarity. or, Determine whether any image-text pair consisting of any image and any text in the target dataset is a positive sample. If it is a positive sample, determine that the similarity between any image and any text is greater than the target similarity.

3. The method according to claim 1 or 2, wherein the target dataset is used to train a first image-text processing model, and the first image-text processing model is trained in the following manner: The first image is feature extracted using the first image processing model to obtain the features of the first image. The first image and text processing model is used to extract features from the first text to obtain the features of the first text; the first image and the first text constitute positive or negative samples. If the first similarity between the first image and the first text, obtained based on the features of the first image and the features of the first text, is greater than the target similarity, then the features of the enhanced text of the first text obtained by feature extraction of the enhanced text of the first text through the first image-text processing model are obtained. If the second similarity between the first image and the enhanced text of the first text, obtained based on the features of the first image and the features of the enhanced text of the first text, is greater than the first similarity between the first image and the first text, the image-text pair consisting of the first image and the enhanced text of the first text is determined as a positive sample; if the second similarity between the first image and the enhanced text of the first text is less than the first similarity between the first image and the first text, the image-text pair consisting of the first image and the enhanced text of the first text is determined as a negative sample. The parameters of the first image and text processing model are updated with the goal of increasing the similarity between images and text in positive samples and decreasing the similarity between images and text in negative samples.

4. The method according to claim 1 or 2, wherein the expanded dataset is used to train and obtain a second or third image-text processing model. The second image processing model was trained in the following way: The second text is processed by the second image processing model to extract features from the second text in the expanded dataset, thereby obtaining the features of the second text. The features of the second text are processed to obtain the predicted image corresponding to the second text; the parameters of the second image processing model are updated with the goal of the predicted image being closer to the second image and the predicted image being farther away from the third image; the second image is the image in the expanded dataset that constitutes a positive sample with the second text, and the third image is the image in the expanded dataset that constitutes a negative sample with the second text; The third image processing model is trained in the following way: The features of the fourth image in the expanded dataset are obtained by extracting features from the third image processing model. The features of the fourth image are processed to obtain the predicted text corresponding to the fourth image; the parameters of the third image-text processing model are updated with the goal of the predicted text being closer to the third text and the predicted text being farther away from the fourth text; the third text is the text in the expanded dataset that constitutes a positive sample with the fourth image, and the fourth text is the text in the expanded dataset that constitutes a negative sample with the fourth image.

5. The method according to claim 3, wherein, The target dataset consists of images and text from multiple positive samples, with each positive sample containing one image and one text; different positive samples contain different images and different texts.

6. The method according to claim 5, wherein, For any test image in the test set, there are multiple extreme values ​​in the similarity between any test image obtained based on the first model and each test text in the test set; the first model is any one of the first image-text processing model and the target image-text processing model; the test set consists of images and text from multiple one-to-one positive samples; The difference between the maximum and minimum values ​​among the extreme values ​​corresponding to the first image and text processing model is less than the difference between the maximum and minimum values ​​among the extreme values ​​corresponding to the target image and text processing model. The target image processing model is obtained in the following way: The target image-text processing model is used to extract features from image samples and text samples respectively, to obtain the first sample features of the image samples and the second sample features of the text samples; The parameters of the target image and text processing model are updated based on a preset target. The preset target includes: if the image sample and the text sample constitute a one-to-one positive sample, the similarity between the image sample and the text sample obtained based on the first sample feature and the second sample feature increases; if the image sample and the text sample constitute a negative sample, the similarity between the image sample and the text sample obtained based on the first sample feature and the second sample feature decreases.

7. The method according to claim 1, wherein obtaining the second similarity of the enhanced text of the first image and the first text comprises: Read the second similarity from the target storage area; or, Obtain the enhanced text of the first text; The enhanced text of the first image and the first text is processed to obtain the second similarity.

8. A data processing apparatus, the apparatus comprising: The module is configured to obtain a second similarity between the first image and the first text in the target dataset if the first similarity between the first image and the first text is greater than the target similarity. An expansion module is configured to add the image-text pair consisting of the first image and the enhanced text as a positive sample to the target dataset if the second similarity is greater than the first similarity, and to add the image-text pair consisting of the first image and the enhanced text as a negative sample to the target dataset if the second similarity is less than the first similarity, thereby obtaining an expanded dataset.

9. An electronic device, comprising: Memory, used to store programs; A processor is configured to invoke and execute the program in the memory, thereby implementing the various steps of the data processing method as described in any one of claims 1-7.

10. A readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the data processing method as described in any one of claims 1-7.

Images