Image-text matching method, electronic equipment and storage medium

Abstract

The invention provides an image-text matching method, electronic equipment and a storage medium, and relates to the technical field of data processing. The method comprises the following steps: firstly, acquiring a text; performing word segmentation processing on the text to obtain a word segmentation result comprising individual characters and phrases, one phrase comprising at least two individual characters; then, a feature vector corresponding to the single word is determined through a first embedding module of a pre-trained text encoder, and a feature vector corresponding to the phrase is determined through a second embedding module of the text encoder; the first embed module and the second embed module are obtained by training in different training stages, and the first embed module and the second embed module are obtained by training in different training stages; then encoding the feature vectors corresponding to the single words and the feature vectors corresponding to the phrases, and determining text semantic vectors corresponding to the text; and finally carrying out image-text matching based on the text semantic vector. Therefore, the image-text matching accuracy is improved, and the user experience is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method for image-text matching, an electronic device, and a storage medium. Background Art

[0002] With the rapid development of information technology and the internet, users have a growing need for image-text matching. For example, the gallery apps on electronic devices like mobile phones and tablets may contain multiple images. To find an image, users can enter search text in the gallery app's search box, and the electronic device will display images that match the search text. Alternatively, when users need to insert an image into an article, they can use the article's text to search for matching images in the browser.

[0003] However, in image-text matching tasks, it is easy for the image found by the electronic device to be irrelevant to the search text entered by the user, resulting in low accuracy of image matching, which affects the user experience. Summary of the Invention

[0004] In order to solve the above problems, the present application provides a picture-text matching method, electronic device and storage medium, the purpose of which is to improve the accuracy of picture-text matching and effectively improve the user experience.

[0005] In a first aspect, the present application provides a method for image-text matching. Exemplarily, the method can be applied to electronic devices, which can be terminals such as mobile phones, tablet computers, and laptop computers, or servers such as cloud servers and independent physical servers.

[0006] In this method, the electronic device first obtains text. For example, the text can be text input by a user, or text sent by other electronic devices that are communicatively connected to the electronic device, or pre-stored text; the electronic device then performs word segmentation processing on the text to obtain a word segmentation result including single words and phrases, wherein a phrase includes at least two single words. For example, taking the text "water cup on the table" as an example, the word segmentation result includes the phrase "water cup", and the remaining text can all be segmented into single words; the pre-trained text encoder includes a first embedding module and a second embedding module, and the two are trained separately in different training stages. The electronic device can determine the feature vector corresponding to the single word in the word segmentation result through the first embedding module, and determine the feature vector corresponding to the phrase in the word segmentation result through the second embedding module; then the electronic device encodes the feature vector corresponding to the single word and the feature vector corresponding to the phrase to determine the text semantic vector corresponding to the obtained text; finally, the electronic device performs image-text matching based on the text semantic vector. For example, based on this, a task of obtaining an image by using text matching can be performed, and a task of matching text by using an image can also be performed.

[0007] In this way, the problem of the text encoder's incorrect understanding of the semantics of the phrases split into single words due to splitting the phrases in the text into single words is solved, and the generated text semantic vector has strong representation ability and can retain the continuous semantics of the phrase; at the same time, the pre-trained text encoder includes a first embedding module and a second embedding module, which can realize the disassembly of the parameters of the first embedding module and the second embedding module. During the training process of the text encoder, the two can be trained separately at different stages, that is, the text encoder can be trained separately for the semantic understanding ability of single words in the text, and the text encoder can be trained separately for the semantic understanding ability of phrases in the text, avoiding the mutual influence between the two and further improving the semantic understanding ability of the text as a whole. Therefore, the image-text matching method provided by the present application can improve the accuracy of image-text matching and effectively improve the user experience.

[0008] In one possible implementation, the pre-trained text encoder is a pre-trained CLIP model, that is, the pre-trained CLIP model includes the pre-trained text encoder. The training steps of the pre-trained CLIP model may include: the electronic device first obtains a first image training sample and a first text training sample corresponding to the first image training sample. For example, an image may be obtained as the first image training sample, and then a content description text of the image is obtained as the corresponding first text training sample; the CLIP model to be fine-tuned includes a first embedding module and a second embedding module. The electronic device may adjust the first model parameter of the second embedding module included in the CLIP model to be fine-tuned based on the first text training sample, the first image training sample, and a first preset loss function, while keeping the remaining model parameters unchanged, that is, freezing the model parameters of the CLIP model to be fine-tuned except the first model parameter. For example, during the training process, a contrast loss training method may be used for training. After the training is completed, the pre-trained CLIP model is obtained.

[0009] In this way, during the training of the fine-tuned CLIP model, only the first model parameters of the second embedding module are adjusted, and the second embedding module is used to semantically understand phrases in the text. Therefore, the second embedding module's ability to understand the semantics of phrases in the text can be trained separately, avoiding interfering with the first embedding module's ability to understand the semantics of individual words in the text.

[0010] In one possible implementation, the step of obtaining the first embedding module and the second embedding module included in the CLIP model to be fine-tuned may include: the electronic device first obtains a second image training sample and a second text training sample corresponding to the second image sample. For example, the number of the second image training samples can be greater than the number of the first image training samples, and the same applies to the second text training samples; the electronic device then adjusts the model parameters of the CLIP model to be pre-trained based on the second text training sample, the second image training sample, and a second preset loss function, wherein the CLIP model to be pre-trained includes a third embedding module, that is, the model parameters of the third embedding module are trained and adjusted; then, when the parameter-adjusted CLIP model meets the pre-training cutoff condition, that is, training can be stopped at this time, the electronic device splits the third embedding module included in the parameter-adjusted CLIP model to obtain the first embedding module and the second embedding module included in the CLIP model to be fine-tuned. In this way, the model parameters of the first embedding module and the second embedding module can be disassembled, facilitating the subsequent separate training of the model parameters of the second embedding module.

[0011] In one possible implementation, the image-text matching method may further include: the electronic device adding multiple phrases to a word list of a word segmenter to obtain a post-addition word list, where the word list originally includes multiple words, and the post-addition word list includes multiple words and multiple phrases; accordingly, the step of segmenting the text to obtain a word segmentation result may include: the electronic device performing word segmentation on the text using the word segmenter based on the post-addition word list to obtain a word segmentation result. In this way, the word segmentation result of the text includes both words and phrases, avoiding the splitting of semantically continuous phrases into words, which is beneficial to improving the accuracy of image-text matching.

[0012] In a possible implementation, the added vocabulary also includes a first mapping relationship between multiple words and multiple identifiers, and a second mapping relationship between multiple phrases and multiple identifiers. For example, word 1 has a first mapping relationship with 1, word 2 has a first mapping relationship with 2, phrase 1 has a second mapping relationship with 10000, etc. Correspondingly, the image-text matching method may also include: the electronic device performs identifier mapping on the word segmentation result based on the first mapping relationship and the second mapping relationship to obtain an identifier sequence corresponding to the word segmentation result. For example, the electronic device performs identifier mapping on the words in the word segmentation result based on the first mapping relationship, and performs identifier mapping on the phrases in the word segmentation result based on the second mapping relationship to obtain an identifier sequence including the identifiers of the words and the identifiers of the phrases; the electronic device then determines the feature vector corresponding to the word based on the identifier sequence through the first embedding module of the pre-trained text encoder, and determines the feature vector corresponding to the phrase through the second embedding module of the pre-trained text encoder. In this way, complex text can be mapped to a simpler identifier sequence, reducing the difficulty of feature extraction and helping to improve computing efficiency.

[0013] In one possible implementation, the above-mentioned step of determining the feature vector corresponding to the single word based on the identifier sequence through the first embedding module of the pre-trained text encoder, and determining the feature vector corresponding to the phrase through the second embedding module of the pre-trained text encoder, can include: the electronic device first splits the identifier sequence to obtain a first identifier sequence for representing the single word and a second identifier sequence for representing the phrase; then the electronic device extracts the feature vector corresponding to the single word from the first embedding module of the pre-trained text encoder, and extracts the feature vector corresponding to the phrase from the second embedding module of the pre-trained text encoder; accordingly, the electronic device then merges the feature vector corresponding to the single word and the feature vector corresponding to the phrase based on the position of the single word in the text and the position of the phrase in the text to obtain the feature vector corresponding to the text; finally, the feature vector corresponding to the text is encoded to obtain the text semantic vector corresponding to the text. In this way, the identifier sequence is split into the first identifier sequence and the second identifier sequence, which facilitates the distinction between single words and phrases, and then the two are input into the first embedding module and the second embedding module respectively.

[0014] In one possible implementation, the step of performing image-text matching based on text semantic vectors may include: the electronic device determining, based on the text semantic vectors, an image that matches the text from a plurality of pre-stored images. This allows the image-text matching task to be performed using text matching. The representational capabilities of the text semantic vectors obtained in this application are enhanced, thereby improving the accuracy of image-text matching for this task.

[0015] In one possible implementation, the pre-stored plurality of images includes visual semantic vectors corresponding to the plurality of images; accordingly, the step of determining an image matching the text from the pre-stored plurality of images based on the text semantic vector may include: the electronic device first calculates the similarity between the plurality of visual semantic vectors corresponding to the plurality of images and the text semantic vectors corresponding to the text, thereby obtaining similarity calculation results corresponding to the plurality of images, indicating the distance between each image and the text; the electronic device then sorts the plurality of similarity calculation results in descending order, that is, placing images with closer distances in front; and then the electronic device determines the K images with the top K similarity calculation results from the plurality of images as images matching the text, that is, determining K images with higher similarity calculation results, where K is a positive integer and K is less than or equal to the number of the plurality of images. In this way, K images can be matched based on the text semantic vector of the text, and the text semantic vector can retain the continuous semantics of the phrase and has stronger representation ability, so that K more matching images can be obtained, thereby improving the accuracy of image-text matching.

[0016] In one possible implementation, the visual semantic vectors corresponding to the multiple images are obtained using an image encoder from a pre-trained CLIP model. This allows for the generation of visual semantic vectors that represent the semantic meaning of the images, improving the accuracy of image-text matching.

[0017] In one possible implementation, the text may include multiple texts. In the image-text matching method, before the electronic device performs image-text matching based on the text semantic vector, it can also obtain an image; determine the visual semantic vector of the image; accordingly, the image-text matching method may include: the electronic device may first obtain multiple texts, exemplarily, the multiple texts may be multiple pre-stored texts; the electronic device then performs word segmentation processing on the multiple texts respectively to obtain word segmentation results corresponding to the multiple texts; then, the electronic device determines the feature vector corresponding to the single word for each word segmentation result corresponding to the text through the first embedding module of the pre-trained text encoder, and determines the feature vector corresponding to the phrase through the second embedding module of the pre-trained text encoder; then, the electronic device encodes the feature vector corresponding to the single word and the feature vector corresponding to the phrase for each word segmentation result corresponding to the text, and determines the text semantic vector corresponding to the text; finally, the electronic device determines the text that matches the image from the multiple texts based on the visual semantic vector and the multiple text semantic vectors. In this way, the task of obtaining text by image matching in the image-text matching task is realized, and the representation ability of the multiple text semantic vectors obtained in the present application is enhanced, so the image-text matching accuracy of the task can be improved.

[0018] In a possible implementation, the above-mentioned step of obtaining text may include: obtaining the text sent by the terminal. For example, the image-text matching method can be applied to a server such as a cloud server, and the cloud server obtains the text sent by the terminal; the cloud server then performs word segmentation processing on the text to obtain a word segmentation result; then, the cloud server determines the feature vector corresponding to the single word through the first embedding module of the pre-trained text encoder, and determines the feature vector corresponding to the phrase through the second embedding module of the pre-trained text encoder; then the cloud server encodes the feature vector corresponding to the single word and the feature vector corresponding to the phrase to determine the text semantic vector corresponding to the text; the cloud server performs image-text matching based on the text semantic vector, and finally sends the image-text matching result obtained by the image-text matching to the terminal so that the terminal can present it to the user. In this way, it is shown that the terminal can implement the image-text matching task based on the image-text matching method provided in the embodiment of the present application, and the terminal and server connected in communication can also implement the image-text matching task based on the image-text matching method provided in the embodiment of the present application.

[0019] In a second aspect, the present application provides an electronic device comprising a memory and a processor; the memory stores computer program code, and the computer program code comprises computer instructions; one or more processors call computer instructions to enable the electronic device to execute the image-text matching method of the first aspect mentioned above.

[0020] In a third aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the image-text matching method of the first aspect described above is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of a search result provided in an embodiment of the present application;

[0022] Figure 2 A schematic diagram of a model structure of a text encoder provided in an embodiment of the present application;

[0023] Figure 3 A flowchart of a text encoder generating a text semantic vector provided in an embodiment of the present application;

[0024] Figure 4 A schematic diagram of word vector mapping using an embedding layer provided in an embodiment of the present application;

[0025] Figure 5 A schematic diagram of another embedding layer for word vector mapping provided in an embodiment of the present application;

[0026] Figure 6 A schematic diagram of the pre-training stage of the CLIP model training process provided in an embodiment of the present application;

[0027] Figure 7 A schematic diagram of the fine-tuning phase of the CLIP model training process provided in an embodiment of the present application;

[0028] Figure 8 A signaling interaction diagram of a picture-text matching method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the description of the following embodiments clear and concise, the vocabulary involved in the embodiments of the present application is first explained. It should be understood that this explanation is for a clearer understanding of the embodiments of the present application and does not necessarily constitute a limitation of the embodiments of the present application.

[0030] CLIP model: The CLIP (Contrastive Language-Image Pre-Training) model is a neural network model for matching images and text. In some embodiments, the CLIP model's text encoder and image encoder are trained to produce a text encoder that outputs text semantic vectors for text and an image encoder that outputs visual semantic vectors for images.

[0031] Text semantic vector: The text can be input into the text encoder to obtain a vector that can characterize the semantic features of the entire text. For example, the text encoder can adopt models such as transformers commonly used in natural language processing (NLP), which is not limited in this application. In an embodiment of the present application, the text encoder includes an original embedding module (also referred to as a first embedding module), a newly added embedding module (also referred to as a second embedding module), a transformer layer, and a mapping layer. The text can be input into the text encoder to obtain a text semantic vector corresponding to the text.

[0032] Visual semantic vector: The image can be input into an image encoder to obtain a visual semantic vector. For example, the image encoder can use a CNN model or a VIT model, which is not limited in this application. In an embodiment of this application, an image stored in a gallery application can be input into the image encoder to obtain the visual semantic vector corresponding to the image.

[0033] Vocabulary: A dictionary or vocabulary used by a word segmenter. In the embodiment of the present application, the word segmenter's vocabulary includes words, phrases (a phrase includes at least two words), punctuation marks, special symbols, etc. used for word segmentation.

[0034] Embedding layer: This layer is used to extract feature vectors from the input data and convert discrete words or symbols in the input data into continuous vector representations. In this embodiment of the application, the embedding layer is used to map individual words or phrases in the text into a continuous vector space, obtaining the initial text vector corresponding to the text.

[0035] Transformer layer: This layer is also called a transformer layer. It is used to capture the dependencies and contextual information in the input sequence through a self-attention mechanism and an encoder-decoder structure. In this embodiment of the application, the encoder of the transformer layer is used to capture the dependencies and contextual information of the initial text vector output by the embedding layer and output a text encoding vector.

[0036] Image: In the embodiment of the present application, the image includes pictures and video frames in a video.

[0037] The following is a comparative explanation of the technical advantages of the image-text matching method, electronic device, and storage medium provided by this application, in conjunction with related technologies. For ease of understanding, an example scenario is used for illustration.

[0038] In related technologies, after an electronic device obtains the search text input by a user, it usually implements image-text matching based on a CLIP model. The electronic device can input the search text into the text encoder of the CLIP model to obtain a vector corresponding to the search text, and then determine a vector with high similarity from the corresponding vectors of multiple pictures, and present the corresponding picture to the user as a search result, wherein the corresponding vectors of multiple pictures can be generated in advance based on the image encoder of the CLIP model.

[0039] Assume that the electronic device is a mobile phone, and the phone's gallery application stores multiple pictures. The image encoder of the CLIP model can obtain the vectors corresponding to the multiple pictures. After the user enters the search text "water cup" in the search box of the gallery application, the phone can obtain the vector corresponding to "water cup" through the text encoder of the CLIP model. Based on this, the phone can search and match among the vectors corresponding to the multiple pictures, and display the pictures matching "water cup" as search results.

[0040] However, if Figure 1 As shown, the mobile phone can search for pictures 1 and 2 that match "water cup", but it can also search for pictures 3 and 4 that only match "water". In other words, pictures 3 and 4 are not related to the search text "water cup", resulting in reduced accuracy of image-text matching and affecting the user experience.

[0041] The inventors have found that the reason why the image-text matching task in the above situation has low accuracy is that the word segmenter divides the text into individual characters (also called single words). That is to say, when there are phrases in the text (a phrase includes at least two single words), the phrases with continuous semantics will also be split into single words, resulting in incorrect vector representation corresponding to the text, which in turn reduces the accuracy of image-text matching.

[0042] Therefore, in order to solve the above problems, the embodiment of the present application provides a method for image-text matching. In this method, the word list of the word segmenter is expanded, and phrases are added to the word list containing single words; after the electronic device obtains the text, the text is segmented by the word segmenter to obtain a segmentation result containing single words and phrases; the electronic device splits the single words and phrases contained in the segmentation result, and inputs the single words into the original embedding module of the text encoder of the CLIP model to obtain the initial single word text vector, and inputs the phrase into the newly added embedding module of the text encoder to obtain the initial phrase text vector, and then merges the initial single word text vector and the initial phrase text vector according to the original position of the single words and phrases in the text to obtain the initial text vector corresponding to the text; then, based on the initial text vector, the text semantic vector corresponding to the text is obtained through the transformer layer and mapping layer of the text encoder; then the electronic device performs image-text matching based on the text semantic vector.

[0043] In this way, the problem of the CLIP model's misunderstanding of phrases due to splitting phrases in the text into single words is solved; at the same time, single words and phrases are input into different embedding modules to realize the weight decomposition of the original embedding module and the newly added embedding module. During training, the newly added embedding module can be fine-tuned separately, which not only avoids affecting the CLIP model's ability to understand the semantics of single words in the text, but also improves the CLIP model's ability to understand the semantics of phrases in the text. It can improve the model performance of the CLIP model, and then improve the ability to understand the semantics of the text, generate text semantic vectors with strong representation ability, help improve the accuracy of image-text matching in image-text matching tasks, and effectively improve the user experience.

[0044] Next, combine Figure 2 The structure of the text encoder of the CLIP model provided in the embodiments of the present application is introduced in detail.

[0045] like Figure 2 As shown in Figure 3, the text encoder includes a splitting module, an embedding layer, a merging module, a transformer layer, and a mapping layer.

[0046] The splitting module is used to split the ID sequence obtained by the word segmenter through word segmentation, serialization and mapping of the search text to obtain single-word ID subsequences and phrase ID subsequences. That is, the splitting module can split the single-word ID subsequences representing single words and the phrase ID subsequences representing phrases.

[0047] The embedding layer is used to capture the semantic information of the text. In some embodiments, the embedding layer may include an original embedding module and a newly added embedding module. The original embedding module is responsible for receiving the single-word ID subsequence output by the splitting module, converting the single-word ID subsequence into a real number vector of fixed dimension (also called feature extraction), and obtaining an initial single-word text vector (which may be called a feature vector corresponding to the single word); the newly added embedding module is responsible for receiving the phrase ID subsequence output by the splitting module, converting the phrase ID subsequence into a real number vector of fixed dimension, and obtaining an initial phrase text vector (which may be called a feature vector corresponding to the phrase).

[0048] The merging module is used to merge the initial single-word text vector and the initial phrase text vector to obtain an initial text vector (which can be called a feature vector corresponding to the text). In some embodiments, the merging module can merge the initial single-word text vector and the initial phrase text vector based on the original position of the single word in the search text and the original position of the phrase in the search text, so that the position of the initial single-word text vector in the obtained initial text vector is the same as that of the single word in the search text, and the position of the initial phrase text vector in the initial text vector is the same as that of the phrase in the search text.

[0049] The transformer layer is used to receive the initial text vector output by the merging module and generate a richer and more abstract text representation (which can be called the text encoding vector corresponding to the search text). Figure 2 As shown, the transformer layer may include a transformer encoder. In some embodiments, the transformer encoder includes multiple identical layers, each of which includes a self-attention mechanism and a feedforward neural network, which can capture the contextual information in the input initial text vector and the relationship between words (such as single words and phrases, single words and single words, or phrases and phrases, etc.).

[0050] The mapping layer is used to receive the text encoding vector output by the transformer layer. It can convert the text encoding vector output by the transformer layer and the visual semantic vector output by the image encoder to the same dimension and distribution, and output the text semantic vector, which is convenient for subsequent calculation of the vector similarity between the two through the CLIP model, thereby achieving cross-modal matching.

[0051] In addition, in some embodiments, the image encoder of the CLIP model can use a deep convolutional neural network as its basic architecture, such as ResNet-50, etc., which is not limited in this application. The image encoder can perform a series of convolution, pooling, and activation operations on the image to extract image features and convert them into visual semantic vectors with the same dimension and distribution as the text semantic vector, thereby providing a basis for subsequent cross-modal image-text matching.

[0052] Next, combine Figure 3 This paper introduces in detail the process of generating text semantic vectors based on search texts through a text encoder with the above structure. Figure 3 As shown, the text semantic vector can be obtained through the following steps:

[0053] S301: The word segmenter adds the phrase to the vocabulary.

[0054] In related technologies, the word list of a tokenizer typically contains Chinese characters. When tokenizing search text, the tokenizer breaks down phrases with continuous meanings into individual characters, which can easily confuse the meanings of phrases and individual characters. For example, a tokenizer might break down the word "water cup" into "water" and "cup," but the meaning of "water" is different from that of "water cup." Another example is a tokenizer might break down the word "backpack" into "back" and "bag," but the meaning of "back" is different from that of "backpack."

[0055] Therefore, in some embodiments, phrases with continuous meanings can be added to the word list of the word segmenter (the word list after adding the phrases can also be called the added word list) to facilitate obtaining word segmentation results including phrases and retaining the semantics of the phrases in the search text.

[0056] In some embodiments, step S301 can be understood as a preparatory step before generating a text semantic vector based on the search text, and can be completed in advance.

[0057] S302: The word segmenter obtains the search text input by the user.

[0058] In some embodiments, the search text may be text entered by the user into the search box of the gallery application, or may be text entered by the user into the search box of the browser application, and this application does not limit this.

[0059] It should be understood that the above-mentioned use of search text to match images (which can be called text matching image task) is one of the scenarios of image-text matching task. Image-text matching task can also include using image to match text task.

[0060] In some embodiments, before performing the image-to-text task, the word segmenter can pre-acquire multiple texts, and then the text encoder can generate corresponding text semantic vectors for each of these texts. When performing the image-to-text task, these text semantic vectors can be used to match the image to be matched, thereby achieving the image-to-text task.

[0061] S303: The word segmenter segments and serializes the search text to obtain a text unit sequence.

[0062] When the word segmenter performs word segmentation on the search text, it will split the search text into multiple text units called tokens. A text unit token can be a single word or a phrase.

[0063] In some embodiments, the word segmenter may use a maximum forward match query method to search the vocabulary. For example, it may start with the first character of the search text and attempt to match the longest phrase in the vocabulary. If the match is successful, the phrase is separated from the search text as a text unit and the remaining text is processed. If the match fails, the second longest phrase in the vocabulary is attempted to match, and so on.

[0064] For example, assuming the search text is "a water cup on the table" and the maximum length of a phrase in the vocabulary is 5, we will first select 5 consecutive characters from the first character of the search text "a water cup on the table" for matching. If the match fails, we will select 4 consecutive characters from the first character "a water cup on the table" for matching until a match is successful. Assuming that "table" is matched successfully, we will cut "table" from "a water cup on the table" as a text unit, and continue matching the remaining text, eventually obtaining 6 text units of "table", "on", "of", "one", "one", and "water cup" as the word segmentation results.

[0065] The word segmenter then serializes the word segmentation results, that is, converts the word segmentation results into a preset format that can be transmitted for subsequent processing. For example, the preset format can be a binary format, JSON, XML or other data format, which is not limited in this application.

[0066] For example, the word segmenter can serialize "table", "on", "of", "one", "a", and "cup" to obtain '[CLS]A cup on the table[PAD]...[PAD]' as a text unit sequence. [CLS] indicates a special word at the beginning of the sequence. The text unit sequence is usually of a preset length, and [PAD] is used to pad the edges of the sequence. Assuming the preset length of the text unit sequence is 50, [CLS] and "table", "on", "of", "one", "a", and "cup" occupy 7 positions, so 43 [PAD]s are required for padding.

[0067] The word segmenter in the related art will segment and serialize "a cup of water on the table" to obtain '[CLS]A cup of water on the table[PAD][PAD]……[PAD]'. Assuming that the preset length of the text unit sequence is 50, [CLS] and "table", "on", "of", "one", "water", and "cup" occupy 8 positions, so 42 [PAD] need to be filled.

[0068] S304: The word segmenter maps the text unit sequence into an ID sequence based on the vocabulary.

[0069] In some embodiments, the word segmenter's vocabulary will assign a unique ID to the stored words or phrases, etc. In the embodiment of the present application, the mapping relationship between words and IDs can be called a first mapping relationship, and the mapping relationship between phrases and IDs can be called a second mapping relationship.

[0070] The word segmenter can then search the word table to match corresponding IDs for multiple text units in the text unit sequence, thereby obtaining an ID sequence.

[0071] For example, through the mapping relationship between text units and IDs stored in the vocabulary, it can be matched that the IDs corresponding to "table", "on", "of", "one", "one" and "cup" are "101", "890", "954", "100", "380" and "22230" respectively, so the ID sequence is [101 890 954 100 380 22230].

[0072] S305: The splitting module splits the ID sequence into single-word ID subsequences and phrase ID subsequences.

[0073] Should be understood that, if the word list of word segmenter includes newly added phrases, then the ID range corresponding to individual words and the ID range corresponding to phrases can be determined. Exemplarily, the word list includes 20,000 individual words at the beginning, which are mapped one by one with ID1-20,000, and 5,000 phrases are subsequently added, which are mapped one by one with ID20,001-25,000. Based on the ID size in the ID sequence, the splitting module can determine the ID corresponding to individual words and the ID corresponding to phrases, and the individual word ID with ID in [1, 20,000] is composed of an individual word ID subsequence, and the phrase ID with ID in [20,001, 25,000] is composed of a phrase ID subsequence, so that [101, 890, 954, 100,380] can be obtained as an individual word ID subsequence (also referred to as a first identification sequence), and

[22230] as a phrase ID subsequence (also referred to as a second identification sequence).

[0074] S306: The original embedding module of the embedding layer performs word vector mapping on the single-word ID subsequence to obtain an initial single-word text vector, and the newly added embedding module of the embedding layer performs word vector mapping on the phrase ID subsequence to obtain an initial phrase text vector.

[0075] like Figure 4As shown, in the related art, the embedding layer includes an embedding module, which directly performs word vector mapping on the ID sequence [101 890 954 22230 100] to obtain the initial text vector [word vector a, word vector b, word vector c, word vector d, word vector e]. However, this requires training the entire embedding layer, which requires more training data and graphics card training resources. It may also interfere with the word vector mapping ability of the CLIP model's embedding layer for single words when training with newly added phrases as training samples.

[0076] Therefore, in an embodiment of the present application, the embedding module is split into the original embedding module and the newly added embedding module, so as to train only the word vector mapping ability of the newly added embedding module for the newly added phrases.

[0077] like Figure 5 As shown in the figure, assuming that the ID sequence is [101 890 954 22230 100], the splitting module can first split it into [101 890 954 100] and

[22230] . The original embedding module then performs word vector mapping on [101 890 954 100] to obtain [word vector 1, word vector 2, word vector 3, word vector 4]. The newly added embedding module performs word vector mapping on

[22230] to obtain [word vector 5].

[0078] S307: The merging module merges the initial single-word text vector and the initial phrase text vector to obtain an initial text vector based on the original position of each text unit in the search text.

[0079] It should be understood that the positions of phrases and words in the search text are not fixed. In order to avoid affecting the semantics of the search text, the initial word text vector and the initial phrase text vector need to be merged according to their original positions.

[0080] like Figure 5 As shown, the merging module puts word vector 5 between word vector 3 and word vector 4, and obtains [word vector 1, word vector 2, word vector 3, word vector 5, word vector 4] as the initial text vector.

[0081] S308: The transformer encoder encodes the initial text vector to obtain a text encoding vector.

[0082] S309: The mapping layer performs vector mapping on the text encoding vector to obtain a text semantic vector corresponding to the search text.

[0083] It should be understood that in order to improve the model performance, generalization ability and adaptability of the CLIP model, it is necessary to train the CLIP model before applying it. Figure 6 and Figure 7 This paper introduces the training process of the CLIP model in detail.

[0084] In some embodiments, the image encoder of the CLIP model and the text encoder of the CLIP model may be trained together based on a contrastive learning training method.

[0085] In some embodiments, the training process of the CLIP model may include a pre-training phase and a fine-tuning phase.

[0086] The pre-training phase of the CLIP model can be broken down into the following steps 1-7.

[0087] In some embodiments, as Figure 6 As shown in the figure, during the pre-training stage, the embedding layer of the text encoder includes an embedding module (also called the third embedding module), the vocabulary of the word segmenter contains single words, and the model parameters of the text encoder and image encoder included in the CLIP model (also called the CLIP model to be pre-trained) are all adjustable parameters.

[0088] Step 1: Obtain image training samples and text training samples corresponding to the image training samples.

[0089] Both the image encoder and the text encoder of the CLIP model require pre-training with a large number of training samples. Therefore, before model training, it is necessary to obtain training samples for the image encoder of the CLIP model, namely, image training samples (also called first image training samples), and training samples for the text encoder of the CLIP model, namely, text training samples corresponding to the image training samples (also called first text training samples).

[0090] In some embodiments, the image training samples may include picture training samples and video frame training samples. A video frame refers to any frame of a video. A frame is a still picture in a video, and continuous frames can form a video. A picture training sample can be any picture, and a video frame training sample can be any video frame in a video. The text training sample corresponding to the image training sample refers to the text corresponding to the content displayed by the image training sample, that is, the text training sample can express the content displayed by the image training sample. For example, the image training sample is Figure 1 For the picture 1 shown in the figure, the corresponding text training sample can be: "cup of water on the table".

[0091] It should be noted that this application does not limit the method for obtaining text training samples corresponding to image training samples.

[0092] For example, the text training samples corresponding to the image training samples can be manually annotated based on the human's understanding of the image semantics of the image training samples. For another example, the text training samples corresponding to the image training samples can be automatically generated by identifying relevant content such as objects, scenes, and actions in the image training samples. For another example, the text training samples corresponding to the image training samples can be automatically generated by a text generation model used to generate descriptive text for images.

[0093] In addition, it should be noted that this application does not limit the number of image training samples. It is understandable that the text training samples correspond to the image training samples, so the number of the two is the same.

[0094] like Figure 6 As shown, N image training samples can be obtained, and N text training samples corresponding to the N image training samples can be obtained. For example, image training sample 1 corresponds to text training sample 1.

[0095] Step 2: Input the image training sample into the image encoder, and the image encoder outputs the visual semantic vector corresponding to the image training sample.

[0096] like Figure 6 As shown, N image training samples are input into the image encoder to obtain the visual semantic vectors I1, I2, I3...I corresponding to the N image training samples. N .

[0097] Step 3: Input the text training sample into the word segmenter to obtain the ID sequence corresponding to the text training sample.

[0098] like Figure 6 As shown in FIG, N text training samples are input into the word segmenter, and the ID sequences corresponding to the N text training samples can be obtained.

[0099] In some embodiments, the ID sequence may be obtained by performing word segmentation, serialization, and ID mapping on a text training sample by a word segmenter. For implementation, reference may be made to the above embodiments and will not be repeated here.

[0100] Step 4: Input the ID sequence of the text training sample into the text encoder, and the text encoder outputs the text semantic vector corresponding to the text training sample.

[0101] like Figure 6 As shown in the figure, the ID sequences corresponding to the N text training samples are input into the text encoder to obtain the text semantic vectors T1, T2, T3...T corresponding to the N text training samples.N .

[0102] In some embodiments, after the text encoder receives the ID sequence of the text training sample, it outputs the text semantic vector through the embedding module of the embedding layer, the transformer encoder of the transformer layer, and the mapping layer. The implementation method can be found in the above embodiment and will not be repeated here.

[0103] Step 5: Combine each visual semantic vector with multiple text semantic vectors to obtain multiple vector pairs. From the multiple vector pairs, determine the vector pairs with corresponding relationships as positive sample vector pairs, and determine the remaining vector pairs as negative sample vector pairs.

[0104] Contrastive learning is an unsupervised training method, so it is necessary to define positive samples and negative samples from the training samples. In the embodiment of the present application, positive sample vector pairs and negative sample vector pairs are determined from multiple vector pairs.

[0105] In some embodiments, assuming there are N visual semantic vectors and N textual semantic vectors, each visual semantic vector is combined with the N textual semantic vectors to obtain N×N vector pairs. It is understood that image training samples and textual training samples have a corresponding relationship, and therefore the vectors corresponding to them also have a corresponding relationship. Among the N×N vector pairs, the vector pairs consisting of the visual semantic vectors and textual semantic vectors with a corresponding relationship are determined as positive sample vector pairs, that is, there can be N positive sample vector pairs; the remaining vector pairs are determined as negative sample vector pairs, that is, there can be N×(N-1) negative sample vector pairs.

[0106] like Figure 6 As shown, taking I1 as an example, it is related to T1, T2, T3...T N Combining them respectively, we get I1˙T1, I1˙T2, I1˙T3…I1˙T N These N vector pairs, I2, I3...I N Similarly, we can get N×N vector pairs. N ˙T N Such vector pairs with corresponding relationships are determined as positive sample vector pairs, and the rest are determined as negative sample vector pairs.

[0107] Step 6: Calculate the vector similarity between the visual semantic vector and the textual semantic vector in each vector pair.

[0108] For example, the vector cosine similarity between the visual semantic vector and the textual semantic vector in each vector pair may be calculated.

[0109] Step 7: Based on the loss function, the vector similarity corresponding to the positive sample vector pair, and the vector similarity corresponding to the negative sample vector pair, adjust the parameters of the image encoder and the text encoder.

[0110] It can be understood that in the embodiment of the present application, the model training goal is to maximize the similarity between the vectors corresponding to the positive sample vectors and minimize the similarity between the vectors corresponding to the negative sample vectors.

[0111] In addition, in some embodiments, the true vector similarity corresponding to the positive sample vector pair can be represented as 1, and the true vector similarity corresponding to the negative sample vector pair can be represented as 0, and the parameters of the image encoder and the text encoder can be adjusted until the prediction vector similarity corresponding to the positive sample vector pair can approach 1 to the greatest extent, and the prediction vector similarity corresponding to the negative sample vector pair can approach 0 to the greatest extent, that is, the value of the loss function (also referred to as the first preset loss function) is minimized.

[0112] Step 8: When the training cutoff condition is met, the training is terminated to obtain the pre-trained CLIP model.

[0113] Exemplarily, the training cutoff condition (also referred to as the pre-training cutoff condition) may be that a preset number of training times is reached during the model training process, or that the loss value of the loss function during the model training process is less than a loss value threshold, etc.

[0114] Next, the fine-tuning stage of the CLIP model is introduced.

[0115] The fine-tuning phase of the CLIP model can also be broken down into steps 1 to 8.

[0116] In some embodiments, as Figure 7 As shown in the figure, in the fine-tuning stage, the embedding layer of the text encoder is split into the original embedding module and the newly added embedding module. The parameters of the newly added embedding module are adjustable. In addition, the parameters (also called model parameters) of the CLIP model (also called the CLIP model to be fine-tuned) are the same as those of the pre-trained CLIP model mentioned above, and the parameters of the original embedding module are the same as those of the pre-trained CLIP model. Figure 6 The parameters of the embedding module are the same and fixed. During fine-tuning, only the parameters of the newly added embedding module (also called the first model parameters) are adjusted. The word list of the word segmenter contains single words and newly added phrases.

[0117] Step 1: Obtain image training samples and text training samples corresponding to the image training samples.

[0118] In some embodiments, compared to the pre-training stage, the fine-tuning stage requires fewer parameters to be adjusted, so the CLIP model can be trained using a smaller number of training samples than the aforementioned N text training samples and N image training samples.

[0119] For example, Figure 7 As shown, M image training samples (also referred to as second image training samples) can be obtained, and M text training samples (also referred to as second text training samples) corresponding one to one to the M image training samples can be obtained, where M is an integer less than N.

[0120] Step 2: Input the image training sample into the image encoder, and the image encoder outputs the visual semantic vector corresponding to the image training sample.

[0121] like Figure 7 As shown, M image training samples are input into the image encoder to obtain the visual semantic vectors I1, I2, I3...I corresponding to the M image training samples. M .

[0122] Step 3: Input the text training sample into the word segmenter to obtain the ID sequence corresponding to the text training sample.

[0123] like Figure 7 As shown in FIG, M text training samples are input into the word segmenter, and the ID sequences corresponding to the M text training samples can be obtained.

[0124] Step 4: Input the ID sequence of the text training sample into the text encoder, and the text encoder outputs the text semantic vector corresponding to the text training sample.

[0125] like Figure 7 As shown in the figure, M ID sequences are input into the text encoder to obtain the text semantic vectors T1, T2, T3, ... T corresponding to the M text training samples. M .

[0126] In some embodiments, the model structure of the text encoder is as follows Figure 2 As shown, combined with Figure 3 The steps shown generate corresponding text semantic vectors based on text training samples. The implementation method can be found in the above embodiment and will not be repeated here.

[0127] Step 5: Combine each visual semantic vector with multiple text semantic vectors to obtain multiple vector pairs. From the multiple vector pairs, determine the vector pairs with corresponding relationships as positive sample vector pairs, and determine the remaining vector pairs as negative sample vector pairs.

[0128] Based on the above example, in some embodiments, assuming there are M visual semantic vectors and M text semantic vectors, there may be M positive sample vector pairs and M×(M-1) negative sample vector pairs.

[0129] like Figure 7 As shown, I1˙T1, I2˙T2, I3˙T3…I M ˙T M are positive sample vector pairs, and the rest are determined as negative sample vector pairs.

[0130] Step 6: Calculate the vector similarity between the visual semantic vector and the textual semantic vector in each vector pair.

[0131] Step 7: Based on the loss function, the vector similarity between the positive sample vector pairs, and the vector similarity between the negative sample vector pairs, adjust the parameters of the newly added embedding module in the image encoder.

[0132] The loss function may also be referred to as a second preset loss function. In some embodiments, the second preset loss function may be the same as or different from the first preset loss function of the above embodiment.

[0133] In some embodiments, during the fine-tuning stage of the CLIP model, except for the parameters of the newly added embedding module, the parameters of the CLIP model are the same as the parameters of the trained CLIP model, and these parameters are fixed (also known as freezing the parameters of the pre-trained CLIP model). During the fine-tuning process, only the parameters of the newly added embedding module are adjusted.

[0134] Step 8: When the training cutoff condition is met, the training is terminated to obtain the trained CLIP model.

[0135] In some embodiments, the trained CLIP model may also be referred to as a pre-trained CLIP model.

[0136] It should be noted that other implementations of steps 1 to 8 can refer to steps 1 to 8 in the pre-training phase of the CLIP model, and this application will not go into details here.

[0137] Next, the image-text matching method provided in the embodiment of the present application is introduced in combination with the above-trained CLIP model.

[0138] In some embodiments, the electronic device runs an operating system such as Android. The layered architecture can divide the operating system into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. For example, a mobile phone is an electronic device. Figure 8As shown, the mobile phone may include a gallery service module 810, a search module 820, a multimodal understanding module 830, and a natural language understanding module 840, which are used to implement the image-text matching method provided in the embodiment of the present application. These modules may all be located on the same layer of the electronic device, or they may be located on different layers of the electronic device, or they may be located on multiple layers of the electronic device at the same time, and their functions are implemented through software interfaces between layers. This application does not limit this.

[0139] under Figure 8 Taking the electronic device as a mobile phone, and the scenario in which a user searches for pictures in the gallery application of the mobile phone as an example, the image-text matching method provided in the embodiment of the present application is described in detail.

[0140] like Figure 8 As shown, the image-text matching method provided in the embodiment of the present application can be divided into the following two stages: an index building stage and a search stage.

[0141] First, combine Figure 8 Describes in detail the steps involved in the index building phase.

[0142] S801: The gallery service module 810 receives a user's operation of adding or modifying a picture and its attribute information.

[0143] Adding an image refers to a user storing an image in the Gallery app. For example, this might involve taking a photo, downloading a photo, or capturing a screenshot. Modifying an image refers to a user modifying an image already stored in the Gallery app. For example, this might involve cropping, splicing, adding special effects, or adding subtitles.

[0144] In some embodiments, the attribute information of an image may include: the location where the image was acquired, the time the image was acquired, the name of the person, a classification tag, and an event, etc., which is not limited in this application. The classification tag can be used to indicate the type of object shown in the image, such as a person, plant, animal, building, or natural scenery. The event can be used to indicate what the object shown in the image does. For example, the event can be a game, a sport, etc.

[0145] In some embodiments, the classification label can be manually configured by the user, or can be obtained by the mobile phone automatically classifying the image, and this application does not limit this.

[0146] S802: The gallery service module 810 stores pictures and their attribute information.

[0147] The gallery service module 810 stores the pictures and their attribute information in the mobile phone in response to the user's operation of adding or modifying the pictures and their attribute information.

[0148] For example, the picture and its attribute information may be stored in a local file of the mobile phone, and the user may view the picture through various channels such as a local folder of the mobile phone and a gallery application.

[0149] As another example, with the user's authorization, the picture and its attribute information can be stored in the cloud for backup, thereby reducing the memory pressure of the mobile phone.

[0150] S803: The gallery service module 810 calls the multimodal understanding module 830 to perform visual semantic understanding on the image.

[0151] Visual semantic understanding refers to the ability of a mobile phone to understand the meaning of the content displayed in an image, such as the type, quantity, location, and relationship between objects in the image.

[0152] It should be understood that since visual semantic understanding requires a large amount of computing resources, in some embodiments, in order not to affect user use, step S803 can be performed when the mobile phone is in a charging state with the screen off.

[0153] In some embodiments, the multimodal understanding module 830 can implement visual semantic understanding based on the CLIP model trained in the above embodiments, input the image into the image encoder of the CLIP model, and generate a visual semantic vector corresponding to the image.

[0154] Based on the above introduction, the image encoder and text encoder of the CLIP model can map images and search texts into vectors of the same dimension and distribution, facilitating subsequent matching between visual semantic vectors and text semantic vectors.

[0155] S804: The multimodal understanding module 830 returns the visual semantic vector of the image to the gallery service module 810.

[0156] S805: The image library service module 810 stores the visual semantic vector of the image.

[0157] After receiving the visual semantic vector of the image returned by the multimodal understanding module 830, the image library service module 810 can store it. In some embodiments, the image library service module 810 is configured with a database, and the image library service module 810 can store the visual semantic vector of the image in the database.

[0158] S806 : The gallery service module 810 sends the visual semantic vector and attribute information of the image to the search module 820 .

[0159] In some embodiments, the gallery service module 810 may store the visual semantic vectors of the images returned by the multimodal understanding module 830 , and then send the attribute information of the images and their visual semantic vectors to the search module 820 in batches.

[0160] S807: The search module 820 constructs an index corresponding to the image.

[0161] In some embodiments, the search module 820 may combine the visual semantic vector of the image and the attribute information of the image to construct an index corresponding to the image. Based on the above example, the index of the image may include its visual semantic vector, as well as one or more of the acquisition time, acquisition location, person name, classification label, or event, etc., which is not limited in this application.

[0162] Exemplarily, the search module 820 may include an index library, and the search module 820 may store the index corresponding to the image in the index library, so that subsequent search and matching can be performed based on the index library.

[0163] Next, combine Figure 8 Continue with the details of the steps involved in the search phase.

[0164] S808: The gallery service module 810 receives the user's input operation for the search text.

[0165] The user can enter the search text in the search interface provided by the mobile phone. For example, the user can enter the search text in the search interface provided by the mobile phone. Figure 1 The interface 3 shown includes a search box in which a search text is input, such as "water cup", or other search texts such as "water cup on the table".

[0166] S809 : The gallery service module 810 sends the search text to the search module 820 .

[0167] S810: The search module 820 calls the multimodal understanding module 830 to perform text semantic understanding on the search text.

[0168] The search module 820 calls the multimodal understanding module 830 to perform text semantic understanding on the search text to obtain a text semantic vector of the search text.

[0169] Text semantic understanding refers to enabling the mobile phone to understand the meaning expressed in the text.

[0170] In some embodiments, the multimodal understanding module 830 provides the trained CLIP model introduced in the above embodiment, which can be combined with Figure 2 The model structure of the text encoder shown, and Figure 3In the steps shown, the search text is first input into the word segmenter to obtain an ID sequence, and then the ID sequence is input into the text encoder of the CLIP model to obtain the text semantic vector corresponding to the search text.

[0171] S811: The modality understanding module 830 returns the text semantic vector corresponding to the search text to the search module 820.

[0172] S812: The gallery service module 820 calls the natural language understanding module 840 to perform named entity recognition on the search text.

[0173] In some embodiments, the natural language understanding module 840 may perform named entity recognition on the search text based on a natural language understanding model to obtain entities contained in the search text.

[0174] For example, an entity may include: time, place, person's name, category label, etc. Based on the above example "a water cup on the table", the entity included is "water cup".

[0175] S813: The natural language understanding module 840 returns the entities contained in the search text to the search module 820.

[0176] S814: The search module 820 performs a recall in the index library based on the text semantic vector and entity corresponding to the search text.

[0177] The recall of text semantic vectors (also called vector recall) refers to recalling the indexes in the index library that match the text semantic vectors corresponding to the search text.

[0178] In some embodiments, vector similarities can be calculated between the text semantic vector corresponding to the search text and the visual semantic vectors included in multiple indexes in the index library to obtain similarity calculation results corresponding to the multiple indexes. K indexes with higher vector similarities among the multiple similarity calculation results are used as vector recall results. Where K is an integer greater than 0. For example, K can be a pre-set number of vector recall results, such as 5, 8, or 10.

[0179] Vector similarity refers to the degree of similarity between two vectors and can be calculated using a variety of methods. For example, the degree of similarity can be determined by calculating the cosine similarity of the two vectors. Other methods may also be used, and this application does not limit this.

[0180] Entity recall (also called entity recall) refers to recalling the index in the index library that matches the entity in the search text.

[0181] Based on the introduction of the above embodiment, the index may include attribute information of the image, and the attribute information includes entities. For example, entities may be included in the acquisition time, acquisition location, classification label, etc. Taking the search text as "the sky photographed in city B on National Day" as an example, there is an entity "city B" for the location, an entity "National Day" for the time, and an entity "sky" related to the content of the picture shown in the picture. Then, matching can be performed among the entities corresponding to multiple indexes, and an index matching the entity in the search text can be obtained as the entity recall result.

[0182] In some embodiments, the intersection results or union results of the vector recall results and the entity recall results may be sorted, and the sorted results may be used as search results.

[0183] S815: The search module 820 returns the search results to the gallery service module 810.

[0184] The search module 820 returns the sorted search results to the gallery service module 810 .

[0185] S816: The gallery service module 810 displays the search results to the user.

[0186] Furthermore, it is understood that in actual applications, mobile phones typically store images and videos in the gallery application. Therefore, when searching in the search interface provided by the gallery application, both image and video search results can be displayed simultaneously. In other words, the index library can include indexes corresponding to videos in addition to indexes corresponding to images.

[0187] In some embodiments, a video may correspond to multiple indexes, that is, a video may be divided into multiple video segments, a video frame may be selected from each video segment, its corresponding visual semantic vector may be determined through the CLIP model, and the visual semantic vector may be constructed as the index corresponding to the video segment, then the video search result may be the video segment.

[0188] In some embodiments, the index corresponding to the video segment may further include attribute information of the video segment.

[0189] It should be noted that the location of the gallery service module 810, search module 820, multimodal understanding module 830, and natural language understanding module 840 on a mobile phone or other electronic device is merely an example; these modules can also be located on a cloud server, with the cloud server utilizing the interaction of these four modules to implement the steps involved in the index building phase. During the search phase, the steps involved in the search phase can be implemented based on the interaction between a mobile phone or other electronic device and the cloud server. This application does not impose any limitations on this.

[0190] For example, the mobile phone can send the search text entered by the user to the gallery service module of the cloud server, so that the gallery service module of the cloud server interacts with other modules to implement the steps included in the search stage. The gallery service module of the cloud server then sends the search results to the mobile phone, so that the mobile phone displays the search results to the user.

[0191] It should be noted that the above scenarios are only examples, and the image-text matching method provided in the embodiments of the present application can also be applied to other scenarios.

[0192] In some embodiments, the image-text matching method provided in this application can also implement the task of matching text based on an image, that is, the image-text matching task mentioned above.

[0193] In a possible implementation, the steps for the electronic device to implement the above-mentioned image-to-text matching task may also include: an index building phase and a search phase.

[0194] During the index construction stage, the electronic device can first obtain multiple texts; then perform text semantic understanding on the multiple texts through the text encoder in the CLIP model trained in the embodiment of the present application, and generate text semantic vectors corresponding to the multiple texts; then the electronic device constructs the text semantic vector into an index corresponding to the text and stores it in the text index library.

[0195] For example, the above-mentioned multiple texts can be stored in a local file of the mobile phone or in a cloud server, which is not limited in this application.

[0196] During the search phase, the electronic device can obtain the picture that needs to be searched and matched, and call the image encoder in the trained CLIP model to perform visual semantic understanding of the picture, and generate a visual semantic vector corresponding to the picture; then the electronic device performs vector recall in the text index library based on the visual semantic vector, and obtains the text corresponding to at least one text semantic vector that matches the visual semantic vector as the search result; finally, the electronic device can display the search results to the user.

[0197] Exemplarily, the above-mentioned acquisition of the picture to be matched may be that the user drags the picture into the picture search box, or that the gallery application receives the user's selection operation for the picture stored in the gallery application, which is not limited in this application.

[0198] In some embodiments, the electronic device can intelligently caption images stored in a gallery application. After a user selects an image and triggers the gallery application's intelligent captioning function, the electronic device can use the CLIP model's image encoder to perform visual semantic understanding of the image, generate its corresponding visual semantic vector, and then determine a text semantic vector that matches the visual semantic vector from pre-stored text semantic vectors corresponding to multiple texts. The text corresponding to the matching text semantic vector is then presented to the user as the intelligent captioning result.

[0199] For example, the text semantic vectors corresponding to the above-mentioned multiple pre-stored texts may be pre-stored in a cloud server or pre-stored in a local file of the mobile phone, and this application does not limit this.

[0200] It should be noted that this application does not limit the type of electronic device. For example, the electronic device may be a mobile phone, tablet computer, desktop computer, laptop computer, notebook computer, ultra-mobile personal computer (UMPC), handheld computer, netbook computer, personal digital assistant (PDA), wearable electronic device, smart watch, etc. The electronic device may also be a server, such as an independent physical server, a server cluster or distributed system composed of multiple physical servers, and a cloud server. This application does not impose any special restrictions on the specific form of the above electronic devices.

[0201] An embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed by a computer, can implement one or more steps in any of the above-mentioned image-text matching methods.

[0202] The computer readable storage medium may be a non-transitory computer readable storage medium, for example, a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0203] Another embodiment of the present application further provides a computer program product comprising instructions, which, when executed by a computer, can implement one or more steps in any of the above-mentioned image-text matching methods.

[0204] The electronic device, computer-readable storage medium, and computer program product provided in this embodiment are all used to execute the corresponding image-text matching method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding image-text matching method provided above, and will not be repeated here.

[0205] The terms "first", "second" and "third" in the specification, claims and drawings of this application are used to distinguish different objects rather than to limit a specific order.

[0206] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0207] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for image-text matching, characterized in that: include: Get text; Performing word segmentation processing on the text to obtain a word segmentation result; the word segmentation result includes single words and phrases, and a phrase includes at least two single words; Determining a feature vector corresponding to the single word using a first embedding module of a pre-trained text encoder, and determining a feature vector corresponding to the phrase using a second embedding module of the pre-trained text encoder; the first embedding module and the second embedding module are trained separately at different training stages; Performing encoding processing on the feature vectors corresponding to the single words and the feature vectors corresponding to the phrases to determine a text semantic vector corresponding to the text; Image-text matching is performed based on the text semantic vector.

2. The method according to claim 1, characterized in that The pre-trained text encoder belongs to a pre-trained CLIP model; the pre-trained CLIP model is trained by the following steps: Acquire a first image training sample and a first text training sample corresponding to the first image sample; Based on the first text training sample, the first image training sample, and a first preset loss function, a first model parameter of a second embedding module included in a CLIP model to be fine-tuned is adjusted, and model parameters of the CLIP model to be fine-tuned other than the first model parameter are kept unchanged, thereby training to obtain the pre-trained CLIP model; the CLIP model to be fine-tuned includes a first embedding module and a second embedding module.

3. The method according to claim 2, characterized in that The first embedding module and the second embedding module included in the CLIP model to be fine-tuned are obtained by the following steps: Acquire a second image training sample and a second text training sample corresponding to the second image sample; Adjusting model parameters of a CLIP model to be pre-trained based on the second text training sample, the second image training sample, and a second preset loss function; the CLIP model to be pre-trained includes a third embedding module; When the CLIP model after parameter adjustment meets the pre-training cutoff condition, the third embedding module included in the CLIP model after parameter adjustment is split to obtain the first embedding module and the second embedding module included in the CLIP model to be fine-tuned.

4. The method according to claim 1, wherein The method further comprises: Adding multiple phrases to the vocabulary of the word segmenter to obtain an added vocabulary; the vocabulary includes multiple words; The word segmentation processing of the text to obtain a word segmentation result includes: Based on the added vocabulary, the text is segmented by the word segmenter to obtain the word segmentation result.

5. The method according to claim 4, characterized in that The added vocabulary also includes a first mapping relationship between the multiple words and the multiple identifiers in a one-to-one correspondence, and a second mapping relationship between the multiple phrases and the multiple identifiers in a one-to-one correspondence. The method further includes: Based on the first mapping relationship and the second mapping relationship, performing identifier mapping on the word segmentation result to obtain an identifier sequence corresponding to the word segmentation result; The determining of the feature vector corresponding to the single word by a first embedding module of a pre-trained text encoder, and the determining of the feature vector corresponding to the phrase by a second embedding module of the pre-trained text encoder, comprises: Based on the identification sequence, a feature vector corresponding to the single word is determined by a first embedding module of a pre-trained text encoder, and a feature vector corresponding to the phrase is determined by a second embedding module of the pre-trained text encoder.

6. The method according to claim 5, characterized in that The step of determining the feature vector corresponding to the single word by using a first embedding module of a pre-trained text encoder based on the identification sequence, and determining the feature vector corresponding to the phrase by using a second embedding module of the pre-trained text encoder, comprises: Splitting the identification sequence to obtain a first identification sequence for representing the single word and a second identification sequence for representing the phrase; Performing feature extraction on the first identification sequence using a first embedding module of the pre-trained text encoder to obtain a feature vector corresponding to the single word, and performing feature extraction on the second identification sequence using a second embedding module of the pre-trained text encoder to obtain a feature vector corresponding to the phrase; The encoding process of the feature vectors corresponding to the single words and the feature vectors corresponding to the phrases to determine the text semantic vector corresponding to the text includes: Based on the position of the single word in the text and the position of the phrase in the text, merging the feature vector corresponding to the single word and the feature vector corresponding to the phrase to obtain a feature vector corresponding to the text; The feature vector corresponding to the text is encoded to obtain a text semantic vector corresponding to the text.

7. The method according to any one of claims 1 to 6, characterized in that The performing image-text matching based on the text semantic vector includes: Based on the text semantic vector, an image matching the text is determined from a plurality of pre-stored images.

8. The method according to claim 7, characterized in that The plurality of pre-stored images include visual semantic vectors corresponding to the plurality of images respectively; and determining an image matching the text from the plurality of pre-stored images based on the text semantic vector includes: Calculating similarities between the plurality of visual semantic vectors and the text semantic vector respectively, and obtaining similarity calculation results corresponding to the plurality of images respectively; Sort multiple similarity calculation results in descending order; K images ranked in top K in similarity calculation results are determined from the multiple images as images matching the text; K is a positive integer, and K is less than or equal to the number of the multiple images.

9. The method according to claim 8, characterized in that The visual semantic vectors corresponding to the multiple images are obtained through an image encoder of a pre-trained CLIP model.

10. The method according to any one of claims 1 to 6, characterized in that The text includes multiple texts; before performing image-text matching based on the text semantic vector, the method further includes: Acquire images; determining a visual semantic vector of the image; The method comprises: obtaining a text; performing word segmentation processing on the text to obtain a word segmentation result; determining a feature vector corresponding to the single word through a first embedding module of a pre-trained text encoder, and determining a feature vector corresponding to the phrase through a second embedding module of the pre-trained text encoder; encoding the feature vector corresponding to the single word and the feature vector corresponding to the phrase to determine a text semantic vector corresponding to the text; and performing image-text matching based on the text semantic vector, including: Get multiple texts; Performing word segmentation processing on the multiple texts respectively to obtain word segmentation results corresponding to the multiple texts respectively; For each word segmentation result corresponding to the text, determine the feature vector corresponding to the single word through the first embedding module of the pre-trained text encoder, and determine the feature vector corresponding to the phrase through the second embedding module of the pre-trained text encoder; For each word segmentation result corresponding to the text, encoding the feature vector corresponding to the single word and the feature vector corresponding to the phrase is performed to determine the text semantic vector corresponding to the text; Based on the visual semantic vector and multiple text semantic vectors, text matching the image is determined from the multiple texts.

11. The method according to any one of claims 1 to 10, characterized in that The obtaining of text includes: Get the text sent by the terminal; After the image-text matching based on the text semantic vector, the method further includes: The image-text matching result is sent to the terminal.

12. An electronic device, characterized in that: including memory and processor; The memory is coupled to the processor, and the memory is used to store computer program code, where the computer program code includes computer instructions. One or more of the processors call the computer instructions to enable the electronic device to execute the image-text matching method according to any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image-text matching method according to any one of claims 1 to 11 is implemented.

Images