Cross-modal retrieval method and device, equipment and storage medium

Abstract

The invention relates to the technical field of computers, and provides a cross-modal retrieval method and device, equipment and a storage medium. The method comprises the steps of obtaining a retrieval text, and extracting a target text feature of the retrieval text; for each to-be-retrieved image, respectively executing the following operations to obtain an image matching result corresponding to each to-be-retrieved image: respectively carrying out similarity matching on a global feature and a plurality of local features of one to-be-retrieved image and the target text feature to obtain a corresponding feature matching result; selecting one feature matching result meeting a preset similarity matching condition from the plurality of feature matching results corresponding to the one to-be-retrieved image as an image matching result corresponding to the one to-be-retrieved image; based on the obtained image matching results, at least one to-be-retrieved image meeting a preset retrieval condition is selected from the to-be-retrieved images to serve as a retrieval result corresponding to the retrieval text.

Description

Technical Field

[0001] The present application relates to the field of computer technology and provides a cross-modal retrieval method, apparatus, device and storage medium. Background Art

[0002] Cross-modal retrieval refers to similarity matching and retrieval between different data modalities (such as text, images, audio, etc.). For example, given a query text description, images matching the description are retrieved from a database containing images.

[0003] Existing cross-modal retrieval models primarily employ a training approach similar to the Contrastive Language-Image Pretraining (CLIP) model. Specifically, during training, visual and textual features are extracted from a large number of image-text pairs, aligned, and supervised using a contrastive loss function to train the cross-modal retrieval model's cross-modal retrieval capabilities. The contrastive loss function aims to maximize the similarity of matching image-text pairs while simultaneously maximizing the similarity of mismatched image-text pairs.

[0004] However, when using the above training method for model training, it is usually necessary to learn the similarity between the features of the entire image and the features of a complete text. This training method is more suitable for simple images and search terms. Once the images in the retrieval task are more complex, the problem of low retrieval hit rate will arise. For example, assuming that the elements that match the search term in image A only account for 10%, then since the similarity comparison is performed by comparing the text features of the search term with the visual features of the entire image of image A, the elements that are not related to the search term and account for 90% of image A will lower the similarity between image A and the search term, resulting in even if there are elements in image A that match the search term, they will not be hit during retrieval.

[0005] Therefore, how to improve the cross-modal retrieval capability of cross-modal retrieval models for complex retrieval tasks is an urgent issue. Summary of the Invention

[0006] The embodiments of the present application provide a cross-modal retrieval method, apparatus, device, and storage medium to address the problem of poor cross-modal retrieval capability of cross-modal retrieval models for complex retrieval tasks in related technologies.

[0007] In a first aspect, an embodiment of the present application provides a cross-modal retrieval method, comprising:

[0008] Obtaining a search text and extracting target text features of the search text;

[0009] For each image to be retrieved, perform the following operations to obtain the image matching results corresponding to each image to be retrieved:

[0010] Performing similarity matching on the global features and multiple local features of an image to be retrieved and the target text features respectively to obtain corresponding feature matching results; the local features represent: visual features of a specified area of ​​the image to be retrieved;

[0011] Selecting, from among the plurality of feature matching results corresponding to the image to be retrieved, a feature matching result that meets a preset similarity matching condition as the image matching result corresponding to the image to be retrieved;

[0012] Based on the obtained image matching results, at least one image to be retrieved that meets the preset retrieval conditions is selected from the images to be retrieved as the retrieval result corresponding to the retrieval text.

[0013] In a second aspect, an embodiment of the present application further provides a cross-modal retrieval method and apparatus, including:

[0014] An acquisition unit, used to acquire the search text;

[0015] A feature extraction unit, configured to extract target text features of the search text;

[0016] An image matching unit is configured to perform the following operations for each image to be retrieved, respectively, to obtain an image matching result corresponding to each image to be retrieved: performing similarity matching on a global feature and multiple local features of a image to be retrieved with the target text features, respectively, to obtain corresponding feature matching results; the local features represent visual features of a specified area of ​​the image to be retrieved; and selecting, from the multiple feature matching results corresponding to the image to be retrieved, a feature matching result that meets a preset similarity matching condition as the image matching result corresponding to the image to be retrieved;

[0017] The retrieval unit is configured to select at least one image to be retrieved that meets a preset retrieval condition from among the images to be retrieved based on the obtained image matching results as a retrieval result corresponding to the retrieval text.

[0018] In one possible implementation, the feature extraction unit includes a text encoder and a visual encoder, the target text features are extracted by the text encoder; the global features and the multiple local features of each image to be retrieved are obtained by pre-extracting features of each image to be retrieved by the visual encoder;

[0019] The visual encoder and the text encoder are trained by the training unit through the following method:

[0020] Acquire a training sample set; the training sample set includes a plurality of image samples and a text annotation set including a global text annotation and a local text annotation for each of the plurality of image samples;

[0021] Compressing a pre-trained large visual model through feature distillation to obtain initial parameters of a visual encoder; the large visual model has the same structure as the visual encoder;

[0022] By performing similarity matching on the visual features of each of the multiple image samples extracted by the visual encoder and the text features of each of the global text annotations and each of the local text annotations extracted by the text encoder, the parameters of each of the visual encoder and the text encoder are adjusted until the first training end condition is met, thereby obtaining the trained visual encoder and text encoder.

[0023] In one possible implementation, the training sample set also includes a graph-text matching label, which represents the matching relationship between the image sample and the global text annotations and local text annotations included in the text annotation set; the training unit performs similarity matching between the visual features of each of the multiple image samples extracted by the visual encoder and the text features of each of the global text annotations and each of the local text annotations extracted by the text encoder, and adjusts the parameters of the visual encoder and the text encoder until the first training end condition is met, and the trained visual encoder and text encoder are obtained. The training unit is used to: use the visual encoder to extract the visual features of each of the multiple image samples, and use the text encoder to extract the text features of each of the global text annotations and each of the local text annotations; for each of the extracted visual features, calculate the similarity between it and each of the text features, and adjust the parameters of the visual encoder and the text encoder according to the similarity loss between the similarity and the corresponding graph-text matching label, until the first training end condition is met, and the trained visual encoder and text encoder are obtained.

[0024] In one possible implementation, when the training unit compresses the pre-trained visual large model through feature distillation to obtain the initial parameters of the visual encoder, it is specifically used to: for the multiple image samples, perform the following operations respectively: extract the reference features of an image sample through the visual large model; perform random transformation on the one image sample to obtain multiple augmented samples corresponding to the one image sample; perform feature extraction on the multiple augmented samples through the visual encoder to obtain multiple learning features; adjust the parameters of the visual encoder based on the learning loss between each learning feature and the reference feature until the second training end condition is met, and then determine the initial parameters of the visual encoder.

[0025] In one possible implementation, the visual encoder extracts the visual features of each of the multiple image samples by the following method: for the multiple image samples, the following operations are respectively performed: an image sample is input into the visual encoder, and features of the image sample are extracted through multiple convolutional layers in the visual encoder to obtain global visual features of the image sample; based on preset region division rules, interest region pooling is performed on each local region of the image sample to obtain multiple local visual features of the image sample; and the global visual features of the image sample and multiple local visual features are feature-fused through the embedding layer in the visual encoder to obtain the visual features of the image sample.

[0026] In one possible implementation, when the training unit obtains a training sample set, it is specifically used to: obtain multiple candidate samples, and perform the following operations for each candidate sample: perform image recognition on a candidate sample through the visual large model to obtain the global text annotation and local text annotation of the candidate sample; perform feature extraction on the candidate sample to obtain the global visual features and local visual features of the candidate sample, and perform feature extraction on the global text annotation and local text annotation of the candidate sample to obtain the global text features and local text features of the candidate sample; calculate the similarity between the global visual features and the global text features, and the similarity between the corresponding local visual features and the local text features; when the calculated similarities are all greater than a preset similarity threshold, determine that the candidate sample is an image sample; otherwise, determine that the candidate sample is not an image sample.

[0027] In one possible implementation, when the video to be retrieved is included in the database to be retrieved, the image to be retrieved includes at least one frame of image in the video to be retrieved; then the retrieval unit is further used to determine that the retrieval result includes the video to be retrieved when at least one image to be retrieved that meets the preset retrieval conditions includes an image belonging to the video to be retrieved.

[0028] In a third aspect, an embodiment of the present application further provides a computer device comprising a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the steps of any one of the above-mentioned cross-modal retrieval methods.

[0029] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium comprising a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of any one of the above-mentioned cross-modal retrieval methods.

[0030] In a fifth aspect, an embodiment of the present application further provides a computer program product, comprising computer instructions, which are executed by a processor to perform the steps of any one of the above-mentioned cross-modal retrieval methods.

[0031] The beneficial effects of this application are as follows:

[0032] The embodiments of the present application provide a cross-modal retrieval method, apparatus, device and storage medium. In this method, by dividing the image to be retrieved into multiple local areas and extracting local features corresponding to each local area, features with a smaller proportion in the image to be retrieved can also be extracted, thereby constructing a fine-grained feature expression of the image to be retrieved.

[0033] Furthermore, because the global features and multiple local features of the image to be retrieved are all matched with the target text features for similarity, even if an element in the image to be retrieved accounts for a small proportion, it can be retrieved due to the high similarity when the local features to which it belongs are matched with the target text features. This solves the problem of low retrieval hit rate in related technologies when the images in the retrieval task are complex. This improves the cross-modal retrieval capability of the cross-modal retrieval model for complex retrieval tasks.

[0034] Other features and advantages of the present application will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present application. The purposes and other advantages of the present application can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0036] Figure 1 This is an optional schematic diagram of an application scenario in an embodiment of the present application;

[0037] Figure 2 A schematic diagram of a cross-modal search method according to an embodiment of the present invention;

[0038] Figure 3 A schematic diagram of local features provided in an embodiment of the present application;

[0039] Figure 4 A schematic diagram of the visual encoder network structure provided in an embodiment of the present application;

[0040] Figure 5 An exemplary flow chart of a method for training a visual encoder and a text encoder provided in an embodiment of the present application;

[0041] Figure 6 A flowchart of a method for determining an image sample according to an embodiment of the present application;

[0042] Figure 7A A schematic diagram of a random transformation method provided in an embodiment of the present application;

[0043] Figure 7B A schematic diagram of a random transformation method provided in an embodiment of the present application;

[0044] Figure 7C A schematic diagram of a random transformation method provided in an embodiment of the present application;

[0045] Figure 7D A schematic diagram of a random transformation method provided in an embodiment of the present application;

[0046] Figure 7E A schematic diagram of a random transformation method provided in an embodiment of the present application;

[0047] Figure 8 Schematic diagram of the characteristic distillation process provided in the embodiment of the present application;

[0048] Figure 9 Schematic diagram of the image-text alignment training process for the visual encoder and text encoder provided in an embodiment of the present application;

[0049] Figure 10 Schematic diagram of the structure of a cross-modal retrieval method and device;

[0050] Figure 11 A schematic diagram of the hardware structure of a computer device to which the embodiments of the present application are applied;

[0051] Figure 12 A schematic diagram of the hardware structure of another computer device to which an embodiment of the present application is applied. DETAILED DESCRIPTION

[0052] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of the technical solutions of this application, but not all of them. Based on the embodiments described in this application document, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the technical solutions of this application.

[0053] The following is a brief introduction to the design concept of the embodiment of this application:

[0054] Cross-modal retrieval involves similarity matching and retrieval across different data modalities (e.g., text, images, and audio). For example, given a query text description, images matching that description are retrieved from a database of images. Alternatively, given a query image, text descriptions matching that image are retrieved from a database of text descriptions.

[0055] Existing cross-modal retrieval models primarily employ a training approach similar to the Contrastive Language-Image Pretraining (CLIP) model. In CLIP's training approach, images and text are first mapped to form an image-text pair dataset. Then, for each image-text pair, the similarity is calculated in feature space and supervised using a contrastive loss function. This approach minimizes the similarity of positive samples (image-text pairs where the image and text match) and minimizes the similarity of negative samples (image-text pairs where the image and text do not match), ultimately resulting in a trained cross-modal retrieval model.

[0056] However, when using the above training method for model training, it is usually necessary to learn the similarity between the features of the entire image and the features of a complete text. This training method is more suitable for simple images and search terms. Once the images in the retrieval task are more complex, the problem of low retrieval hit rate will arise. For example, assuming that the elements that match the search term in image A only account for 10%, then since the similarity comparison is performed by comparing the text features of the search term with the visual features of the entire image of image A, the elements that are not related to the search term and account for 90% of image A will lower the similarity between image A and the search term, resulting in even if there are elements in image A that match the search term, they will not be hit during retrieval.

[0057] Therefore, how to improve the cross-modal retrieval capability of cross-modal retrieval models for complex retrieval tasks is an urgent issue.

[0058] In view of this, an embodiment of the present application provides a cross-modal retrieval method, apparatus, device and storage medium. The method includes: obtaining and extracting target text features of the retrieval text. For each image to be retrieved, the following operations are performed to obtain the image matching results corresponding to each image to be retrieved: the global features and multiple local features of an image to be retrieved are respectively matched with the target text features for similarity to obtain corresponding feature matching results; among the multiple feature matching results corresponding to an image to be retrieved, a feature matching result that meets the preset similarity matching conditions is selected as the image matching result corresponding to an image to be retrieved. Finally, based on the obtained image matching results, at least one image to be retrieved that meets the preset retrieval conditions is selected from each image to be retrieved as the retrieval result corresponding to the retrieval text.

[0059] Through the above method, the image to be retrieved can be divided into multiple local areas, and the local features corresponding to each local area can be extracted, so that the features with a smaller proportion in the image to be retrieved can also be extracted, and a fine-grained feature expression of the image to be retrieved can be constructed.

[0060] Furthermore, because the global features and multiple local features of the image to be retrieved are all matched with the target text features for similarity, even if an element in the image to be retrieved accounts for a small proportion, it can be retrieved due to the high similarity when the local features to which it belongs are matched with the target text features. This solves the problem of low retrieval hit rate in related technologies when the images in the retrieval task are complex. This improves the cross-modal retrieval capability of the cross-modal retrieval model for complex retrieval tasks.

[0061] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application can be combined with each other if there is no conflict.

[0062] User terminals include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft, etc. The embodiments of the present invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.

[0063] Figure 1One application scenario is shown, including a physical terminal device 110 and a server 130. Each physical terminal device 110 establishes a communication connection with the server 130 via a wired or wireless network. A person requiring cross-modal search can enter a search text in the display interface 120 of the physical terminal device 110 and send the search text to the server 130. The server 130 retrieves the search text and extracts target text features from the search text. The following operations are then performed for each image to be searched in the server 130 to obtain an image matching result corresponding to each image to be searched: The global features and multiple local features of the image to be searched are matched against the target text features for similarity, obtaining corresponding feature matching results. Among the multiple feature matching results corresponding to the image to be searched, a feature matching result that meets preset similarity matching conditions is selected as the image matching result corresponding to the image to be searched. Finally, based on the obtained image matching results, at least one image to be searched that meets the preset search conditions is selected from each image to be searched as the search result corresponding to the search text. The server 130 can then send the obtained search results to the physical terminal device 110. After receiving the search results, the physical terminal device 110 may display them on the display interface 120 so that an object that needs to perform a cross-modal search can obtain the search results.

[0064] Among them, the physical terminal device 110 of the embodiment of the present application can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this.

[0065] The server 130 in the embodiment of the present application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. This application does not impose any restrictions on this.

[0066] based on Figure 1 The application scenario shown in Figure 2 This is an exemplary flow chart of a cross-modal retrieval method provided in an embodiment of the present application. This process can be applied to Figure 1 The server shown may include the following steps 201-203:

[0067] Step 201: Acquire the search text and extract the target text features of the search text.

[0068] The search text can be a text description of the search requirement, such as a word, phrase, or sentence. For example, if a searcher wants to retrieve images or videos that include umbrellas, the search text can be the word: "umbrella." Another example is if a searcher wants to retrieve images or videos of cars on the road, the search text can be the phrase: "cars on the road." Another example is if a searcher wants to retrieve images or videos of a puppy sitting on the side of the road, the search text can be the sentence: "a puppy sitting on the side of the road."

[0069] The target text features may be extracted by a text encoder. It should be noted that the text encoder may be implemented using any model that can extract text features, such as the bidirectional semantic encoding representation model (Bidirectional Encoder Representations from Transformers, BERT) built based on transformers, and this application does not limit this.

[0070] Step 202: For each image to be retrieved, perform the following steps 2021-2022 respectively to obtain the image matching results corresponding to each image to be retrieved:

[0071] Step 2021: perform similarity matching on the global features and multiple local features of an image to be retrieved and the target text features respectively to obtain corresponding feature matching results.

[0072] Local features represent the visual features of a specified area of ​​an image to be retrieved. An image to be retrieved can be divided into multiple areas according to a preset area division rule, and the visual feature distribution of each area is used as a local feature of the image to be retrieved.

[0073] For example, see Figure 3 Schematic diagram of local features provided for an embodiment of the present application. Assume that the image A to be retrieved is divided into 2*2 regions according to a preset region division rule, that is, the image is divided into four regions A1, A2, A3, and A4 as shown in the figure. Based on the position of region A1 in the image to be retrieved, the visual feature f1 of region A1 can be extracted; based on the position of region A2 in the image to be retrieved, the visual feature f2 of region A2 can be extracted; based on the position of region A3 in the image to be retrieved, the visual feature f3 of region A3 can be extracted; based on the position of region A4 in the image to be retrieved, the visual feature f4 of region A4 can be extracted. That is, when the image to be retrieved is divided into 2*2 regions, four local features of the image to be retrieved can be obtained: f1, f2, f3, and f4.

[0074] In a possible implementation, the image to be retrieved may be any image included in the database to be retrieved. When the database to be retrieved includes a video to be retrieved, the image to be retrieved may also include at least one frame of image in the video to be retrieved.

[0075] In some embodiments, the global features and multiple local features of each image to be retrieved may be obtained by pre-extracting features from each image to be retrieved using a visual encoder. The dimensions of the global features and local features extracted by the visual encoder are consistent with the dimensions of the target text features extracted by the text encoder, for example, both may be 256 or 1024.

[0076] Optional, see Figure 4 This is a diagram of the visual encoder network structure provided in the embodiment of the present application. The visual encoder may include a convolutional layer, an interest region pooling layer, a fully connected layer, and an embedding layer. Figure 4 As shown, the visual encoder can perform the following process for each image to be retrieved to extract the global features and multiple local features of each image to be retrieved:

[0077] After an image to be retrieved is input into the encoder, the global feature extraction of the image to be retrieved can be performed through multiple convolutional layers to obtain the global features of the image to be retrieved. At the same time, the region of interest pooling layer of the visual encoder can be used to extract the global features of the image to be retrieved based on the preset region division rules ( Figure 4 (For example, the image is divided into 2x2 regions.) Regions of interest (ROIs) are pooled for each local region of the image to be retrieved, obtaining multiple local features of the image. These local features and global features are then mapped to the same scale. Finally, multiple fully connected layers and embedding layers are used for further processing, resulting in the visual encoder outputting global and local features of the image to be retrieved.

[0078] In one example, the preset region division rule can be to divide the image to be retrieved into 3*3. First, the image to be retrieved is input into the visual encoder and the global features of the image to be retrieved are obtained through multiple convolutional layers. Then, according to the coordinates of each region in the region to be retrieved, 9 local features are obtained through region of interest pooling. These 9 local features and the global features are mapped to the same scale (for example, 7*7). Finally, the global vector feature f and the vector features of 9 local regions (f1, f2, f3, f4, f5, f6, f7, f8, f9) are obtained through multiple fully connected layers and embedding layers.

[0079] This application solves the problem in related technologies that for images to be retrieved in complex scenes, the target to be retrieved may only occupy a small area in the image to be retrieved, resulting in failure to retrieve the target, by extracting finer-grained features when extracting visual features, thereby improving the cross-modal retrieval capability for complex scenes.

[0080] It should be noted that Figure 4 The network structure diagram of the visual encoder shown is only exemplary. The network structure of the visual encoder can be any structure that can extract the global features and local features of the image, and this application does not limit this.

[0081] In some embodiments, the global features and multiple local features of the image to be retrieved obtained according to the above method are respectively matched with the target text features for similarity. When obtaining the corresponding feature matching results, each feature can be first L2 normalized (L2 normalization) to represent each feature as a vector, and then the cosine similarity between the global features and the multiple local features and the target text features is calculated to satisfy formula (1). The result obtained by formula (1) is the similarity matching result between the corresponding two features. The value range of cosine similarity is between -1 and 1. The closer the value is to 1, the more similar the two features are, and the closer the value is to -1, the less similar the two features are.

[0082] S=cos(θ)=(A·B) / (||A||*||B||) Formula (1)

[0083] Where S represents the similarity between A and B, A and B represent the vector representations of the target text features and visual features, respectively. Visual features include global features and local features. · represents the dot product operation of the vectors. ||A|| and ||B|| represent the norms of feature vector A and feature vector B, respectively.

[0084] For example, suppose the image to be retrieved includes a global feature f and four local features f1, f2, f3, f4, and the target text feature is f t , then f and f can be calculated respectively by formula (1) t The similarity between S1, f1 and f t The similarity S2 between f2 and f t The similarity between S3, f3 and f t The similarity between S4, f4 and f t The similarity S5 between them, the five similarity results S1-S5 obtained are the feature matching results.

[0085] In other embodiments, the global features and multiple local features of the image to be retrieved obtained according to the above method are respectively matched with the target text features for similarity. When the corresponding feature matching results are obtained, each feature is L2 normalized and each feature is represented as a vector. The distance (for example, Euclidean distance) between the global features and the multiple local features and the target text features can also be calculated to determine the similarity between the two features, satisfying formula (2). The value range of the Euclidean distance is a non-negative real number. The smaller the value, the more similar the two features are, and the larger the value, the less similar the two features are.

[0086] S=||AB|| Formula (2)

[0087] Where ||AB|| represents the norm of the difference between eigenvector A and eigenvector B, also known as the Euclidean distance.

[0088] It should be noted that the similarity matching method can be selected based on actual conditions or experience, and this application does not limit this.

[0089] In one possible implementation, in order to further improve the retrieval coverage, for images that include text in each image to be retrieved, an optical character recognition (OCR) algorithm can be used to recognize the text in the image in advance, and a text encoder can be used to extract features. The obtained image text features are used as features of the image and are matched with the target text features for similarity. The obtained similarity is used as a feature matching result of the corresponding image to be retrieved to execute step 2022.

[0090] Step 2022: Select a feature matching result that meets a preset similarity matching condition from among multiple feature matching results corresponding to an image to be retrieved as an image matching result corresponding to an image to be retrieved.

[0091] Optionally, the preset similarity matching condition may be to use the maximum value among the multiple feature matching results as the image matching result corresponding to the search image. Alternatively, the preset similarity matching condition may be to use the feature matching result greater than a preset similarity threshold among the multiple feature matching results as the image matching result corresponding to the search image.

[0092] For example, assuming that each image to be retrieved includes one global feature and four local features, the feature matching results obtained for the image to be retrieved A are 0.5, 0.2, 0.6, 0.4, and 0.4, respectively. Since the maximum value among them is 0.6, when the preset similarity matching condition is to take the maximum value of multiple feature matching results as the image matching result corresponding to the retrieval image, the image matching result corresponding to the image to be retrieved A is 0.6.

[0093] Step 203: Based on the obtained image matching results, at least one image to be retrieved that meets the preset retrieval conditions is selected from the images to be retrieved as the retrieval result corresponding to the retrieval text.

[0094] In some embodiments, the preset search condition can be that any image to be retrieved whose image matching result is greater than a preset similarity threshold can be used as a search result. For example, assuming the preset similarity threshold is 0.8, all images to be retrieved whose image matching result is greater than 0.8 can be used as the search result corresponding to the search text. In other words, assuming that there are 10 images to be retrieved whose image matching results are all greater than 0.8, then the search results can include these 10 images to be retrieved. Assuming that there is one image to be retrieved whose image matching result is greater than 0.8, then the search results can include this image to be retrieved.

[0095] In other embodiments, the preset search condition may further include that only the image to be searched with the maximum image matching result is used as the search result. For example, if only one of the images to be searched has an image matching result of 0.95, and the image matching results of the other images to be searched are all less than 0.95, then the search result is that image to be searched.

[0096] Furthermore, when there are multiple to-be-retrieved images with the highest image matching results, the image with the highest image matching results among these multiple to-be-retrieved images can be used as the retrieval result. For example, if the to-be-retrieved images include three to-be-retrieved images with an image matching result of 0.95, and the image matching results corresponding to the other to-be-retrieved images are all less than 0.95, then the retrieval result can be these three to-be-retrieved images.

[0097] Alternatively, when there are multiple images to be retrieved with the highest image matching result, one of the multiple images to be retrieved with the highest image matching result can be selected as the retrieval result. For example, if the images to be retrieved include three images A, B, and C with an image matching result of 0.95, and the image matching results corresponding to the other images to be retrieved are all less than 0.95, then the retrieval result can be A, B, or C.

[0098] In some other embodiments, the preset search conditions may further include setting the number of retrieved images included in the search results. For example, the image matching results may be sorted from largest to smallest, and the first ten images to be retrieved may be selected as the search results.

[0099] It should be noted that the preset search conditions can be preset based on actual conditions or experience, and this application does not limit this.

[0100] In a possible implementation, when the database to be retrieved includes the video to be retrieved and at least one image to be retrieved that meets the preset retrieval conditions includes an image belonging to the video to be retrieved, it can also be determined that the retrieval result includes the corresponding video to be retrieved.

[0101] Optionally, when at least one image to be retrieved that meets the preset retrieval conditions includes an image belonging to a video to be retrieved, the image to be retrieved and the video to be retrieved can be displayed in the retrieval results, or the corresponding image to be retrieved can be replaced by the video to be retrieved.

[0102] In one possible implementation, the present invention provides a cross-modal retrieval model, which includes a text encoder, a visual encoder, and a retrieval framework. Figure 2 In the process shown, the text encoder is used to extract the target text features of the retrieved text, the visual encoder is used to extract a global feature and multiple local features of the image to be retrieved, and the retrieval framework is used to perform similarity matching between the global feature and multiple local features of the image to be retrieved and the target text features, and obtain the corresponding similarity matching results.

[0103] In one possible implementation, see Figure 5 This is an exemplary flow chart of a method for training a visual encoder and a text encoder provided in an embodiment of the present application. The visual encoder and the text encoder in the above cross-modal retrieval model are both trained using this method, and the process may include steps 501-503:

[0104] Step 501: Obtain a training sample set.

[0105] The training sample set includes multiple image samples and their respective text annotation sets. The text annotation set includes a global text annotation and multiple local text annotations for the corresponding image sample. Local text annotation refers to the text annotation of a specified area of ​​the image sample. The number of local text annotations is related to the preset area division rule. For example, if the preset area division rule is to divide the image sample into 2*2 areas, each image sample includes 5 local text annotations, and each divided area corresponds to a local text annotation.

[0106] For details, see Figure 6 The following is a flow chart of the image sample determination method provided in the embodiment of the present application. When obtaining the training sample set, the server can first obtain multiple candidate samples and perform the following operations on each candidate sample: Figure 6 The process shown in the figure determines some samples among the candidate samples as image samples:

[0107] Step 601: Perform image recognition on a candidate sample using a large visual model to obtain a global text annotation and a local text annotation of the candidate sample.

[0108] Because the visual model possesses excellent visual perception capabilities, it can be used as an annotator to generate global and local text annotations for each candidate sample. Furthermore, by adding local text annotations, the problem of directly extracting whole-image features as retrieval vectors, which can lead to poor visual identification of the corresponding elements when the target element occupies a small proportion of the target image and the target image itself is complex, as is the case with related techniques.

[0109] Step 602: extract features from a candidate sample to obtain global visual features and local visual features of the candidate sample, and extract features from the global text annotation and local text annotation of the candidate sample to obtain global text features and local text features of the candidate sample.

[0110] Since the text annotations generated by the visual large model in step 601 may have certain errors, in order to avoid affecting the training of the visual encoder and the text encoder, the candidate samples can be screened and cleaned by calculating the similarity between the visual features and the text features and comparing them with the preset similarity threshold.

[0111] Specifically, a larger image-text pair model, such as the OpenClip G / 14 model, can be used to extract features from a candidate sample and the global text annotation and local text annotation of a candidate sample, respectively, to obtain the global visual features and local visual features of the candidate sample, as well as the global text features and local text features.

[0112] It should be noted that the preset similarity thresholds mentioned in different steps of this application may be the same or different, and this application does not limit this.

[0113] Step 603: Calculate the similarity between the global visual features and the global text features, and the similarity between the corresponding local visual features and the local text features.

[0114] If the calculated similarities are greater than the preset similarity threshold, step 604 is executed; otherwise, step 605 is executed.

[0115] The OpenClip G / 14 model is used to calculate the similarity between the corresponding visual features and text features. If the similarity is low, it can be considered that the association between the image and text is weak, and the corresponding candidate sample may be mislabeled, and the candidate sample is determined not to be an image sample. If the similarity is high, it can be considered that the association between the image and text is strong, and the corresponding candidate sample is determined to be an image sample.

[0116] For example, suppose that only the similarity between a certain local visual feature and a local text feature of a candidate sample is lower than the preset similarity threshold, while the similarities between the other visual features and the corresponding text features are higher than the preset similarity threshold. However, since the candidate sample does not meet the condition that the calculated similarities are all higher than the preset similarity threshold, the candidate sample is determined not to be an image sample.

[0117] Step 604: Determine that the candidate sample is an image sample.

[0118] Step 605: Determine whether the candidate sample is an image sample.

[0119] For example, if the preset region division rule is to divide the image sample into 2*2 regions,

[0120] In a possible implementation, the training sample set further includes an image-text matching label, which represents the matching relationship between the image sample and each global text annotation and local text annotation included in the text annotation set.

[0121] Specifically, when configuring image-text matching tags, you can configure them according to different types of text annotations:

[0122] (1) For local text annotation, configure the image-text matching tag as follows:

[0123] When a local text annotation belongs to a local area of ​​a certain image sample, the local area and the local text annotation are marked as matched, and the image-text matching label between the local area and the local text annotation is set to 1. The image sample and the local text annotation are marked as matched, and the image-text matching label between the image sample and the local text annotation is set to 1. At the same time, the image-text matching labels between the local text annotation and other local areas of the image sample, other image samples included in the training sample set, and each local area of ​​other image samples need to be set to 0.

[0124] For example, suppose image sample B is divided into 2x2 regions: B1, B2, B3, and B4. "A puppy" is the local text annotation corresponding to B4. For image sample B, the image-text matching labels between "A puppy" and B1, and between "A puppy" and B, are set to 1. The image-text matching labels between "A puppy" and B2, B3, and B4 are all set to 0. Furthermore, the image-text matching labels between "A puppy" and all other image samples in the training set are all set to 0.

[0125] (2) For global text annotation, configure the image-text matching tag as follows:

[0126] When a global text annotation belongs to a certain image sample, the image sample and the global text annotation can be recorded as a match, and the image-text matching label between the image sample and the global text annotation is set to 1. However, the local areas of the image may or may not be related to the global text annotation. In order to avoid introducing erroneous supervision information, the image-text matching labels between the global text annotation and the local areas of the image sample can be set to special identifiers, such as 255. At the same time, when the loss function is subsequently calculated, the image-text matching label does not participate in the calculation. In addition, the image-text matching labels between the global text annotation and other image samples included in the training sample set, as well as the local areas of other image samples, are also required to be set to 0.

[0127] For example, suppose image sample B is divided into 2x2 regions: B1, B2, B3, and B4. "A puppy sitting on the roadside" is the global text annotation corresponding to B. For image sample B, the image-text matching label between "A puppy sitting on the roadside" and B is set to 1, and the image-text matching labels between "A puppy sitting on the roadside" and B1, B2, B3, and B4 are all set to 255. Furthermore, the image-text matching labels between "A puppy sitting on the roadside" and other image samples in the training set are all set to 0.

[0128] Step 502: Compress the pre-trained large visual model through feature distillation to obtain the initial parameters of the visual encoder.

[0129] Among them, the visual large model is a model with fixed parameters and strong visual perception capabilities. The visual large model has the same structure as the visual encoder.

[0130] Specifically, when the server compresses the pre-trained large visual model through feature distillation to obtain the initial parameters of the visual encoder, it can perform the following operations on the multiple image samples: first, the reference features of an image sample can be extracted using the large visual model. Then, the image sample can be randomly transformed to obtain multiple augmented samples corresponding to the image sample. The visual encoder can then perform feature extraction on the multiple augmented samples to obtain multiple learning features. Finally, based on the learning loss between each learning feature and the reference feature, the parameters of the visual encoder can be adjusted until the second training end condition is met, and the initial parameters of the visual encoder are determined.

[0131] For details, see Figures 7A-7E Schematic diagram of random transformation methods provided in an embodiment of the present application. When performing random transformation on an image sample to obtain multiple augmented samples corresponding to the image sample, the random transformation methods may include but are not limited to the following methods:

[0132] (1) Random image masking: Divide the image sample into several blocks and randomly mask some of them. The masking ratio usually does not exceed 50%. Figure 7A As shown in FIG, assuming that the image sample is divided into 2*2 blocks, one or two blocks can be randomly blocked, and the blocked image is used as the augmented sample.

[0133] (2) Rotation or Mirroring: Rotate or mirror the image sample by setting the random probability of rotation and mirroring. For example, the random probability of rotation and mirroring can be set to 0.5. In addition, for rotation, the rotation angle can be pre-configured, for example, it can be set to randomly select any angle between -10° and 10° as the rotation angle. Figure 7B As shown in the figure, when rotating, 5° can be randomly selected as the rotation angle. Then, the center point of the image sample is used as the rotation center, and the corresponding augmented sample is obtained by rotating 5°. When mirroring, the image sample is horizontally flipped to obtain the corresponding augmented sample.

[0134] (3) Add border: Figure 7C As shown in , a border is randomly added to the periphery of the image sample with a certain probability, and the color of the border is random. The width of the border can be pre-configured according to the size of the image sample. Figure 7C As shown in FIG, assuming that the size of the image sample is 100*100 pixels, the pre-configured border width is 10 pixels, and the color of the randomly added border is black, after adding the border, an augmented sample of size 120*120 pixels can be obtained.

[0135] (4) Add smear or mosaic: Randomly add smear or mosaic on the image sample with a certain probability, and limit the occlusion area of ​​the image sample to no more than 20%. Figure 7D As shown, when painting, you can paint the randomly selected area with a random color, such as black. When adding mosaics, you can add mosaics by using a mean filter on the randomly selected area.

[0136] (5) Cropping: Randomly crop the image sample with a certain probability. The remaining area after cropping is usually larger than 80% of the original image sample. Figure 7E As shown, the starting point coordinates of the cropping area, the width of the cropping area, and the height of the cropping area can be randomly selected to obtain the cropped augmented samples.

[0137] It should be noted that the random transformation may employ one or more of the aforementioned methods, or may employ a random transformation method other than the aforementioned methods to process the image samples to obtain corresponding multiple augmented samples. For example, random affine or perspective transformations may be performed on the image samples, which is not limited in this application.

[0138] See also Figure 8This is a schematic diagram of the characteristic distillation process provided in the embodiment of this application. Figure 8 As shown in the figure, the large visual model is used as the teacher network (teacher), the visual encoder is used as the student network (student), and feature distillation is used to obtain a visual encoder with better visual perception ability.

[0139] A reference feature v_t is extracted from image sample 1 using the large visual model. A visual encoder is then used to extract multiple learning features v_s = {v_s1, v_s2, v_s3, ...} from each of the augmented samples of image sample 1. Based on the learning loss between each learning feature and the reference feature, the parameters of the visual encoder are adjusted until the second training end condition is met, at which point the initial parameters of the visual encoder are determined.

[0140] Among them, the learning loss can use absolute value loss (L1 Loss) as the loss function, that is, the learning loss can be L1(v_t, v_s), satisfying formula (2).

[0141] L1(v_t, v_s)=1 / n*Σ|v_t-v_si| Formula (2)

[0142] Where n represents the number of augmented samples, v_si represents each learning feature, and i is an integer between 1 and n.

[0143] Optionally, the second training end condition can be set according to actual conditions. For example, it can be a fixed number of iterations, or it can be to end the training when the indicator on the pre-configured validation set reaches a preset threshold. This application does not limit this.

[0144] Through this approach, the student network (visual encoder) can effectively distill and learn better visual perception capabilities from the teacher network (visual model). At the same time, due to the random transformation of image samples, the visual encoder can also be made more robust to relevant image transformations and interference.

[0145] Step 503: By performing similarity matching between the visual features of each of the multiple image samples extracted by the visual encoder and the text features of each global text annotation and each local text annotation extracted by the text encoder, the parameters of the visual encoder and the text encoder are adjusted until the first training end condition is met, thereby obtaining the trained visual encoder and text encoder.

[0146] In one possible implementation, step 503 may be specifically performed by extracting visual features from each of the multiple image samples using a visual encoder, and extracting text features from each global text annotation and each local text annotation using a text encoder. For each extracted visual feature, similarity is calculated between it and each text feature, and based on the similarity loss between the similarity and the corresponding image-text matching label, the parameters of the visual encoder and text encoder are adjusted until the first training end condition is met, thereby obtaining the trained visual encoder and text encoder.

[0147] See also Figure 9 Schematic diagram of the image-text alignment training process of the visual encoder and text encoder provided in the embodiment of this application. Take an image sample as an example to illustrate. The image sample is based on Figure 9 The area division method shown by the dotted line is divided into 2*2 areas. The image sample is input into the visual encoder for feature extraction to obtain the global visual features and four local visual features of the image sample. The global text annotation and each local text annotation corresponding to the image sample are input into the text encoder for feature extraction to obtain global text features and local text features. The similarity is calculated for each visual feature and each text feature respectively. The visual encoder and the text encoder can then be trained for image-text alignment based on the similarity loss between the similarity and the corresponding image-text matching label. Among them, the similarity loss can be calculated using the cross entropy loss function. This application does not limit the loss function used.

[0148] Optionally, the first training end condition may be the same as or different from the second training end condition. The first training end condition may also be set based on actual conditions or experience. For example, it may be a fixed number of iterations, or it may be set to end training when an indicator on a pre-configured validation set reaches a preset threshold. This application does not limit this.

[0149] In one possible implementation, the visual encoder extracts the visual features of multiple image samples by the following method: for multiple image samples, perform the following operations respectively: input an image sample into the visual encoder, perform feature extraction on the image sample through multiple convolutional layers in the visual encoder, and obtain the global visual features of the image sample. Based on the preset region division rules, perform interest region pooling on each local region of an image sample to obtain multiple local visual features of an image sample. Through the embedding layer in the visual encoder, the global visual features of an image sample and multiple local visual features are fused to obtain the visual features of an image sample. The specific method of extracting visual features by the visual encoder can be found in Figure 2 The relevant descriptions in the method embodiment shown are not repeated here.

[0150] Based on the same inventive concept as the above method embodiment, the present application embodiment also provides a cross-modal retrieval device. Figure 10 The cross-modal retrieval apparatus 1000 may include:

[0151] An acquisition unit 1001 is used to acquire a search text;

[0152] A feature extraction unit 1002 is used to extract target text features of the search text;

[0153] The image matching unit 1003 is configured to perform the following operations for each image to be retrieved, respectively, to obtain an image matching result corresponding to each image to be retrieved: performing similarity matching on the global features and multiple local features of the image to be retrieved and the target text features, respectively, to obtain corresponding feature matching results; the local features represent visual features of a specified area of ​​the image to be retrieved; and selecting, from the multiple feature matching results corresponding to the image to be retrieved, a feature matching result that meets a preset similarity matching condition as the image matching result corresponding to the image to be retrieved;

[0154] The retrieval unit 1004 is configured to select, based on the obtained image matching results, at least one image to be retrieved that meets a preset retrieval condition from the images to be retrieved as a retrieval result corresponding to the retrieval text.

[0155] In one possible implementation, the feature extraction unit 1002 includes a text encoder and a visual encoder, and the target text features are extracted by the text encoder; the global features and the multiple local features of each image to be retrieved are obtained by pre-extracting features of each image to be retrieved by the visual encoder;

[0156] The visual encoder and the text encoder are trained by the training unit 1005 through the following method:

[0157] Acquire a training sample set; the training sample set includes a plurality of image samples and a text annotation set including a global text annotation and a local text annotation for each of the plurality of image samples;

[0158] Compressing a pre-trained large visual model through feature distillation to obtain initial parameters of a visual encoder; the large visual model has the same structure as the visual encoder;

[0159] By performing similarity matching on the visual features of each of the multiple image samples extracted by the visual encoder and the text features of each of the global text annotations and each of the local text annotations extracted by the text encoder, the parameters of each of the visual encoder and the text encoder are adjusted until the first training end condition is met, thereby obtaining the trained visual encoder and text encoder.

[0160] In a possible implementation, the training sample set further includes an image-text matching label, wherein the image-text matching label represents a matching relationship between the image sample and each global text annotation and local text annotation included in the text annotation set;

[0161] The training unit 1005 performs similarity matching between the visual features of each of the multiple image samples extracted by the visual encoder and the text features of each of the global text annotations and each of the local text annotations extracted by the text encoder, and adjusts the parameters of each of the visual encoder and the text encoder until the first training end condition is met, and the trained visual encoder and text encoder are obtained. The training unit 1005 is used to: use the visual encoder to extract the visual features of each of the multiple image samples, and use the text encoder to extract the text features of each of the global text annotations and each of the local text annotations; for each of the extracted visual features, calculate the similarity between it and each of the text features, and adjust the parameters of each of the visual encoder and the text encoder according to the similarity loss between the similarity and the corresponding image-text matching label, until the first training end condition is met, and the trained visual encoder and text encoder are obtained.

[0162] In one possible implementation, when the training unit 1005 compresses the pre-trained visual large model through feature distillation to obtain the initial parameters of the visual encoder, it is specifically used to: for the multiple image samples, perform the following operations respectively: extract the reference features of an image sample through the visual large model; perform random transformation on the one image sample to obtain multiple augmented samples corresponding to the one image sample; perform feature extraction on the multiple augmented samples through the visual encoder to obtain multiple learning features; adjust the parameters of the visual encoder based on the learning loss between each learning feature and the reference feature until the second training end condition is met, and then determine the initial parameters of the visual encoder.

[0163] In one possible implementation, the visual encoder extracts the visual features of each of the multiple image samples by the following method: for the multiple image samples, the following operations are respectively performed: an image sample is input into the visual encoder, and features of the image sample are extracted through multiple convolutional layers in the visual encoder to obtain global visual features of the image sample; based on preset region division rules, interest region pooling is performed on each local region of the image sample to obtain multiple local visual features of the image sample; and the global visual features of the image sample and multiple local visual features are feature-fused through the embedding layer in the visual encoder to obtain the visual features of the image sample.

[0164] In one possible implementation, when the training unit 1005 obtains a training sample set, it is specifically used to: obtain multiple candidate samples, and perform the following operations for each candidate sample: perform image recognition on a candidate sample through the visual large model to obtain the global text annotation and local text annotation of the candidate sample; perform feature extraction on the candidate sample to obtain the global visual features and local visual features of the candidate sample, and perform feature extraction on the global text annotation and local text annotation of the candidate sample to obtain the global text features and local text features of the candidate sample; calculate the similarity between the global visual features and the global text features, and the similarity between the corresponding local visual features and the local text features; when the calculated similarities are all greater than the preset similarity threshold, determine that the candidate sample is an image sample; otherwise, determine that the candidate sample is not an image sample.

[0165] In one possible implementation, when the video to be retrieved is included in the database to be retrieved, the image to be retrieved includes at least one frame of image in the video to be retrieved; then the retrieval unit 1004 is further used to determine that the retrieval result includes the video to be retrieved when at least one image to be retrieved that meets the preset retrieval conditions includes an image belonging to the video to be retrieved.

[0166] For the convenience of description, the above parts are divided into each module (or unit) according to function and described separately. In the embodiment of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuit or memory) or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. In addition, each module or unit can be a part of the overall module or unit that includes the module or unit function.

[0167] After introducing the cross-modal retrieval method and apparatus according to an exemplary embodiment of the present application, a computer device according to another exemplary embodiment of the present application will be introduced.

[0168] Those skilled in the art will appreciate that various aspects of the present application can be implemented as systems, methods, or program products. Therefore, various aspects of the present application can be specifically implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation that combines hardware and software aspects, which may be collectively referred to herein as a "circuit," "module," or "system."

[0169] Based on the same inventive concept as the above method embodiment, the present application embodiment also provides a computer device. In one embodiment, the computer device can be a server, such as Figure 1 In this embodiment, the structure of the computer device is as follows Figure 11 As shown, it may include at least a memory 1101 , a communication module 1103 , and at least one processor 1102 .

[0170] Memory 1101 is used to store computer programs executed by processor 1102. Memory 1101 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and programs required for running instant messaging functions, while the data storage area may store various instant messaging messages and operating instruction sets.

[0171] Memory 1101 may be a volatile memory, such as random-access memory (RAM); a non-volatile memory, such as read-only memory, flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); or any other medium capable of carrying or storing a desired computer program in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 1101 may be a combination of the aforementioned memories.

[0172] The processor 1102 may include one or more central processing units (CPUs) or digital processing units, etc. The processor 1102 is configured to implement the above cross-modal retrieval method when calling the computer program stored in the memory 1101 .

[0173] The communication module 1103 is used to communicate with terminal devices and other servers.

[0174] The specific connection medium between the memory 1101, the communication module 1103 and the processor 1102 is not limited in the embodiment of the present application. Figure 11 In the embodiment, the memory 1101 and the processor 1102 are connected via a bus 1104. Figure 11 The connections between the other components are shown in bold lines for illustration only and are not intended to be limiting. The bus 1104 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Figure 11 The diagram shows a single thick line, but this does not indicate that there is only one bus or one type of bus.

[0175] The memory 1101 stores a computer storage medium, which stores computer executable instructions. The computer executable instructions are used to implement the cross-modal retrieval method of the embodiment of the present application. The processor 1102 is used to execute the above cross-modal retrieval method, such as Figure 2 shown.

[0176] In another embodiment, the computer device may also be other computer devices, such as Figure 1 The physical terminal device 110 shown in FIG. In this embodiment, the structure of the computer device can be as follows Figure 12 As shown, it includes: a communication component 1210, a memory 1220, a display unit 1230, a camera 1240, a sensor 1250, an audio circuit 1260, a Bluetooth module 1270, a processor 1280 and other components.

[0177] The communication component 1210 is used to communicate with the server. In some embodiments, it may include a wireless fidelity (WiFi) module. The WiFi module is a short-range wireless transmission technology. The electronic device can help the object send and receive information through the WiFi module.

[0178] The memory 1220 can be used to store software programs and data. The processor 1280 executes various functions and data processing of the physical terminal device 110 by running the software programs or data stored in the memory 1220. The memory 1220 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. The memory 1220 stores an operating system that enables the physical terminal device 110 to run. In the present application, the memory 1220 can store an operating system and various application programs, and may also store a computer program that executes the cross-modal retrieval method of the embodiment of the present application.

[0179] The display unit 1230 can also be used to display information input by or provided to an object, as well as a graphical user interface (GUI) for displaying various menus of the physical terminal device 110. Specifically, the display unit 1230 may include a display screen 1232 disposed on the front of the physical terminal device 110. The display screen 1232 may be configured in the form of a liquid crystal display, a light-emitting diode, or the like. The display unit 1230 can be used to display the search text input interface, the search result output interface, and the like in the embodiments of the present application.

[0180] The display unit 1230 can also be used to receive input digital or character information and generate signal input related to the object setting and function control of the physical terminal device 110. Specifically, the display unit 1230 may include a touch screen 1231 set on the front of the physical terminal device 110, which can collect touch operations of objects on or near it, such as clicking a button, dragging a scroll box, etc.

[0181] The touch screen 1231 can be covered on the display screen 1232, or the touch screen 1231 and the display screen 1232 can be integrated to realize the input and output functions of the physical terminal device 110. The integrated display screen can be simply called a touch screen. In this application, the display unit 1230 can display applications and corresponding operation steps.

[0182] The camera 1240 can be used to capture static images, and the subject can publish the images captured by the camera 1240 through an application. The camera 1240 can be one or more. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then transmitted to the processor 1280 for conversion into a digital image signal.

[0183] The physical terminal device may further include at least one sensor 1250, such as an accelerometer 1251, a distance sensor 1252, a fingerprint sensor 1253, and a temperature sensor 1254. The terminal device may also be equipped with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, a light sensor, and a motion sensor.

[0184] The audio circuit 1260, the speaker 1261, and the microphone 1262 can provide an audio interface between the object and the physical terminal device 110. The audio circuit 1260 can transmit the electrical signal converted from the received audio data to the speaker 1261, which converts it into a sound signal for output. The physical terminal device 110 can also be equipped with a volume button for adjusting the volume of the sound signal. On the other hand, the microphone 1262 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1260 and converted into audio data. The audio data is then output to the communication component 1110 for transmission to, for example, another physical terminal device 110, or the audio data is output to the memory 1220 for further processing.

[0185] The Bluetooth module 1270 is used to exchange information with other Bluetooth devices having a Bluetooth module through the Bluetooth protocol. For example, a physical terminal device can establish a Bluetooth connection with a wearable electronic device (such as a smart watch) that also has a Bluetooth module through the Bluetooth module 1270 to exchange data.

[0186] The processor 1280 is the control center of the physical terminal device. It uses various interfaces and lines to connect various parts of the entire terminal. It executes various functions of the terminal device and processes data by running or executing software programs stored in the memory 1220 and calling data stored in the memory 1220. In some embodiments, the processor 1280 may include one or more processing units; the processor 1280 may also integrate an application processor and a baseband processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the baseband processor mainly processes wireless communications. It is understandable that the above-mentioned baseband processor may not be integrated into the processor 1280. In the present application, the processor 1280 can run the operating system, application programs, user interface display and touch response, as well as the cross-modal retrieval method of the embodiment of the present application. In addition, the processor 1280 is coupled to the display unit 1230.

[0187] In addition, it should be noted that in the specific implementation of this application, object data related to cross-modal retrieval, etc. is involved. When the above embodiments of this application are applied to specific products or technologies, it is necessary to obtain the object's permission or consent, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0188] In some possible implementations, various aspects of the cross-modal retrieval method provided in the present application may also be implemented in the form of a program product, which includes a computer program. When the program product is run on a computer device, the computer program is used to cause the computer device to execute the steps of the cross-modal retrieval method according to various exemplary embodiments of the present application described above in this specification. For example, the computer device may execute the following steps: Figure 2Follow the steps shown in .

[0189] The program product can use any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0190] The program product of the embodiment of the present application may be a portable compact disc read-only memory (CD-ROM) and include a computer program, and can be run on an electronic device. However, the program product of the present application is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with a command execution system, apparatus, or device.

[0191] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a readable computer program. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with a command execution system, apparatus, or device.

[0192] The computer program embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0193] The computer program for performing the operations of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The computer program can be executed entirely on the user's computer device, partially on the user's computer device, as a stand-alone software package, partially on the user's computer device and partially on a remote computer device, or entirely on the remote computer device. In the case of a remote computer device, the remote computer device can be connected to the user's computer device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer device (e.g., through the Internet using an Internet service provider).

[0194] It should be noted that although several units or subunits of the device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, depending on the embodiment of the application, the features and functions of two or more units described above can be embodied in a single unit. Conversely, the features and functions of a single unit described above can be further divided and embodied by multiple units.

[0195] Furthermore, although the operations of the method of the present application are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0196] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain a computer-usable computer program.

[0197] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program commands. These computer program commands can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the commands executed by the processor of the computer or other programmable data processing device generate commands for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0198] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising a command device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0199] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0200] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0201] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A cross-modal retrieval method, characterized in that: include: Obtaining a search text and extracting target text features of the search text; For each image to be retrieved, perform the following operations to obtain the image matching results corresponding to each image to be retrieved: Performing similarity matching on the global features and multiple local features of an image to be retrieved and the target text features respectively to obtain corresponding feature matching results; the local features represent: visual features of a specified area of ​​the image to be retrieved; Selecting, from among the plurality of feature matching results corresponding to the image to be retrieved, a feature matching result that meets a preset similarity matching condition as the image matching result corresponding to the image to be retrieved; Based on the obtained image matching results, at least one image to be retrieved that meets the preset retrieval conditions is selected from the images to be retrieved as the retrieval result corresponding to the retrieval text.

2. The method according to claim 1, characterized in that The target text features are extracted by a text encoder; the global features and multiple local features of each image to be retrieved are obtained by pre-extracting features of each image to be retrieved by a visual encoder; The visual encoder and the text encoder are trained by the following method: Acquire a training sample set; the training sample set includes a plurality of image samples and a text annotation set including a global text annotation and a local text annotation for each of the plurality of image samples; Compressing a pre-trained large visual model through feature distillation to obtain initial parameters of a visual encoder; the large visual model has the same structure as the visual encoder; By performing similarity matching on the visual features of each of the multiple image samples extracted by the visual encoder and the text features of each of the global text annotations and each of the local text annotations extracted by the text encoder, the parameters of each of the visual encoder and the text encoder are adjusted until the first training end condition is met, thereby obtaining the trained visual encoder and text encoder.

3. The method according to claim 2, characterized in that The training sample set further includes an image-text matching label, wherein the image-text matching label represents a matching relationship between the image sample and each global text annotation and local text annotation included in the text annotation set; The method further comprises: performing similarity matching between the visual features of each of the plurality of image samples extracted by the visual encoder and the text features of each of the global text annotations and each of the local text annotations extracted by the text encoder, and adjusting the parameters of each of the visual encoder and the text encoder until a first training end condition is met, thereby obtaining a trained visual encoder and a text encoder, including: Extracting visual features of each of the plurality of image samples using the visual encoder, and extracting text features of each of the global text annotations and each of the local text annotations using the text encoder; For each of the extracted visual features, the similarity between it and each of the text features is calculated respectively, and according to the similarity loss between the similarity and the corresponding image-text matching label, the respective parameters of the visual encoder and the text encoder are adjusted until the first training end condition is met, thereby obtaining the trained visual encoder and text encoder.

4. The method according to claim 2, characterized in that The pre-trained large visual model is compressed by feature distillation to obtain the initial parameters of the visual encoder, including: For the multiple image samples, perform the following operations respectively: Extract reference features of an image sample through a large visual model; Performing random transformation on the one image sample to obtain a plurality of augmented samples corresponding to the one image sample; Extracting features from the plurality of augmented samples using the visual encoder to obtain a plurality of learning features; Based on the learning loss between each learning feature and the reference feature, the parameters of the visual encoder are adjusted until the second training end condition is met, and the initial parameters of the visual encoder are determined.

5. The method according to claim 2 or 3, characterized in that The visual encoder extracts the visual features of each of the multiple image samples by the following method: For the multiple image samples, perform the following operations respectively: Inputting an image sample into the visual encoder, performing feature extraction on the image sample through multiple convolutional layers in the visual encoder, and obtaining a global visual feature of the image sample; Based on a preset region division rule, performing region of interest pooling on each local region of the image sample to obtain multiple local visual features of the image sample; The global visual features and multiple local visual features of the image sample are fused through the embedding layer in the visual encoder to obtain the visual features of the image sample.

6. The method according to any one of claims 2 to 4, characterized in that The obtaining of the training sample set includes: Get multiple candidate samples and perform the following operations on each candidate sample: Performing image recognition on a candidate sample using the large visual model to obtain a global text annotation and a local text annotation of the candidate sample; Performing feature extraction on the candidate sample to obtain global visual features and local visual features of the candidate sample, and performing feature extraction on the global text annotation and local text annotation of the candidate sample to obtain global text features and local text features of the candidate sample; Calculate the similarity between the global visual features and the global text features, as well as the similarity between the corresponding local visual features and the local text features; When the calculated similarities are all greater than a preset similarity threshold, the candidate sample is determined to be an image sample; otherwise, the candidate sample is determined not to be an image sample.

7. The method according to any one of claims 1 to 4, characterized in that When the database to be retrieved includes a video to be retrieved, the image to be retrieved includes at least one frame of image in the video to be retrieved; When at least one image to be retrieved that meets the preset retrieval condition includes an image belonging to the video to be retrieved, it is determined that the retrieval result includes the video to be retrieved.

8. A cross-modal retrieval method and device, characterized in that: include: An acquisition unit, used to acquire the search text; A feature extraction unit, configured to extract target text features of the search text; The image matching unit is used to perform the following operations for each image to be retrieved to obtain the image matching results corresponding to each image to be retrieved: performing similarity matching on the global features and multiple local features of the image to be retrieved and the target text features to obtain corresponding feature matching results; the local features represent the visual features of the specified area of ​​the image to be retrieved; Selecting a feature matching result that meets a preset similarity matching condition from among the multiple feature matching results corresponding to the image to be retrieved as the image matching result corresponding to the image to be retrieved; The retrieval unit is configured to select at least one image to be retrieved that meets a preset retrieval condition from among the images to be retrieved based on the obtained image matching results as a retrieval result corresponding to the retrieval text.

9. A computer device, characterized in that: The method comprises a processor and a memory, wherein the memory stores program codes, and when the program codes are executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The method comprises a program code, and when the program code is run on a computer device, the program code is used to enable the computer device to execute the steps of the method according to any one of claims 1 to 7.

Images