Image-text mutual retrieval method and system based on neighbor perception and complementary semantic enhancement

Abstract

The invention provides an image-text mutual retrieval method and system based on neighbor perception and complementary semantic enhancement, and the method comprises the steps: obtaining to-be-retrieved information and a retrieval target database, the to-be-retrieved information being a text and retrieval target data being an image, or the to-be-retrieved information being an image and retrieval target data being a text; performing instance-level feature extraction and fragment-level feature extraction on the to-be-retrieved information and the retrieval target in the retrieval target database; based on the to-be-retrieved information and the instance-level features and the fragment-level features of the retrieval targets in the retrieval target database, the similarity between the to-be-retrieved information and the retrieval targets in the retrieval target database is obtained through a pre-constructed neighbor perception complementary semantic enhancement network, and an image or text retrieval result is obtained based on the similarity.

Description

Technical Field

[0001] The present invention belongs to the technical field of image and text retrieval, and in particular relates to an image and text mutual retrieval method and system based on neighbor perception and complementary semantic enhancement. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Considering the exponential expansion of multimodal data on the Internet, cross-modal retrieval has become an important and fundamental issue in academia and industry. Among various cross-modal retrieval tasks, image-text retrieval (ITR) is a challenging task. Existing image-text retrieval methods are mainly divided into two categories: one is ITR based on instance-level alignment, and the other is ITR based on fragment-level alignment. Among them, ITR based on instance-level alignment aims to obtain the instance-level correspondence between the whole image and the sentence, while ITR based on fragment-level alignment studies the fragment-level local correspondence between the visual area and the text words.

[0004] The inventors found that ITR based on instance-level alignment mainly focuses on instance-level alignment between images and sentences, ignoring the fact that instance-level alignment usually originates from fragment-level alignment of complex aggregation between specific image regions and text words; while ITR based on fragment-level alignment mainly focuses on the correspondence between local regions of images and words or phrases in text. The advantage of this method is that it can capture richer semantic information and improve the accuracy of retrieval. However, such methods still have some defects, mainly including: Incompleteness of modal information: Fragment-level alignment methods may not be able to fully capture all relevant information in images or texts. For example, some details in images may not be mentioned in the corresponding text descriptions, or some descriptions in the text may not directly correspond to specific areas of the image. The lack of such modal information may lead to inaccurate retrieval results; at the same time, although some researchers have used a strategy combining fragment-level alignment with instance-level alignment to design image-text mutual retrieval, existing methods ignore potential semantic associations, resulting in low retrieval accuracy. Summary of the invention

[0005] In order to overcome the deficiencies of the above-mentioned prior art, the present invention provides a method and system for image-text mutual retrieval based on neighbor perception and complementary semantic enhancement, so as to solve the problem that traditional schemes usually only consider a single level of semantic information, resulting in low retrieval accuracy; and the existing schemes considering multi-level semantic information fusion ignore the potential semantic associations, that is, do not consider the use of complementary semantic information, resulting in poor accuracy of similarity calculation in the cross-modal retrieval process of images and texts.

[0006] According to a first aspect of an embodiment of the present invention, a method for image-text mutual retrieval based on neighbor perception and complementary semantic enhancement is provided, comprising:

[0007] Acquire information to be retrieved and a retrieval target database, wherein the information to be retrieved is text and the retrieval target data is an image, or the information to be retrieved is an image and the retrieval target data is text;

[0008] Perform instance-level feature extraction and segment-level feature extraction on the information to be retrieved and the retrieval target in the retrieval target database respectively;

[0009] Based on the instance-level features and segment-level features of the information to be retrieved and the retrieval target in the retrieval target database, the similarity between the information to be retrieved and the retrieval target in the retrieval target database is obtained through a pre-built neighbor-aware complementary semantic enhancement network, and the image or text retrieval result is obtained based on the similarity;

[0010] Among them, the neighbor-aware complementary semantic enhancement network specifically performs the following processing procedures: calculating the instance-level image-text similarity matrix based on the instance-level features of the information to be retrieved and the retrieval target; calculating the fragment-level image-text similarity matrix based on the fragment-level features of the retrieval target in the information to be retrieved and the retrieval target database; wherein, in the process of calculating the instance-level image-text similarity matrix and the fragment-level image-text similarity matrix, the neighbor relationship of the information to be retrieved and the retrieval target in the preset data set is obtained respectively, and the neighbor-level image-text similarity matrix is ​​obtained based on the neighbor relationship, and the obtained instance-level image-text similarity matrix and fragment-level image-text similarity matrix are optimized with the neighbor-level image-text similarity matrix as supplementary semantics, and the image-text similarity result enhanced with complementary semantics is obtained based on the weighted fusion of the optimized instance-level image-text similarity matrix and the fragment-level image-text similarity matrix.

[0011] Furthermore, the instance-level image-text similarity matrix is ​​calculated by the following formula:

[0012]

[0013] in, is the instance-level feature vector extracted from the image, is the instance-level feature vector extracted from the text, sim S (v i ,r j ) is the image v i With text r j The instance-level image-text similarity between .

[0014] Furthermore, the segment-level image-text similarity matrix is ​​calculated by the following formula:

[0015]

[0016] in, Retrieve global text similarity for images, Retrieve the global similarity of the image for text, where X is the number of segments in the image, Y is the number of words in the text, x is the xth segment, y is the yth segment, and v i is the image, r j For text, for and The correlation score between for and The relevance score of is the x-th image segment feature, represents the xth text context feature, represents the feature of the yth text word, Represents the y-th image context feature.

[0017] Furthermore, the obtained instance-level image-text similarity matrix and fragment-level image-text similarity matrix are optimized using the neighbor-level image-text similarity matrix as supplementary semantics, wherein a hybrid loss function based on triple ranking loss and consistency ranking loss is adopted in the optimization process.

[0018] Furthermore, the instance-level feature extraction is specifically as follows: for image data, the original image is used as the input of a pre-trained deep learning model to obtain instance-level features of the image; for text data, each word in the original text data is one-hot encoded, and based on the encoding result, the feature vector representation of each word is obtained through a bidirectional gated recurrent unit, and based on the mean of the feature vector representations of all words in the text, the instance-level features of the text are obtained.

[0019] Furthermore, the segment-level feature extraction is specifically as follows: for image data, the original image is divided into several segments, and the features of each segment are extracted through a pre-trained deep learning model, and the set consisting of the features of all the segments is used as the segment-level features of the image; for text data, the features of each word in the original text data are obtained through a bidirectional gated recurrent unit, and the set consisting of the features of all the words is used as the segment-level features of the text.

[0020] According to a second aspect of an embodiment of the present invention, a picture-text mutual retrieval system based on neighbor perception and complementary semantic enhancement is provided, comprising:

[0021] A data acquisition unit, which is used to acquire information to be retrieved and a search target database, wherein the information to be retrieved is text and the search target data is an image, or the information to be retrieved is an image and the search target data is text;

[0022] A feature extraction unit, which is used to perform instance-level feature extraction and segment-level feature extraction on the information to be retrieved and the retrieval target in the retrieval target database respectively;

[0023] A retrieval result acquisition unit, which is used to obtain the similarity between the information to be retrieved and the retrieval target in the retrieval target database based on the instance-level features and segment-level features of the information to be retrieved and the retrieval target in the retrieval target database through a pre-built neighbor-aware complementary semantic enhancement network, and obtain an image or text retrieval result based on the similarity;

[0024] Among them, the neighbor-aware complementary semantic enhancement network specifically performs the following processing procedures: calculating the instance-level image-text similarity matrix based on the instance-level features of the information to be retrieved and the retrieval target; calculating the fragment-level image-text similarity matrix based on the fragment-level features of the retrieval target in the information to be retrieved and the retrieval target database; wherein, in the process of calculating the instance-level image-text similarity matrix and the fragment-level image-text similarity matrix, the neighbor relationship of the information to be retrieved and the retrieval target in the preset data set is obtained respectively, and the neighbor-level image-text similarity matrix is ​​obtained based on the neighbor relationship, and the obtained instance-level image-text similarity matrix and fragment-level image-text similarity matrix are optimized with the neighbor-level image-text similarity matrix as supplementary semantics, and the image-text similarity result enhanced with complementary semantics is obtained based on the weighted fusion of the optimized instance-level image-text similarity matrix and the fragment-level image-text similarity matrix.

[0025] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, comprising a memory, a processor, and a computer program stored and running on the memory, wherein when the processor executes the program, the method for mutual retrieval of images and texts based on neighbor perception and complementary semantic enhancement is implemented.

[0026] According to a fourth aspect of an embodiment of the present invention, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method for mutual retrieval of images and texts based on neighbor perception and complementary semantic enhancement is implemented.

[0027] According to a fifth aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, characterized in that when the computer program is executed by a processor, it implements the above-mentioned image-text mutual retrieval method based on neighbor perception and complementary semantic enhancement.

[0028] One or more of the above technical solutions have the following beneficial effects:

[0029] The present invention provides a method and system for image-text mutual retrieval based on neighbor perception and complementary semantic enhancement. The scheme can effectively improve the accuracy of image-text similarity calculation by comprehensively capturing and effectively integrating complementary semantics distributed at the instance level, fragment level and neighbor level through the proposed neighbor perception complementary semantic enhancement network.

[0030] The scheme of the present invention converts the intramodal correlation into cross-modal correlation, estimates the image-text similarity at the neighbor level based on the KNN (K nearest neighbor) relationship, and thus mines the complementary semantics at the neighbor level. By introducing the complementary semantics at this level, the accuracy of image-text retrieval (ITR) can be significantly improved.

[0031] The scheme described in the present invention optimizes multi-level image-text similarity through a proposed hybrid loss, including a triple ranking loss and a consistency ranking loss, wherein the former aims to separate relevant and irrelevant samples in a bipolar manner, while the latter focuses on distinguishing irrelevant samples in a consistent manner; a multi-margin sampling strategy is adopted to scan the embedding space through the consistency ranking loss, and the embedding function is optimized under the guidance of the image-text similarity at the neighbor level, thereby further improving the accuracy of retrieval.

[0032] The solution described in the present invention proposes to use complementary semantics at the neighbor level to supplement the information that may be missing in the fragment-level and instance-level alignment. This method can discover semantic information that cannot be captured in direct alignment by analyzing the neighbor relationship of images or texts in the data set. For example, if an image contains a specific object that is not mentioned in the text description, then by analyzing the neighbor images of the image and their corresponding text descriptions, it may be possible to find semantic information related to the object, thereby supplementing and enhancing the retrieval results.

[0033] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0035] Figure 1 are examples of the three complementary semantic levels described in the embodiments of the present invention;

[0036] Figure 2 A schematic diagram of a neighbor-aware complementary semantic enhancement network structure described in an embodiment of the present invention;

[0037] Figure 3 is a flowchart of the neighbor level image-text similarity calculation described in an embodiment of the present invention;

[0038] Figure 4 It is a schematic diagram of the embedding space before and after the optimization based on the consistency sorting loss described in an embodiment of the present invention;

[0039] Figure 5 It is a schematic diagram of a PR curve of the comparison results between the solution of the present invention and the existing solution in the Corel 5K data set described in an embodiment of the present invention;

[0040] Figure 6 It is a schematic diagram of a PR curve of the comparison results of the solution of the present invention and the existing solution in the Pascal Sentence data set described in the embodiment of the present invention;

[0041] Figure 7 It is a schematic diagram of a PR curve of the comparison results of the solution described in the present invention and the existing solution in the Wiki data set described in the embodiment of the present invention;

[0042] Figure 8 The present invention is a flowchart of a method for image-text mutual retrieval based on neighbor perception and complementary semantic enhancement described in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0044] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0045] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0046] 1) Understandably, a promising approach to address the heterogeneity gap is to identify and exploit complementary semantics between modalities. However, the complementary nature of images and texts poses a challenge as they may contain different amounts of information to represent the same semantic concept. Therefore, previous approaches have mainly focused on mining the imbalanced and complementary relationships between different data modalities (e.g., images and texts) and granularities (e.g., images and image fragments). That is, these approaches mainly focus on two levels of complementary semantics:

[0047] 2) Instance level: complementary semantics between instances of different modalities;

[0048] 3) Fragment level: complementary semantics between fine-grained fragments.

[0049] However, these traditional methods may fail to capture enough complementary semantics to accurately measure cross-modal similarity.

[0050] like Figure 1 As shown, three levels of complementary semantics associated with the concept of “airplane” are demonstrated, which exist in different modalities, different granularities, and visual and textual neighbors. In particular, Figure 1 (a) and (b) in the figure describe the complementarity under different modes and granularity. Figure 1 (c) and (d) in Figure 3 show the complementarity in the neighbors of the target instance. Specifically, the semantic elements at the instance level, fragment level, and neighbor level are marked with green, blue, and red boxes, respectively. To effectively bridge the heterogeneous gap and achieve accurate image-text similarity, our proposed Neighbor-Aware Complementary Semantic Enhancement Network (NCSE) captures the complementary semantics of these three levels through the S branch, F branch, and N branch, respectively. Subsequently, these levels are integrated and optimized through a hybrid loss.

[0051] First, in Figure 1 (a) and (b) in Figure 3 demonstrate the complementarity of different modalities and granularity. The green box depicts the overall semantic concept of “airplane” presented through the entire image and text. In contrast, the blue box represents the detailed aspects of “airplane” revealed through image fragments and phrases. We observe that some semantic information can be accurately conveyed by text, but not by images. For example, Figure 1 The green box in (b) provides information about the aircraft belonging to "Lufthansa" and its model is "Boeing 747". Similarly, Figure 1 The green box in (a) contains semantic information such as "sky" and "cloud" that are not present in the text. In addition, Figure 1 The blue boxes in (a) correspond to specific parts of the aircraft, including "engine", "landing gear" and "wing". Figure 1 The blue boxes in (b) represent segment-level semantic details, such as “large passenger plane”, “just took off” and “Lufthansa Boeing 747”.

[0052] Secondly, Figure 1 The complementarity of the target instance neighbors is explained in Figure 1 As shown in (c) and (d) in Figure 1 In (c), the visual neighbors of the target image clearly depict the semantic elements of “aircraft service car” and “terminal”. Figure 1 In (d) of the figure, the textual neighbors of the target text contain the semantic elements of “tarmac” and “in the terminal”. It is worth noting that these semantic elements are Figure 1The target instances in (a) and (b) are not fully described in the figure. Therefore, the semantic information of neighbors serves as a valuable supplement to the target instance and enhances the overall semantic understanding.

[0053] For ease of understanding, the following is a brief description of the traditional image-text retrieval (ITR) method and the basic content of deep metric learning:

[0054] (1) Image-text retrieval

[0055] In recent years, thanks to the progress in text mining and multimedia information processing, a large number of studies have emerged in the field of ITR. These studies can be divided into two categories:

[0056] 1) ITR based on instance-level alignment;

[0057] 2) ITR based on fragment-level alignment.

[0058] The first category aims to obtain instance-level correspondence between the overall image and the sentence, while the second category studies the fragment-level local correspondence between the visual region and the text words, which are described in detail below:

[0059] 1) Traditional ITR based on instance-level alignment:

[0060] Instance-level alignment aims to discover a mapping matrix that projects heterogeneous data into a shared embedding space, providing a global common representation for each instance. Therefore, many studies have proposed methods to generate superior feature representations to accommodate both modalities.

[0061] Initially, canonical correlation analysis (CCA) and its variants were a reliable embedding method. It learns a linear mapping that projects two modalities into a shared space to maximize the correlation between them. Gong et al. developed multi-view CCA by constructing a third view of high-level semantics. With the advantages of deep neural networks (DNNs) in extracting important visual and textual features, researchers use DNNs to learn the shared space of ITR. For example, DCCA proposed by Andrew et al. extends traditional CCA through a deep network structure to maximize the correlation of the two sub-networks. Yan et al. combined DNN with CCA to associate image and text representations through deep canonical correlation analysis. Feng et al. introduced the corresponding autoencoder (Corr-AE) to alleviate the correlation learning error between different modalities through feature mapping in a shared space.

[0062] However, the above methods mainly focus on instance-level alignment between images and sentences, ignoring that instance-level alignment usually originates from fragment-level alignment of complex aggregations between specific image regions and text words. Therefore, the relationships learned by instance-level alignment methods are relatively rough and require interpretability, which can be improved by fragment-level alignment methods, as described below.

[0063] 2) Traditional ITR based on fragment-level alignment

[0064] The fragment-level alignment-based method can infer the correlation between image regions and text words, thereby improving the accuracy of ITR and gradually becoming the mainstream of ITR.

[0065] Recently, Peng et al. proposed cross-modal correlation learning (CCL) to improve performance by leveraging fragment-level information and adopting multi-task learning techniques. Lee et al. introduced stacked cross attention mechanism (SCAN) to match fragment-level details of image and text modalities using attention mechanism to promote the inference of image-text similarity. Subsequently, Chen et al. constructed additional semantic consistency constraints based on SCAN to enhance fragment-level alignment and achieve good performance. However, due to the complexity of semantics, these methods may not be able to fully obtain the best fragment-level correspondence. Therefore, selectively focusing on local regions is a solution to find the best fragment alignment. Liu et al. tried to selectively align local semantics across modalities. PFAN++ proposed to incorporate object location information into its visual representation to improve retrieval performance. Since then, multiple methods with similar goals have emerged one after another, such as Zhuge et al. established an alignment-guided masking method and Diao et al. adopted an attention filtering method.

[0066] (2) Deep Metric Learning

[0067] Metric learning aims to measure similarity through a loss function to ensure that semantically related samples are close and irrelevant samples are effectively separated. In previous work, a variety of metric learning methods have been proposed for different tasks, which can be mainly divided into two categories: 1) non-ranking loss and 2) ranking loss. Specifically:

[0068] 1) Non-ranking loss

[0069] Non-ranking loss is a loss function used for ITR, but unlike ranking loss, non-ranking loss does not consider the relative ranking of samples, but treats each sample independently. Although it does not involve sample ranking, non-ranking loss still considers the similarity or distance between samples while minimizing the difference or distance between samples. Therefore, in many ITR studies, non-ranking loss functions have achieved good performance. Although these loss functions perform well in guiding the model to effectively distinguish image-text pairs of different categories, ensuring that image-text pairs of the same category are close and image-text pairs of different categories are separated. However, they are not enough to capture the subtle and continuous semantic associations between images and texts. Therefore, other methods need to be explored to effectively give deep learning models the ability to learn and understand complex semantic relationships between images and texts.

[0070] (2) Ranking loss

[0071] Unlike cross entropy loss or mean squared error loss, non-ranking loss functions aim to predict labels or values. Ranking loss functions can consider the relationship between samples, such as the similarity or distance between positive and negative samples, and better capture the interactive information between samples. Diao et al. used a bidirectional triplet ranking loss function to train the proposed SGR and SAF module (SGRAF). However, the traditional triplet loss does not continuously mine sample pairs, making the mining of image-text pairs not comprehensive enough. To solve this problem, Zhou et al. proposed a ladder loss, which can alleviate the discontinuity problem and enhance the stability and convergence of the optimization process. The ladder loss function depends on multiple hyperparameters, and fine-tuning these hyperparameters is challenging and time-consuming. Therefore, it cannot be ensured that the model always continuously mines all valuable image-text pairs. Therefore, we design a hybrid loss to ensure the continuity of image-text pair mining and the simplicity of parameters.

[0072] Based on the above theory and in combination with the problems existing in the prior art, in one or more embodiments, Figure 8 As shown, a method for image-text mutual retrieval based on neighbor perception and complementary semantic enhancement is provided, including the following processing steps:

[0073] Step 1: Obtain information to be retrieved and a retrieval target database, wherein the information to be retrieved is text and the retrieval target data is an image, or the information to be retrieved is an image and the retrieval target data is text;

[0074] Step 2: Perform instance-level feature extraction and segment-level feature extraction on the information to be retrieved and the retrieval target in the retrieval target database respectively;

[0075] In a specific implementation, the instance-level feature extraction is specifically as follows: for image data, the original image is used as the input of a pre-trained deep learning model to obtain instance-level features of the image; for text data, each word in the original text data is one-hot encoded, and based on the encoding result, the feature vector representation of each word is obtained through a bidirectional gated recurrent unit, and based on the mean of the feature vector representations of all words in the text, the instance-level features of the text are obtained.

[0076] In a specific implementation, the segment-level feature extraction is specifically as follows: for image data, the original image is divided into several segments, and the features of each segment are extracted through a pre-trained deep learning model, and the set consisting of the features of all the segments is used as the segment-level features of the image; for text data, the features of each word in the original text data are obtained through a bidirectional gated recurrent unit, and the set consisting of the features of all the words is used as the segment-level features of the text.

[0077] Specifically, the feature extraction includes the following processing steps:

[0078] Consider a dataset The dataset consists of N pairs of images and texts. We first extract features at the instance level and the segment level, and then encode them into a shared embedding space. For image feature extraction, we adopt convolutional neural network (CNN), while for text feature extraction, we use bidirectional GRU (Bi-GRU). In addition, the image v i contains X fragments, and the text r j Consists of Y words.

[0079] Among them, instance-level feature extraction is specifically as follows:

[0080] In this embodiment, we take the image v i and text r j The instance-level feature vector of and The specific instance-level feature extraction process is described as follows.

[0081] In one or more embodiments, the image v i The eigenvector of It can be extracted from the FC-7 layer of AlexNet pre-trained on ImageNet. Subsequently, the feature vector is converted to a 1024-dimensional embedding space through a fully connected layer.

[0082]

[0083] Among them, W S and b S They represent the weight matrix and bias term respectively, which are the parameters that need to be optimized during training.

[0084] In addition, the extraction of text features is divided into two steps. First, the words in each text are one-hot encoded, where each word is represented by a binary vector with all zeros, and only the position corresponding to the index of the word in the vocabulary is 1. In particular, the text r j The unique hot vector of the yth word in is represented as Second, the raw text features are converted into a 1024-dimensional embedding space to directly compare images with text. To this end, we adopt a bidirectional gated recurrent unit (Bi-GRU) to capture the text r j The context information is processed from both forward and reverse directions.

[0085]

[0086] in, and Represent the forward and reverse hidden states of the Bi-GRU, respectively. j The feature vector of the yth word in is represented as follows:

[0087]

[0088] Finally, the text j The feature representation at the instance level is obtained by calculating the mean of all word vectors, expressed as Where Y represents the text r j The number of words in .

[0089] In a specific implementation, the segment-level feature extraction is specifically as follows:

[0090] We will image v i Divide evenly into The features of these fragments are extracted using AlexNet. Subsequently, they are mapped to a 1024-dimensional embedding space, and the feature vector at the fragment level is represented as in Represents image v i The feature vector of the xth segment in v i In addition, we use Bi-GRU to generate r j The features of each word in and generate r j The segment-level feature vector in Represents text r j The feature vector of the yth word in , and Y represents r j The number of words in .

[0091] Step 3: Based on the instance-level features and segment-level features of the information to be retrieved and the retrieval target in the retrieval target database, the similarity between the information to be retrieved and the retrieval target in the retrieval target database is obtained through a pre-built neighbor-aware complementary semantic enhancement network, and the image or text retrieval result is obtained based on the similarity;

[0092] Among them, the neighbor-aware complementary semantic enhancement network specifically performs the following processing procedures: calculating the instance-level image-text similarity matrix based on the instance-level features of the information to be retrieved and the retrieval target; calculating the fragment-level image-text similarity matrix based on the fragment-level features of the retrieval target in the information to be retrieved and the retrieval target database; wherein, in the process of calculating the instance-level image-text similarity matrix and the fragment-level image-text similarity matrix, the neighbor relationship of the information to be retrieved and the retrieval target in the preset data set is obtained respectively, and the neighbor-level image-text similarity matrix is ​​obtained based on the neighbor relationship, and the obtained instance-level image-text similarity matrix and fragment-level image-text similarity matrix are optimized with the neighbor-level image-text similarity matrix as supplementary semantics, and the image-text similarity result enhanced with complementary semantics is obtained based on the weighted fusion of the optimized instance-level image-text similarity matrix and the fragment-level image-text similarity matrix.

[0093] In a specific implementation, the instance-level image-text similarity matrix is ​​calculated using the following calculation formula:

[0094]

[0095] in, is the instance-level feature vector extracted from the image, is the instance-level feature vector extracted from the text, sim S (v i ,r j ) is the image v i With text r j The instance-level image-text similarity between .

[0096] In a specific implementation, the segment-level image-text similarity matrix is ​​calculated using the following calculation formula:

[0097]

[0098] in, Retrieve global text similarity for images, Retrieve the global similarity of the image for text, where X is the number of segments in the image, Y is the number of words in the text, x is the xth segment, y is the yth segment, and v i is the image, r j For text, for and The correlation score between for and The relevance score of is the x-th image segment feature, represents the xth text context feature, represents the feature of the yth text word, Represents the y-th image context feature.

[0099] In a specific implementation, the derivation process of the above formula is as follows:

[0100] For calculation

[0101] First, calculate the similarity between the image segment and the text segment:

[0102] Then normalize the similarity:

[0103] Then form the text context features:

[0104] Then calculate the similarity between text context features and image fragments:

[0105] The final polymer formation

[0106] For calculation

[0107] First, calculate the similarity between the image segment and the text segment:

[0108] Then normalize the similarity:

[0109] Then form the image context features:

[0110] Then calculate the image context features and text word similarity:

[0111] The final polymer formation

[0112] in, Represents the global similarity of image retrieval text, which is the global similarity obtained by aggregating the previous segment-level similarities; Represents the global similarity of text retrieval images, which is the global similarity obtained by aggregating the previous segment-level similarities.

[0113] In a specific implementation, the neighbor-level image-text similarity matrix is ​​used as a supplementary semantic to optimize the obtained instance-level image-text similarity matrix and the fragment-level image-text similarity matrix, wherein a hybrid loss function based on triple ranking loss and consistency ranking loss is adopted in the optimization process.

[0114] In the solution described in this embodiment, it is first necessary to define the ITR (i.e., image text retrieval) problem:

[0115] Assume that the image-text dataset is represented as There are N images and N paragraphs of text In particular, there are a total of N image-text pairs in D. We consider an image Q I As query input. Retrieval results R in the direction of "image to text (I2T)" T is designed to provide a set of text, like this:

[0116]

[0117] Similarly, for the text query Q T , retrieval in the direction of “text to image (T2I)” aims to return a set of images R I , as shown below:

[0118]

[0119] Among them, the function sim(·) is used to determine the image-text similarity. Therefore, the main goal of the ITR problem is to calculate the cross-modal similarity between image and text. In order to learn a more effective function sim(·) by capturing more complementary semantics, this embodiment proposes the following neighbor-aware complementary semantic enhancement network (hereinafter referred to as NCSE).

[0120] The overall structure of the Neighbor-aware Complementary Semantic Enhancement (NCSE) network is described in detail below. The network consists of three branches:

[0121] 1) Instance-level image-text similarity calculation branch (S-branch);

[0122] 2) Fragment-level image-text similarity calculation branch (F-branch);

[0123] 3) Neighborhood-level image-text similarity calculation branch (N-branch).

[0124] In particular, the complementary semantics of visual and textual neighbors captured from N-branch are integrated with S-branch and F-branch through a hybrid loss. Subsequently, the similarities learned from S-branch and F-branch are fused to perform image-text retrieval (ITR).

[0125] Let R and V denote the text modality and image modality respectively. For a set of image-text pairs (v i ,r i ), where v i ∈V and r i ∈R, and 1≤i≤N, our goal is to jointly learn two embedding functions for each branch.

[0126] For example: In the case of S-branch, we aim to learn the embedding function and Specifically, the embedding function and Two conditions must be met at the same time:

[0127] 1) Image v i The text r that should be matched with it in the visual semantic embedding space i The distance between them is greater than that between other texts r j The distance between (i≠j) is closer.

[0128] 2) Text i The image v that should be matched with it in the visual semantic embedding space i The distance between them is greater than that between other images v j The distance between (i≠j) is closer.

[0129] To illustrate the joint learning process of the embedding function, we will take the S branch as an example. The objective function of the S branch is Used to optimize the instance-level image-text similarity matrix M S , which is expressed as follows:

[0130]

[0131] in, is the hybrid loss function proposed in the solution described in this embodiment, M S is the instance-level image-text similarity matrix. It should be noted that the similarity matrix M S The element at position (i,j) in is represented by The calculation formula is as follows:

[0132]

[0133] Among them, sim S (vi ,r j ) represents the image v i With text r j The instance-level image-text similarity between .

[0134] In order to fully exploit the complementary semantics at the instance level during the joint embedding optimization process, the hybrid loss function is defined as follows:

[0135]

[0136] in, and denote the triple ranking loss and consistency ranking loss, respectively. In addition, φ1 and φ2 are used as balancing factors for triple ranking loss and consistency ranking loss. In addition, M N represents the neighbor-level image-text similarity matrix, The definitions are as follows.

[0137]

[0138] Among them, sim N (v i ,r j ) represents the image v i With text r j Neighbor-level image-text similarity between .

[0139] and Similarly, the segment-level objective function of F-branch is The definitions are as follows.

[0140]

[0141] Among them, M F is the segment-level image-text similarity matrix. By minimizing the objective function and The optimal similarity matrix with complementary semantic enhancement can be obtained and Specifically, the optimal embedding representations of images and texts in S-branch and F-branch are as follows:

[0142]

[0143] in, and represent the optimal embedding of images and text at the instance level (fragment level), respectively, and v S (v F ) and r S (r F) represent the initial embeddings of images and texts at the instance level (fragment level), respectively.

[0144] For the sake of simplicity, this embodiment only illustrates the optimization process of the S-branch in Algorithm 1.

[0145]

[0146] Finally, in order to balance the weights of instance-level similarity and fragment-level similarity, we adopt a linear weight-based fusion strategy to calculate the final image-text similarity matrix:

[0147]

[0148] st0≤θ1,θ2≤1

[0149] in, For image-text retrieval (ITR), hyperparameters θ1 and θ2 are used to balance the importance of the S-branch and the F-branch.

[0150] like Figure 2 As shown in FIG. 1 , the overall architecture of the NCSE network proposed in this embodiment is provided, in which nine image-text pairs related to three semantic concepts (“dog”, “airplane”, and “bus”) are provided, represented by red, blue, and green, respectively. In each image-text similarity matrix, the top three rows and the leftmost three columns represent “dog”, the middle three rows and three columns correspond to “airplane”, and the bottom three rows and the rightmost three columns involve “bus”. We sort all elements according to the value of each row and assign predefined colors only to the highest-ranked instances related to the correct semantic concept. By combining the best results of the S-branch and the F-branch, the matrix Three different diagonal pieces are shown, similar to the completion of a puzzle. Note that It represents the result of complementary semantic enhancement and contains extensive complementary semantic information from multiple modalities, multiple granularities, and visual and textual neighbors.

[0151] The NCSE network framework consists of three branches: S-branch, F-branch and N-branch, which are responsible for generating instance-level, fragment-level and neighbor-level image-text similarities respectively. In the N-branch, the complementary semantics in visual and textual neighbors are captured by converting intra-modal correlations into inter-modal correlations. Subsequently, the neighbor-level image-text similarity enhances the instance-level and fragment-level image-text similarities through a hybrid loss function. Finally, the enhanced instance-level and fragment-level image-text similarities are fused to achieve image-text retrieval (ITR).

[0152] Specifically, the multi-level image-text similarity calculation includes the following processing steps:

[0153] S-branch, F-branch, and N-branch are used to calculate instance-level, segment-level, and neighbor-level similarities between images and texts, respectively. It is worth noting that in order to improve the accuracy of learning segment-level associations between images and texts, we introduce an attention mechanism in F-branch. This mechanism enables comprehensive aggregation of segment-level matches between image segments and text words.

[0154] Specifically, for S-branch: instance-level image-text similarity calculation branch, the specific processing process includes the following:

[0155] From the image v i and text r j The feature vector extracted from and Input to S-branch, the instance-level image-text similarity can be calculated as follows.

[0156]

[0157] For F-branch: segment-level image-text similarity calculation branch, the specific processing includes the following:

[0158] In order to study the potential alignment relationship between image and text features at the segment level, we construct the F-branch. and As input, and calculate the image v i With text r j Similarity at the segment level. Specifically, the image-text similarity calculation of the F-branch is as follows:

[0159]

[0160] To simplify the explanation, we take the cross-attention embedding space V→R as an example to show the segment-level image-text similarity The detailed steps are as follows:

[0161] First, we use the segment-level feature vector and Calculate the image-text similarity matrix M F This matrix captures the relationship between all fragment-word pairs. Specifically, the xth th The fragment and the yth th The relationship between the words is expressed by the following formula:

[0162]

[0163] Next, we use the matrix M F The column dimensions are normalized as follows:

[0164]

[0165] For image v i The xth th fragments, text context feature vector It is obtained by weighted combination of word embedding and attention mechanism. Specifically, The calculation is as follows:

[0166]

[0167] Among them, the parameter λ is the inverse temperature parameter of the softmax function, which is used to control the smoothness of the attention distribution. In order to evaluate the importance of each image fragment in a specific text context, we estimate it by the following formula and Correlation score between:

[0168]

[0169] Similarity It is obtained by calculating the average of all relevance scores, and the formula is as follows:

[0170]

[0171] For N-branch: neighbor-level image-text similarity calculation, the specific processing includes the following:

[0172] In this branch, image-text similarity is estimated by searching for K nearest neighbors, and the neighbor relationship is used to provide complementary semantics for the S-branch and F-branch. First, we define the KNN relationship as follows.

[0173] Definition 2 (KNN-Relation). Suppose there is a data set X = {x1, x2, …, x n The KNN relation <·,·> on X×X is expressed as the statement <x i ,x j >, if x j ∈X\{x i} belongs to x i The K nearest neighbors of x, where the “\” symbol is used to represent the difference operation of a set. For example, x j ∈X\{x i} means that in set X except element x i All elements except .

[0174] K is a hyperparameter that needs to be determined based on the specific application scenario and the size of the data set. The value of K determines how many neighbors of each data point will be taken into account. Usually, the choice of K is determined through experiments to find the best performance balance point. In this embodiment, the value of K is 8, that is, 8 neighbors are found for an image and 8 neighbors are found for a text.

[0175] The meaning of nearest neighbor: The nearest neighbor is usually defined based on some distance metric, such as Euclidean distance, cosine similarity, etc. In the solution described in this embodiment, for a given data point x i , its K nearest neighbors refer to the distance x in the data set i The nearest K points.

[0176] Determine the K nearest neighbors: According to the KNN algorithm, for each data point x in the data set i , we calculate its distance from all other points in the dataset. Then, we select the K points with the smallest distance as x i In a specific implementation, for an image or text to be retrieved, the steps of calculating the K nearest neighbor data set are as follows:

[0177] Constructing a data set: The K nearest neighbor data set is a candidate neighbor data set constructed from each training data set.

[0178] Feature extraction: For each image and each text in the dataset, feature vectors need to be extracted, which will be used to calculate similarity.

[0179] Distance calculation: Use some distance metric (such as Euclidean distance or cosine similarity) to calculate the distance between the feature vector of the image or text to be retrieved and the feature vector of each image or text in the dataset.

[0180] Select K nearest neighbors: For the image or text to be retrieved, select the K images or texts with the smallest distance as its K nearest neighbors based on the calculated distance.

[0181] Similarity calculation: Using these K nearest neighbors, the similarity between the image or text to be retrieved and other images or texts in the dataset can be calculated for retrieval or recommendation.

[0182] Obviously, the <·,·> in the above definition is an antisymmetric relation. <x i ,x j > Established, <x j ,x i > is not necessarily true. Based on Definition 2, we calculate the image v i and text r j The neighbor-level similarity between is as follows:

[0183]

[0184] in, Is to use v i The K nearest neighbors of v are used to calculate i With r j Similarities between It means using r j The K nearest neighbors of v are used to calculate i With r j The similarities between.

[0185] Secondly, for the image v i , whose K nearest visual neighbors are represented as in Then, the similarity function Designed as follows:

[0186]

[0187] in

[0188]

[0189] Representing images The paired texts are used to calculate the cosine similarity.

[0190] Furthermore, for the text r j , whose K nearest text neighbors are represented as in Then, the similarity function Designed as follows:

[0191]

[0192] in, Established, Represents text Paired images of .

[0193] Specifically, Figure 3 As shown, a flowchart of the neighbor-level image-text similarity calculation is shown, such as Figure 3 As shown, for the target image v i and the target text r j , we collected five visual neighbors and its corresponding paired text and five text neighbors and its corresponding paired image Since the KNN relationship is antisymmetric, the neighbor-level image-text similarity is calculated in the cross-attention embedding space V→R and R→V based on visual neighbors and textual neighbors.

[0194] In a specific implementation, the hybrid loss of the NCSE network consists of two parts:

[0195] 1) Triplet sorting loss;

[0196] 2) Consistency sorting loss.

[0197] The former aims to distinguish relevant from irrelevant samples in a bipolar manner, while the latter strives to distinguish all irrelevant samples in a consistent manner.

[0198] In particular, the term “consistency” means that the similarities between the query and candidate samples are consistent with their corresponding relevance levels. Therefore, our proposed hybrid loss not only effectively distinguishes relevant from irrelevant samples, but also accurately re-ranks irrelevant samples.

[0199] Next, we will take the S branch as an example to demonstrate the proposed hybrid loss, where:

[0200] Triplet Ranking Loss When solving the bidirectional image-text retrieval (ITR) problem, we use a bidirectional triplet ranking loss with a fixed margin to ensure that matched image-text pairs have higher similarity than unmatched pairs.

[0201] In addition, we do not compute the loss for all negative samples, but focus on optimizing the most difficult misaligned samples in the mini-batch, which incur the largest loss. For a true matching image-text pair (v i ,r i ), we collect its unmatched pairs and in and Represents the hard negative examples in a mini-batch.

[0202] Therefore, the bidirectional triplet ranking loss with hard negative sample mining can be described as:

[0203]

[0204] The constraints are:

[0205]

[0206] Among them, γ is the margin hyperparameter and B represents the mini-batch size.

[0207] For the consistency ranking loss, Zhou et al. pointed out that in a set of irrelevant samples, some samples are more relevant to the query than other samples. Therefore, accurately distinguishing irrelevant samples can lead to learning a more consistent embedding space. Inspired by the gradient loss, we introduce the Kendall rank correlation coefficient in the proposed consistency ranking loss.

[0208] Kendall's rank correlation coefficient τ∈[-1,1] is a statistical method for determining the association between two variables. It evaluates the correlation of the degree of ranking of the values ​​of the two variables. Specifically, the value of τ ranges from -1 to 1, where -1 indicates an absolute negative correlation, 0 indicates no correlation, and 1 indicates an absolute positive correlation. τ is defined as follows:

[0209]

[0210] Where n is the number of items, n c and n d denote the number of concordant pairs and discordant pairs, respectively. Note that "concordant pairs" and "discordant pairs" refer to pairs of items with the same or different rankings.

[0211] We propose multi-interval sampling, which aims to effectively distinguish all irrelevant samples. We divide the entire embedding space into three different regions: 1) the inner region, 2) the middle region, and 3) the outer region. These regions are determined by two hyperparameters α1 and α2. The cosine similarity between the anchor point and the sample varies in a specific range: [-1, α2) for samples in the outer region, [α2, α1] for samples in the middle region, and (α1, 1) for samples in the inner region.

[0212] Multiple Margin Sampling: We use a set of margins, each with an interval size of β, to scan the middle region. During the scanning process, we adopt a step size of δ. The total number of margins is expressed as Therefore, multi-margin sampling consists of H individual samples {U1,U2,…,U H}, each sample U i Corresponding to the i-th margin. For the i-th margin, we consider the samples inside it to be relevant, while the samples outside it are considered irrelevant.

[0213] By using multi-margin sampling, we can dynamically distinguish between relevant and irrelevant samples. For the i-th sample U i , the cosine similarity between the anchor point and irrelevant samples ranges from [-1,(α2+(i-1)δ)], while the cosine similarity between the anchor point and relevant samples ranges from [(α2+(i-1)δ+β),1].

[0214] Therefore, the consistency ranking loss is defined as follows:

[0215]

[0216] The constraints are:

[0217] sim N (v i ,r j )-sim N (v i ,r k )>β

[0218] sim N (v j ,r i )-sim N (v k ,r i )>β

[0219] In order to clearly demonstrate the main idea of ​​consistency sorting loss, this embodiment Figure 4 The embedding space before and after optimization is shown in Fig.

[0220] like Figure 4 As shown in Figure 2, in the process of optimizing the consistency ranking loss, we aim to learn a more consistent embedding space through complementary semantic guidance at the neighbor level. For a given query, we rank all candidates according to the neighbor-level image-text similarity. In the first boundary, candidate b is located inside it, while candidates a, c, e, and f are located outside. However, the order of candidates a and b in the original embedding space is not consistent with their neighbor-level image-text similarity with the query. Therefore, we use the consistency ranking loss to optimize candidates a and b so that a is closer to the query and b is pushed away. Subsequently, the consistency ranking loss continues to scan the embedding space to obtain the second and third boundaries. In the second (third) boundary, candidate c(e) is pulled closer to the query, while d(f) is pushed away.

[0221] Furthermore, in order to verify the effectiveness of the solution described in this embodiment, the following experimental verification was carried out in this embodiment:

[0222] We present comprehensive experimental results, including parameter sensitivity, convergence analysis, and ablation studies to evaluate the impact of various components in NCSE. The results consistently show significant improvements on all retrieval tasks and datasets, highlighting the broad applicability and effectiveness of our approach.

[0223] (1) Experimental Dataset

[0224] In this experiment, three major image-text datasets were utilized: Corel 5K, Pascal Sentence, and Wikipedia. Each dataset is divided into three subsets, namely 1) training set, 2) validation set, and 3) test set. These datasets provide valuable resources for evaluating and benchmarking various multimodal techniques.

[0225] The Corel 5K dataset is provided by Corel and contains 5,000 images covering 50 semantic concepts. Each concept contains 100 images, and the label dictionary of the dataset contains 260 labels. The images in the dataset are annotated with 1 to 5 labels. During the data preprocessing process, 8 unlabeled images were removed, and the dataset was finally divided into three subsets: 4,493 images in the training set, 250 images in the validation set, and 249 images in the test set.

[0226] The Pascal Sentence dataset contains 1,000 images from the 2008 PASCAL development kit. Each image is associated with five separate sentences that were crowdsourced through Amazon Mechanical Turk. The dataset focuses on 20 categories and is divided into three subsets: 800 images for training, 100 images for validation, and 100 images for testing. This division facilitates the development and evaluation of models using the dataset.

[0227] The Wikipedia dataset is a widely used benchmark dataset in image-text retrieval research. The dataset contains articles selected from Wikipedia and their corresponding images. The dataset covers 29 categories, focusing on the 10 semantic categories that contain the most articles. Therefore, a dataset containing 2866 image-text pairs is constructed.

[0228] Researchers often use this dataset to evaluate the performance of various cross-media retrieval algorithms.

[0229] (2) Evaluation indicators

[0230] Four standard evaluation metrics are used to evaluate the experimental results in image-text retrieval (ITR): Average Mean Precision (MAP), Precision-Recall (PR) curve, Recall@K and Kendall rank correlation coefficient.

[0231] Among them, MAP is used to calculate the average of all average precision (AP) scores for each query. Its calculation formula is as follows:

[0232]

[0233] Where N represents the total number of samples, R represents the number of true items in the retrieval results, and Rn represents the number of true items in the first n retrieval results. In addition, if the nth returned item is a true item, then φ(n) is equal to 1; otherwise, it is 0. A higher MAP score indicates better performance in ITR.

[0234] The PR curve shows the change in retrieval precision under different recall values. The larger the area enclosed by the curve, the better the performance.

[0235] Recall@K represents the proportion of related terms identified in the first K positions. The higher the value of Recall@K, the better the retrieval performance. Its calculation formula is:

[0236]

[0237] The test set contains N instances, where r_K is defined as 1 if the true result appears in the top K returned results, otherwise 0. We provide the recall of the first ranked result (R@1), the top 5 ranked results (R@5), and the top 10 ranked results (R@10).

[0238] Kendall's rank correlation coefficient is defined in the formula Kendall's τ.

[0239] (3) Specific experimental details

[0240] Several important implementation details are outlined below, including data preprocessing methods, network architecture, and experimental setup.

[0241] 1) Data preprocessing method

[0242] For each image, we extract instance-level and segment-level visual features for the entire image and its segments, respectively. To balance computational resources and data capacity, we evenly divide each image into 3×3 segments, resulting in a total of n=9 segments. Specifically, we use AlexNet pre-trained on the ImageNet dataset to generate CNN features. The feature dimension obtained by the FC7 layer of AlexNet is 4096, denoted as d1=4096.

[0243] Network structure. The CNN features are transformed into a d-dimensional embedding space through a fully connected layer of a sub-network, where d = 1024. Similarly, the dimension of the word embedding of a given text is 300, denoted as d2 = 300. In order to establish the connection between the visual and language domains, we use a bidirectional GRU, and its hidden state dimension is also set to 1024.

[0244] 2) Experimental setup: Key hyperparameters, including β, δ, φ1, φ2, θ1, θ2, are tuned using a grid search strategy on the validation set with a range of predefined values. Following recent work, we evaluate the performance of each model on the validation set after each epoch and determine the best model by comparing the MAP score of the ITR task. Subsequently, the selected best model is evaluated on the test set to generate experimental results. Other hyperparameters are set based on empirical values, e.g., γ = 0.1, B = 64, α1 = 0.4, and α2 = -0.6. In addition, the temperature inversion parameter λ = 9.

[0245] (4) Comparison method

[0246] This experiment comprehensively compares NCSE with 16 state-of-the-art methods. Among them, seven are traditional non-deep neural network (DNN) cross-modal retrieval methods, including CCA, CMCP, HSNN, JGRHML, JRL, JFSSL, and S 2 UPG. The remaining nine are DNN-based methods, namely DCCA, CCL, SCAN, GXN, VSESC, MAVA, SGRAF, NAAF, and VSRN++. Importantly, all the compared methods are implemented using the source code provided by their respective authors.

[0247] The following is a brief introduction to the 16 comparison methods.

[0248] CCA is a well-established dimensionality reduction technique that aims to jointly optimize across two or more spaces. It learns heterogeneous representations of the same data and exploits the correlations between multiple spaces to capture shared information.

[0249] CMCP resolves the meaningful and irrelevant correlations between objects in different modalities. It effectively propagates correlations between heterogeneous data in different combinations and promotes the comprehensive integration of cross-modal information.

[0250] HSNN exploits the concept of heterogeneous similarity to estimate the likelihood that items from various modalities belong to a shared semantic concept. This estimation is achieved by identifying the nearest neighbors of each modality item that are associated with the same semantic concept.

[0251] JGRHML represents a heterogeneous metric learning method that combines a joint graph regularization framework to exploit the structural features of multiple modalities.

[0252] JRL combines sparse and semi-supervised regularization techniques in a unified optimization model and conducts a careful study of multimodal data, examining the impact of labeled and unlabeled data in different modalities.

[0253] JFSSL utilizes modality-specific projection matrix learning and employs multiple modality graph regularization terms to project the data into a shared common space.

[0254] S 2 UPG is a semi-supervised cross-modal feature learning technique that uses a unified graph to capture semantic correlations between multimodal data. It also integrates segment-level information to highlight important components, thereby improving the accuracy of cross-modal correlations.

[0255] DCCA is a method for discovering complex nonlinear mappings between two views of data, resulting in a highly interconnected linear representation.

[0256] CCL uses a hierarchical network architecture to exploit multiple layers of meaning through joint optimization. It introduces a multi-task learning strategy to preserve the correlations between different modalities and within each modality.

[0257] SCAN emphasizes the importance of image-text alignment in cross-modal retrieval. It enables words and image regions to be projected into a common embedding space, revealing meaningful alignments between image regions and words within sentences.

[0258] GXN integrates two generative models in a feature embedding framework to solve the cross-modal retrieval problem. It simultaneously generates high-level abstract representations and local base representations of images and text.

[0259] VSESC builds independent visual-based and text-based embedding spaces and introduces semantic consistency constraints to jointly optimize them. By integrating the semantic similarity of image-text pairs, VSESC effectively solves the cross-modal retrieval task.

[0260] MAVA proposes a dual attention framework to distinguish segment-level data by considering the saliency of data at both relation and segment levels.

[0261] SGRAF utilizes vector-based similarity representation to facilitate matching of images and texts. It also combines graph convolutional neural networks to estimate relation-aware similarity by considering the interactions between elements in the data. In addition, SGRAF integrates instance-level and segment-level similarities through an attention mechanism.

[0262] NAAF introduces a negative-aware attention framework that explicitly considers the positive influence of aligned segments and the negative influence of misaligned segments to infer image-text similarity.

[0263] VSRN++ proposes a framework for image-text embedding learning. It combines region relationship reasoning and instance semantic reasoning to derive text feature vectors for images and text.

[0264] Table 1: MAP scores

[0265]

[0266] Table 2: Corel 5K's Recall@K score

[0267]

[0268] Table 3: Recall@K score of Pascal Sentence

[0269]

[0270] Table 4: Recall@K scores of Wiki

[0271]

[0272] (5) Experimental results

[0273] The MAP scores of NCSE and other comparison methods on three datasets are listed in Table 1. The results show that NCSE has the best overall performance in terms of MAP score. Specifically, compared with the previous best model VSRN++, NCSE's average MAP scores on Corel 5K, Pascal Sentence, and Wiki datasets are improved by 0.35%, 5.39%, and 3.57%, respectively.

[0274] To further evaluate the image-text retrieval (ITR) performance, we adopt the Recall@K metric. In addition, we propose a new evaluation metric R@sum, which is obtained by summing R@1, R@5, and R@10 in the (I→T) and (T→I) directions. This metric provides a more comprehensive and informative evaluation of the Recall@K performance. As shown in Tables 2, 3, and 4, we observe that NCSE does not perform optimally in some metrics on some datasets. For example, in the Corel5K dataset, NCSE performs poorly in R@10 of I→T and R@1 and R@5 of T→I, while in the Pascal Sentence dataset, NCSE also fails to surpass other methods in R@1 and R@5 of T→I. However, in terms of overall performance (R@sum), NCSE still achieves the best results. It is worth noting that NCSE significantly improves the performance of the previous best models on these datasets.

[0275] Next, in Figure 5 , Figure 6 and Figure 7In this paper, we show the PR curves for I→T and T→I ITR tasks for all datasets. The results clearly show that the NCSE method outperforms other methods in terms of overall performance, and the PR curve covers a larger area. It should be emphasized that the data text modality of Pascal Sentences and Wiki is a sentence set, while Corel 5K is a label set. The experimental results show that NCSE performs better in ITR tasks using sentences or labels. In addition, we observe that deep neural network (DNN) based methods generally perform better than non-DNN methods.

[0276] Table 5: Training time comparison with recent state-of-the-art methods

[0277]

[0278] In particular, “s” in the table represents seconds. It should be noted that the source codes of all methods are run on the same machine equipped with only one GPU.

[0279] To provide a more comprehensive evaluation of our proposed method, we show the training time comparison of different DNN methods in Table 5. All methods are tested on the same machine equipped with only one GPU. Our observations are summarized as follows:

[0280] DCCA, CCL, and SCAN generally take less time to train than other DNN methods. However, they do not perform as well as other methods on ITR. Although methods such as GXN, VSESC, MAVA, SGRAF, and NAAF have similar training times to NCSE, NCSE significantly surpasses them on the ITR task. It is worth noting that although VSRN++ performs second only to NCSE on ITR, its training time is significantly longer than other methods.

[0281] Through detailed analysis of the above experimental results, we draw the following observations:

[0282] When comparing seven traditional methods that do not rely on deep neural networks (DNNs), it was found that S 2 UPG results are generally better. This is because S 2 UPG exploits segment-level details in image-text similarity learning, which other methods fail to fully exploit using only instance-level features.

[0283] Deep neural networks perform well in cross-modal similarity learning and can capture nonlinear correlations. For example, DCCA has a significant performance improvement over CCA because DCCA maximizes the correlation between the output layers of two different sub-networks and mines and exploits nonlinear cross-modal correlations by leveraging instance-level data. In addition, CCL outperforms DCCA, in part because it integrates multi-granular features and adopts a hierarchical network structure, which can discover more comprehensive and detailed cross-modal correlations.

[0284] Attention-based models show significant performance improvements over existing methods such as DCCA and CCL. Attention models effectively estimate cross-modal similarity by establishing latent connections between image segments and vocabulary. Specifically, SCAN and VSESC use attention mechanisms to align visual regions and vocabulary when computing cross-modal similarity. Notably, VSESC outperforms SCAN by introducing a semantic consistency constraint in the optimization objective. This constraint ensures semantic consistency between aligned image regions and vocabulary, making cross-modal similarity estimation more accurate.

[0285] In addition, MAVA estimates cross-modal similarity at three levels: instance, relation, and region, thus achieving better performance than SCAN and VSESC. In addition, SGRAF suppresses irrelevant interactions at both instance and segment levels, further improving performance over MAVA. It is worth mentioning that NAAF surpasses all the above methods by simultaneously considering the positive effects of matching segments and the negative effects of mismatching segments.

[0286] As a popular GAN-based cross-modal method, GXN achieves satisfactory performance by generating high-quality feature embeddings through its two generative modules. These features are particularly effective in accurately establishing matching image-text pairs.

[0287] Furthermore, VSRN++ acquires a feature set enhanced with semantically relevant knowledge by building links between image segments, followed by a holistic semantic analysis to identify the most important data for cross-modal similarity learning.

[0288] NCSE achieves the highest overall performance on all datasets due to its ability to fully exploit and integrate three levels of complementary semantics. This enables NCSE to cross the "heterogeneous gap" in ITR. Specifically, NCSE is able to accurately capture complex nonlinear cross-modal associations, which gives it a significant advantage over non-deep neural network methods. In addition, NCSE outperforms methods that only use instance-level or fragment-level data, such as SCAN, VSESC, GXN, and NAAF. Although MAVA and SGRAF fully exploit instance-level and fragment-level data, NCSE still surpasses them due to two key factors:

[0289] 1. NCSE exploits the complementarity between visual and textual neighbors to discover many potential image-text associations, thereby computing image-text similarity more comprehensively.

[0290] 2. NCSE simultaneously optimizes the image-text similarity generated from S-branch, F-branch and N-branch through a hybrid loss function. The hybrid loss function effectively improves the learning of image-text similarity between different branches by separating relevant and irrelevant samples in a bipolar manner and then re-ranking all irrelevant samples in a coherent manner.

[0291] (6) Ablation experiment

[0292] In order to explore the impact of various important components in NCSE, ablation experiments under various settings were performed in this experiment.

[0293] As shown in Table 6, we considered different network configurations (S-branch, F-branch, and N-branch) and various loss functions (consistency ranking loss, triple ranking loss, and ladder loss), and constructed multiple ablation models. Specifically, S, F, and N correspond to S-branch, F-branch, and N-branch, respectively. In addition, Tri and Lad represent triple loss and ladder loss, respectively. It is worth noting that there are nine possible combinations of these six components. For example, NCSE-S-Tri represents a model that only uses S-branch and adopts triple loss function. The experimental results MAP scores are presented in Table 7 respectively. Some noteworthy observations are listed below.

[0294] In the ablation study of this example, we explored the effects of different loss function combinations on image-text retrieval (ITR). The results show that NCSE-SN and NCSE-FN outperform NCSE-S-Tri, NCSE-S-Lad, NCSE-F-Tri, and NCSE-F-Lad on all datasets. This finding suggests that the consistency ranking loss effectively enhances the complementary semantics in the S-branch and the F-branch.

[0295] Experimental results show that NCSE outperforms NCSE-SN and NCSE-FN, highlighting the significant improvement in ITR achieved by combining S-branch and F-branch. In particular, NCSE-FN outperforms NCSE-SN, which is mainly attributed to the ability of F-branch to capture fragment-level details and useful semantic information by leveraging attention mechanism. These findings suggest that the combination of S-branch and F-branch enables different types of data granularity to complement each other, thus significantly enhancing the effect of ITR.

[0296] When all components are integrated, NCSE consistently achieves the best results on all datasets. Based on our experimental results, we can draw the following conclusions:

[0297] (1) Each component of NCSE actively enhances the semantics in image-text similarity learning.

[0298] (2) The complementary semantics at the neighbor level effectively supplement the semantics at the instance level and fragment level.

[0299] (3) The hybrid loss function outperforms other loss functions mainly because it can efficiently separate relevant and irrelevant samples and accurately rank irrelevant samples.

[0300] (4) NCSE successfully bridges the heterogeneous gap and achieves the highest performance by effectively mining and integrating three different levels of complementarity.

[0301] Table 6: Experimental configurations of different models in ablation studies

[0302]

[0303] Table 7: Experimental results of ablation studies on various datasets (specifically showing the MAP scores under each model configuration)

[0304]

[0305] In one or more implementations, corresponding to the method in the above embodiment, this embodiment provides a picture-text mutual retrieval system based on neighbor perception and complementary semantic enhancement, including:

[0306] A data acquisition unit, which is used to acquire information to be retrieved and a search target database, wherein the information to be retrieved is text and the search target data is an image, or the information to be retrieved is an image and the search target data is text;

[0307] A feature extraction unit, which is used to perform instance-level feature extraction and segment-level feature extraction on the information to be retrieved and the retrieval target in the retrieval target database respectively;

[0308] A retrieval result acquisition unit, which is used to obtain the similarity between the information to be retrieved and the retrieval target in the retrieval target database based on the instance-level features and segment-level features of the information to be retrieved and the retrieval target in the retrieval target database through a pre-built neighbor-aware complementary semantic enhancement network, and obtain an image or text retrieval result based on the similarity;

[0309] Among them, the neighbor-aware complementary semantic enhancement network specifically performs the following processing procedures: calculating the instance-level image-text similarity matrix based on the instance-level features of the information to be retrieved and the retrieval target; calculating the fragment-level image-text similarity matrix based on the fragment-level features of the retrieval target in the information to be retrieved and the retrieval target database; wherein, in the process of calculating the instance-level image-text similarity matrix and the fragment-level image-text similarity matrix, the neighbor relationship of the information to be retrieved and the retrieval target in the preset data set is obtained respectively, and the neighbor-level image-text similarity matrix is ​​obtained based on the neighbor relationship, and the obtained instance-level image-text similarity matrix and fragment-level image-text similarity matrix are optimized with the neighbor-level image-text similarity matrix as supplementary semantics, and the image-text similarity result enhanced with complementary semantics is obtained based on the weighted fusion of the optimized instance-level image-text similarity matrix and the fragment-level image-text similarity matrix.

[0310] It should be noted here that each module in this embodiment corresponds to each step in Example 1 one by one, and the specific implementation process is the same, which will not be repeated here.

[0311] In further embodiments, there is also provided:

[0312] An electronic device includes a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method described in Embodiment 1 is performed. For the sake of brevity, it will not be described in detail here.

[0313] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0314] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0315] A computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the method described in embodiment 1 is completed.

[0316] The method in the first embodiment can be directly embodied as a hardware processor, or a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0317] Those skilled in the art will appreciate that the units, i.e., algorithm steps, of the various examples described in the present embodiment can be implemented in electronic hardware or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0318] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for image-text mutual retrieval based on neighbor perception and complementary semantic enhancement, characterized in that: include: Acquire information to be retrieved and a retrieval target database, wherein the information to be retrieved is text and the retrieval target data is an image, or the information to be retrieved is an image and the retrieval target data is text; Perform instance-level feature extraction and segment-level feature extraction on the information to be retrieved and the retrieval target in the retrieval target database respectively; Based on the instance-level features and segment-level features of the information to be retrieved and the retrieval target in the retrieval target database, the similarity between the information to be retrieved and the retrieval target in the retrieval target database is obtained through a pre-built neighbor-aware complementary semantic enhancement network, and the image or text retrieval result is obtained based on the similarity; Among them, the neighbor-aware complementary semantic enhancement network specifically performs the following processing procedures: calculating the instance-level image-text similarity matrix based on the instance-level features of the information to be retrieved and the retrieval target; calculating the fragment-level image-text similarity matrix based on the fragment-level features of the retrieval target in the information to be retrieved and the retrieval target database; wherein, in the process of calculating the instance-level image-text similarity matrix and the fragment-level image-text similarity matrix, the neighbor relationship of the information to be retrieved and the retrieval target in the preset data set is obtained respectively, and the neighbor-level image-text similarity matrix is ​​obtained based on the neighbor relationship, and the obtained instance-level image-text similarity matrix and fragment-level image-text similarity matrix are optimized with the neighbor-level image-text similarity matrix as supplementary semantics, and the image-text similarity result enhanced with complementary semantics is obtained based on the weighted fusion of the optimized instance-level image-text similarity matrix and the fragment-level image-text similarity matrix.

2. The method for image-text mutual retrieval based on neighbor perception and complementary semantic enhancement as claimed in claim 1, characterized in that: The instance-level image-text similarity matrix is ​​calculated by the following formula: in, is the instance-level feature vector extracted from the image, is the instance-level feature vector extracted from the text, and simS(vi,rj) is the instance-level image-text similarity between image vi and text rj.

3. The method for image-text mutual retrieval based on neighbor perception and complementary semantic enhancement as claimed in claim 1, characterized in that: The segment-level image-text similarity matrix is ​​calculated by the following formula: in, Retrieve global text similarity for images, Retrieve the global similarity of images for text, X is the number of segments in the image, Y is the number of words in the text, x is the xth segment, y is the yth segment, vi is the image, rj is the text, for and The correlation score between for and The relevance score of is the x-th image segment feature, represents the xth text context feature, represents the feature of the yth text word, Represents the y-th image context feature.

4. The method for image-text mutual retrieval based on neighbor perception and complementary semantic enhancement as claimed in claim 1, characterized in that: The neighbor-level image-text similarity matrix is ​​used as a supplementary semantic to optimize the obtained instance-level image-text similarity matrix and the fragment-level image-text similarity matrix, wherein a hybrid loss function based on triple ranking loss and consistency ranking loss is adopted in the optimization process.

5. The method for image-text mutual retrieval based on neighbor perception and complementary semantic enhancement as claimed in claim 1, characterized in that: The instance-level feature extraction is specifically as follows: for image data, the original image is used as the input of a pre-trained deep learning model to obtain the instance-level features of the image; for text data, each word in the original text data is one-hot encoded, and based on the encoding result, the feature vector representation of each word is obtained through a bidirectional gated recurrent unit, and based on the mean of the feature vector representations of all words in the text, the instance-level features of the text are obtained.

6. The method for image-text mutual retrieval based on neighbor perception and complementary semantic enhancement as claimed in claim 1, characterized in that: The segment-level feature extraction is specifically as follows: for image data, the original image is divided into several segments, and the features of each segment are extracted through a pre-trained deep learning model, and the set consisting of the features of all the segments is used as the segment-level features of the image; for text data, the features of each word in the original text data are obtained through a bidirectional gated recurrent unit, and the set consisting of the features of all the words is used as the segment-level features of the text.

7. A picture-text mutual retrieval system based on neighbor perception and complementary semantic enhancement, characterized in that: include: A data acquisition unit, which is used to acquire information to be retrieved and a search target database, wherein the information to be retrieved is text and the search target data is an image, or the information to be retrieved is an image and the search target data is text; A feature extraction unit, which is used to perform instance-level feature extraction and segment-level feature extraction on the information to be retrieved and the retrieval target in the retrieval target database respectively; A retrieval result acquisition unit, which is used to obtain the similarity between the information to be retrieved and the retrieval target in the retrieval target database based on the instance-level features and segment-level features of the information to be retrieved and the retrieval target in the retrieval target database through a pre-built neighbor-aware complementary semantic enhancement network, and obtain an image or text retrieval result based on the similarity; Among them, the neighbor-aware complementary semantic enhancement network specifically performs the following processing procedures: calculating the instance-level image-text similarity matrix based on the instance-level features of the information to be retrieved and the retrieval target; calculating the fragment-level image-text similarity matrix based on the fragment-level features of the retrieval target in the information to be retrieved and the retrieval target database; wherein, in the process of calculating the instance-level image-text similarity matrix and the fragment-level image-text similarity matrix, the neighbor relationship of the information to be retrieved and the retrieval target in the preset data set is obtained respectively, and the neighbor-level image-text similarity matrix is ​​obtained based on the neighbor relationship, and the obtained instance-level image-text similarity matrix and fragment-level image-text similarity matrix are optimized with the neighbor-level image-text similarity matrix as supplementary semantics, and the image-text similarity result enhanced with complementary semantics is obtained based on the weighted fusion of the optimized instance-level image-text similarity matrix and the fragment-level image-text similarity matrix.

8. An electronic device comprising a memory, a processor and a computer program stored and running on the memory, characterized in that: When the processor executes the program, the method for image-text mutual retrieval based on neighbor perception and complementary semantic enhancement as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a method for image-text mutual retrieval based on neighbor perception and complementary semantic enhancement as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for image-text mutual retrieval based on neighbor perception and complementary semantic enhancement as described in any one of claims 1 to 6 is implemented.

Images