Method for retrieving images and texts based on semantic filtering and adaptive adjustment

Abstract

The application relates to the technical field of image-text retrieval, in particular to an image-text retrieval method based on semantic filtering and adaptive adjustment, which comprises the following steps: features of input images and input sentences are respectively extracted based on a feature representation module to obtain feature representations of each image region and each word; image-text matching is performed in a text-to-image direction and an image-to-text direction based on a filtering attention module, irrelevant image region and word feature representations are filtered at the same time, and global image region-word relevance representations are obtained; and the image region-word alignment is guided in a cyclic manner based on an adaptive adjustment module, alignment weights between the image region and the word are gradually optimized and updated, and a global similarity score of image text is obtained. The application can reduce the interference of irrelevant samples on relevance learning, and can highlight key information in the focused data.

Description

Technical Field

[0001] This invention relates to the field of image and text retrieval technology, and more specifically to an image and text retrieval method based on semantic filtering and adaptive adjustment. Background Technology

[0002] Image-text matching refers to measuring the semantic similarity between images and text, which is becoming increasingly important for various visual and language tasks. When humans evaluate whether an image and a sentence are similar in the brain, they measure it on the object (or region) that they are focusing on. For example, when associating an image with the sentence "A dog is running on the grass near a wooden fence," humans conceive the image using keywords such as "a dog," "wooden fence," "grass," and "running."

[0003] Currently, most image-text retrieval methods employ attention-based mechanisms, but these methods neglect semantic relationships within a single modality. Furthermore, they lack effective filtering of irrelevant information in image data, which can interfere with image-text matching. To address this, subsequent works have proposed semantic filtering, which filters out irrelevant region-word pairs and performs similarity inference, enabling better interaction between intra- and inter-modal data and thus improving matching accuracy. However, even after semantic filtering, some alignments still contain elements with weak relevance to the topic. While these elements play a role in the matching process, their importance is relatively low compared to other key elements. Therefore, such alignments waste some resources but cannot be completely discarded.

[0004] Therefore, how to further mine the deep semantics between data pairs and focus on the key semantics in the data in order to achieve more accurate cross-modal matching results has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides a text and image retrieval method based on semantic filtering and adaptive adjustment, which can reduce the interference of irrelevant samples on relevance learning and better highlight the key information in the data of interest.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A text-image retrieval method based on semantic filtering and adaptive adjustment includes the following steps:

[0008] An adaptive attention filtering model is constructed, which includes a feature representation module, a filtering attention module, and an adaptive adjustment module.

[0009] Based on the feature representation module, features of the input image and the input sentence are extracted respectively to obtain the image feature representation of each image region and the text feature representation of each word;

[0010] Based on the filtering attention module, image-text matching is performed from the text to image direction and the image to text direction. Irrelevant image region feature representations and word feature representations are filtered at the same time to obtain a global image region-word relevance representation.

[0011] The adaptive adjustment module guides image region-word alignment in a cyclical manner and gradually optimizes and updates the alignment weights between image regions and words to obtain a global similarity score between the image and the text.

[0012] Furthermore, based on the feature representation module, features of the input image are extracted to obtain image feature representations for different regions, including:

[0013] Image features of each region of the input image are extracted based on a pre-trained Faster R-CNN model.

[0014] Image features are mapped to d-dimensional vectors through fully connected layers to generate feature representations of local regions.

[0015] A self-attention mechanism is applied to local regions. This self-attention mechanism uses the average feature as the query and aggregates the feature representations of all image regions to generate a global image feature representation of the input image.

[0016] Furthermore, based on the feature representation module, features of the input text are extracted to obtain text feature representations for each word, including:

[0017] The input statement is split into multiple words and the words are embedded into the Bi-GRU model in an orderly manner.

[0018] The text feature representation of each word is obtained by averaging the forward and backward hidden states at each time step.

[0019] Furthermore, in the filtering attention module, for the text-to-image direction, each word is fixed as a shared semantic, each word is matched with a relevant image region, and irrelevant image regions are filtered out to obtain the relevant image region features of each word; for the image-to-text direction, each image region is fixed as a shared semantic, each image region is matched with a relevant word, and irrelevant words are filtered out to obtain the relevant word features of each image region.

[0020] Based on the filtered relevant image region features and the filtered relevant word features, a global image region-word correlation representation is obtained.

[0021] Furthermore, for the text-to-image direction, the step of the filtering attention module matching each word to a relevant image region includes:

[0022] For each word's text features, cosine similarity is calculated between it and each image region to obtain the similarity value of a word to different image regions.

[0023] The obtained similarity values ​​are normalized to the [0,1] interval by using the softmax activation function, thus completing the pre-assignment of attention scores;

[0024] Compare the relative importance between two image regions. If the pre-assigned attention score of the image region being compared is greater than that of the image region being compared with, then the score of the image region being compared is greater than 0, and it is considered a relevant image region. Otherwise, it is considered an irrelevant image region.

[0025] Based on the calculation of function H, the scores of image regions with scores less than 0 are set to 0, and the scores of image regions with scores greater than 0 are set to 1, thus obtaining the redistributed attention matrix;

[0026] The image features of each image region are multiplied by the new attention matrix, and irrelevant image regions are filtered out to obtain the relevant image regions for each word.

[0027] Furthermore, for the text-to-image direction, the formula for pre-assigning attention scores is:

[0028]

[0029] Where σ represents the softmax activation function; α represents a scaling factor that further increases the difference between relevant and irrelevant image regions; u i The text features of the i-th word are represented by T; T represents transpose. v represents the transpose of the text feature vector of the i-th word; j The j-th image region represents the image features; m represents the number of words in the input statement; n represents the number of image regions in the input image.

[0030] The formula for comparing the relative importance of two image regions is:

[0031]

[0032] Where, f(v) ij ,v it ) represents the relative attention of the j-th image region to the t-th image region for the i-th word; g(v it ) represents the confidence level of the compared image regions, and the confidence score of the t-th region being compared is set as its relevance to the i-th word;

[0033] The reassigned attention matrix w' ij The expression is:

[0034]

[0035] Wherein, H(v) ij The expression indicates whether the j-th image region selected based on the i-th word is relevant; if relevant, it is 1, otherwise 0.

[0036] For the i-th word, the matched relevant image region v' i for:

[0037]

[0038] Global image-text relevance is represented as:

[0039]

[0040] Among them, R(u i ,v' i ) represents the correlation function, which represents the correlation score between the two modes u and v.

[0041] Furthermore, for the image-to-text direction, the step of the filtering attention module matching relevant words for each image region includes:

[0042] For the image features of each image region, cosine similarity is calculated between each word to obtain the similarity value of a certain image region for different words;

[0043] The obtained similarity values ​​are normalized to the [0,1] interval by using the softmax activation function, thus completing the pre-assignment of attention scores;

[0044] Compare the relative importance between two words. If the pre-assigned attention score of the word being compared is greater than that of the word being compared, then the score of the word being compared is greater than 0, and it is considered a relevant word; otherwise, it is considered an irrelevant word.

[0045] Based on the calculation of function H, the scores of words with scores less than 0 are set to 0, and the scores of words with scores greater than 0 are set to 1, thus obtaining the redistributed attention matrix;

[0046] The text features of each word are multiplied by the new attention matrix, and irrelevant words are filtered out to obtain the relevant word features for each image region.

[0047] Furthermore, for the image-to-text direction, the formula for pre-assigning attention scores is:

[0048]

[0049] The formula for comparing the relative importance of two words is:

[0050]

[0051] Where, f(u) ji ,u jt ) represents the relative attention of the i-th word to the t-th word for the j-th image region; g(u jt () indicates the confidence level of the compared word;

[0052] The reassigned attention matrix w' ji The expression is:

[0053]

[0054] Wherein, H(u) ji The value indicates whether the i-th word matched for the j-th image region is relevant. A value of 1 indicates that the word is relevant, while a value of 0 indicates that the word is irrelevant.

[0055] For the j-th image region, the matched word u' j for:

[0056]

[0057] Global image-text relevance is represented as:

[0058]

[0059] Wherein, R(v) j ,u' j The similarity is indicated by the number of regions in an image and the corresponding words in a sentence.

[0060] Furthermore, the adaptive adjustment module's step of aligning image regions with words includes:

[0061] The filtered image region-word features are then used to calculate the alignment vector using the following formula.

[0062]

[0063] Among them, W a This represents the parameter matrix for the linear transformation of the guiding vector; u j Represents the image features of the j-th image region; The vector represents the text feature vector of the j-th word after aggregation optimization; |||| represents the L2 norm.

[0064] For each alignment vector Perform average pooling and set the aggregate weight to 1 / L, then initialize the guidance alignment vector.

[0065]

[0066] Using the initialized guidance alignment vector Guide the next step of aggregate weight update:

[0067]

[0068] in, tanh represents the text feature vector of the j-th word obtained after the first aggregate weight update; tanh represents the activation function. and Each represents a learnable parameter matrix;

[0069] Using the updated parameters Obtain the j-th aligned weight matrix

[0070]

[0071] weight matrix Alignment vector Multiplication guides the alignment vector update to

[0072]

[0073] The nth guidance alignment vector update is:

[0074]

[0075] The parameter matrix W s Alignment vector with the latest guidelines Multiply the results and normalize them using the sigmoid activation function to obtain the global image region-word similarity score in the interval [0, 1].

[0076]

[0077] in, It is a learnable parameter matrix.

[0078] Furthermore, the overall objective function of the adaptive attention filtering model is:

[0079]

[0080] Where S() represents the similarity function used to calculate the similarity score between two modalities; v, t represent positive examples of image region-word pairs, and v', t' represent difficult examples of words and difficult examples of image regions; α represents the marginal parameter controlling the difference between positive and negative pairs, and the similarity of positive pairs should be higher than that of negative pairs; where the function [·] + Equivalent to [·,0].

[0081] As can be seen from the above technical solution, compared with the prior art, the present invention has the following beneficial effects:

[0082] This invention aggregates all alignments involving different saliency scores through an attention filtering module. Furthermore, instead of filtering only irrelevant words, it filters irrelevant region-word pairs simultaneously, reducing interference from completely meaningless alignments and minimizing the interference of irrelevant sample pairs on relevance learning. An adaptive adjustment module redistributes attention scores, increasingly emphasizing alignments from more important words and gradually reducing the weight of less important words. This reallocation of alignment weights highlights key information in the data, focusing attention on more critical regions and words, and achieving more accurate cross-modal matching results. Attached Figure Description

[0083] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0084] Figure 1 The flowchart shows the image and text retrieval method based on semantic filtering and adaptive adjustment provided by this invention.

[0085] Figure 2 The detailed flowchart shows how the adaptive attention filtering model provided by this invention processes input images and sentences. Detailed Implementation

[0086] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0087] like Figure 1As shown in the figure, this invention discloses a text and image retrieval method based on semantic filtering and adaptive adjustment, including the following steps:

[0088] An adaptive attention filtering model is constructed, which includes a feature representation module, a filtering attention module, and an adaptive adjustment module.

[0089] Based on the feature representation module, features of the input image and the input sentence are extracted respectively to obtain the image feature representation of each image region and the text feature representation of each word;

[0090] Based on the filtering attention module, image-text matching is performed from the text to image direction and the image to text direction. Irrelevant image region feature representations and word feature representations are filtered at the same time to obtain a global image region-word relevance representation.

[0091] An adaptive adjustment module guides image region-word alignment in a cyclical manner and progressively optimizes and updates the alignment weights between image regions and words to obtain a global similarity score between the image and the text.

[0092] The steps described above will be further explained below with a specific embodiment.

[0093] Unlike traditional unimodal image or text retrieval, image-text retrieval is a bidirectional mutual retrieval task. Retrieving the most relevant text given a query image is called image-to-text retrieval (I2T), and finding the semantically most similar image given a query text is called text-to-image retrieval (T2I). Although the same thing can be described by visual images and verbal text, there is a huge semantic gap between visual and textual representations. Therefore, the core challenge of image-text retrieval is to bridge the semantic gap by learning the similarity between image and text representations and maximizing the similarity of related image-text pairs.

[0094] This invention relates to a text and image retrieval method based on semantic filtering and adaptive adjustment, comprising the following steps:

[0095] like Figure 2 As shown, the adaptive attention filtering model constructed in this invention consists of three modules: a feature representation module, a filtering attention module, and an adaptive adjustment module.

[0096] The specific functions and data processing procedures of the three modules are explained in detail below:

[0097] 1. Feature Representation Module

[0098] For each input image, image features of each image region are extracted based on the pre-trained Faster R-CNN model.

[0099] Image features are mapped to d-dimensional vectors through fully connected layers, generating feature representations of local regions V = {v1, K, v}. k},in,

[0100] A self-attention mechanism is applied to a local region, and this self-attention mechanism uses average features. As a query, and by aggregating the feature representations of all image regions, a global image feature representation of the input image is generated.

[0101] For the input statement, it is first split into multiple words and then embedded into the Bi-GRU model in an orderly manner. Then, by averaging the forward and backward hidden states at each time step, the text feature representation T = {t1, K, t...} is obtained for each word. L}.in, Let j represent the text feature representation of the j-th word. The global text feature representation is obtained by calculating the features of all words through a self-attention mechanism.

[0102] 2. Attention Filtering Module

[0103] While utilizing local alignments can improve matching performance by discovering finer-grained correspondences between image regions and sentence fragments, less meaningful alignments can hinder discriminative power when all possible alignments are aggregated in an indiscriminate manner. Therefore, this invention proposes a Filtered Attention (AF) module to eliminate irrelevant alignments, such as those with "the", "be", etc.

[0104] Conventional attention mechanisms often focus on regions and words that cannot be aligned, which interferes with the network's alignment learning. To address this, the present invention proposes a filtering attention mechanism that learns a scoring function F to identify segments related to shared semantics. This function removes irrelevant segments from the shared semantics. Here, segments with scores greater than 0 are defined as relevant segments, i.e.:

[0105] H(x) = I(F(x) > 0)

[0106] Where I(g) is the indicator function, and x represents an image region or word.

[0107] Traditional attention mechanisms mitigate the impact of irrelevant region-word alignment during image-text matching by assigning different levels of importance to different regions and words. However, these methods ignore the noise effect of irrelevant region-word pairs on relevance learning. For example, given an image region “guitar,” a traditional attention matrix might notice irrelevant words such as “people,” “face,” and “the.” The learning of correspondences between the image region “guitar” and these irrelevant words could suppress the recognition of relevant region-word pairs.

[0108] Finding an absolute boundary between relevant and irrelevant segments is impractical; some early attention-based methods...

[40] Simply locating local fragments to focus on related sentence segments does not fully understand the semantics and therefore cannot effectively align them. Inspired by the non-local block concept proposed in [the original text], this invention determines the relevance of segments by calculating the relative importance of segments to other segments.

[0109] The principle behind this mechanism is that irrelevant segments are always less important for shared semantics compared to other relevant segments. The scoring function is defined as:

[0110]

[0111] Paired functions f(x) i ,x j Calculate the relative importance of the i-th segment to the j-th segment, g(x j Let represent the confidence level of the compared segment. Then, a weighted comparison result is added to all other segments. If a segment is similar to other relevant segments and has a high confidence level, then that segment can be considered relevant. Therefore, the k-th shared semantic can be simply defined as:

[0112]

[0113] Unlike traditional attention methods that focus on learning attention weights, the purpose of the filtered attention proposed in this invention is to eliminate irrelevant fragments in the context. Furthermore, unlike hard attention which uses random sampling to estimate gradients, filtered attention can directly compute gradients because filtering out irrelevant fragments facilitates the network's forward propagation. This makes network training both efficient and effective.

[0114] In this invention, in the filtering attention module, for the text-to-image direction, each word is fixed as a shared semantic, each word is matched to a relevant image region, and irrelevant image regions are filtered out to obtain the relevant image region features of each word; for the image-to-text direction, each image region is fixed as a shared semantic, each image region is matched to a relevant word, and irrelevant words are filtered out to obtain the relevant word features of each image region.

[0115] Based on the filtered relevant image region features and the filtered relevant word features, a global image region-word correlation representation is obtained.

[0116] Specifically, shared semantics in images and texts are discovered by fixing one modality and looking for related fragments in another modality, where fragments in the fixed modality are considered shared semantics.

[0117] (1) For the text-to-image direction, text words are fixed with shared semantics, and each word needs to be matched to a relevant image region. This generally involves three steps: pre-allocating attention, identifying relevant regions, and reallocating attention. Specific steps include:

[0118] For each word's text features, cosine similarity is calculated between it and each image region to obtain the similarity value of a word across different image regions.

[0119] The obtained similarity values ​​are normalized to the [0,1] interval using the softmax activation function, thus completing the pre-assignment of attention scores.

[0120]

[0121] Where σ represents the softmax activation function; α represents a scaling factor that further increases the difference between relevant and irrelevant image regions, which can be set to 20; u i The text features of the i-th word are represented by T; T represents transpose. v represents the transpose of the text feature vector of the i-th word; j The j-th image region represents the image features; m represents the number of words in the input statement; n represents the number of image regions in the input image.

[0122] To compare the relative importance of two image regions, if the pre-assigned attention score of the compared image region is greater than that of the image region being compared, then the score of the compared image region is greater than 0, indicating it is a relevant image region; otherwise, it is an irrelevant image region. The formula for comparing the relative importance of two image regions is as follows:

[0123]

[0124] Where, f(v) ij ,v it ) represents the relative attention of the j-th image region to the t-th image region for the i-th word; g(v it ) represents the confidence level of the compared image regions, and the confidence score of the t-th region being compared is set as its relevance to the i-th word.

[0125] Based on the calculation of function H, the scores of image regions with scores less than 0 are set to 0, and the scores of image regions with scores greater than 0 are set to 1, resulting in the redistributed attention matrix w. i ' j ;

[0126]

[0127] Where H() is the discriminant function, H(v ij ) This indicates whether the j-th image region selected based on the i-th word is relevant; if relevant, it is 1, otherwise 0.

[0128] The image features of each image region are multiplied by the new attention matrix, and irrelevant image regions are filtered out to obtain the relevant image regions for each word. Unlike traditional attention, the reassigned attention weights set the weights of irrelevant regions to 0, focusing attention only on all relevant regions.

[0129] For the i-th word, the matched relevant image region v' i for:

[0130]

[0131] Global image-text relevance is represented as:

[0132]

[0133] Among them, R(u i v i ') represents the correlation function, which represents the correlation score between the two modes u and v.

[0134] (2) For the image-to-text direction, the step of the filtering attention module matching relevant words for each image region is similar to the above-described text-to-image direction processing, specifically including:

[0135] For each image region's image features, cosine similarity is calculated between it and each word to obtain the similarity value of a given image region for different words.

[0136] The obtained similarity values ​​are normalized to the [0,1] interval using the softmax activation function, thus completing the pre-assignment of attention scores. The formula for pre-assigning attention scores is as follows:

[0137]

[0138] In this process, relevant words can receive more attention, but irrelevant words can also help with semantic sharing between image regions and target text.

[0139] Then, each word is scored based on the pre-allocated attention relative to other words. That is, the relative importance between two words is compared. If the pre-allocated attention score of the compared word is greater than that of the word it is compared with, then the score of the compared word is greater than 0 and it is a relevant word; otherwise, it is an irrelevant word.

[0140] Where, f(u) ji ,u jt g represents the relative attention of the i-th word to the t-th word for the j-th image region; g(u jt () indicates the confidence level of the compared word.

[0141] Based on the calculation of function H, the scores of words with scores less than 0 are set to 0, and the scores of words with scores greater than 0 are set to 1, resulting in a redistributed attention matrix; the redistributed attention matrix w' ji The expression is:

[0142]

[0143] Wherein, H(u) ji The expression indicates whether the i-th word matched for the j-th image region is relevant. A value of 1 indicates that the word is relevant, while a value of 0 indicates that the word is irrelevant.

[0144] The text features of each word are multiplied by the new attention matrix, and irrelevant words are filtered out to obtain the relevant word features for each image region.

[0145] The reassigned attention will be used to perform element-wise multiplication with the relevant lexical feature representations in d-dimensional space, select shared semantics with the j-th image region from the text, compute a weighted combination of relevant words, and for the j-th image region, the matched word u' j for:

[0146]

[0147] The local similarity score R(v) is calculated using cosine similarity. j ,u' j The global relevance score is calculated by averaging the local relevance scores. The global image-text relevance is represented as:

[0148]

[0149] Wherein, R(v) j ,u' j The ) represents the local similarity score, which is the similarity between a certain region in an image and the corresponding word in a sentence.

[0150] 3. Adaptive adjustment module

[0151] The Adaptive Adjustment (AR) module guides word-region alignment in a cyclical manner by progressively optimizing the attention weights calculated from the overall alignment. The AR module progressively optimizes the network's learning ability by iteratively updating the alignment weights between word-region pairs, guided by the parameter optimization of the adaptive adjuster. The AR module primarily operates on the aggregated weights of the global alignment and progressively optimizes them through a feedback mechanism. Its steps for image region-word alignment include:

[0152] The filtered image region-word features are then used to calculate the alignment vector using the following formula.

[0153]

[0154] Among them, W a This represents the parameter matrix for the linear transformation of the guiding vector; u j Represents the image features of the j-th image region; The vector represents the text feature vector of the j-th word after aggregation optimization; |||| represents the L2 norm.

[0155] For each alignment vector Perform average pooling and set the aggregate weight to 1 / L, then initialize the guidance alignment vector.

[0156]

[0157] Using the initialized guidance alignment vector Guide the next step of aggregate weight update:

[0158]

[0159] in, tanh represents the text feature vector of the j-th word obtained after the first aggregate weight update; tanh represents the activation function. and Each represents a learnable parameter matrix;

[0160] Using the updated parameters Obtain the j-th aligned weight matrix

[0161]

[0162] weight matrix Alignment vector Multiplication guides the alignment vector update to

[0163]

[0164] The nth guidance alignment vector update is:

[0165]

[0166] The parameter matrix W s Alignment vector with the latest guidelines Multiply the results and normalize them using the sigmoid activation function to obtain the global image region-word similarity score in the interval [0, 1].

[0167]

[0168] in, It is a learnable parameter matrix.

[0169] In other embodiments, the overall objective function of the adaptive attention filtering model constructed by the present invention is:

[0170]

[0171] Where S() represents the similarity function used to calculate the similarity score between two modalities; v, t represent positive examples of image region-word pairs, and v', t' represent difficult examples of words and difficult examples of image regions; α represents the marginal parameter controlling the difference between positive and negative pairs, and the similarity of positive pairs should be higher than that of negative pairs; where the function [·] + Equivalent to [·,0].

[0172] The present invention also undergoes the following experimental verification.

[0173] To compare the performance of the method of this invention with existing methods, various experiments were conducted on the publicly available datasets Flickr30K and MSCOCO, which are widely used for cross-modal retrieval model training and performance testing. Each image in these datasets is accompanied by five relevant descriptions. Specifically, Flickr30K contains a total of 31,000 images and 155,000 sentences, divided into a training set (29,000 images / 145,000 sentences), a validation set (1,000 images / 5,000 sentences), and a test set (1,000 images / 5,000 sentences). MSCOCO is a large dataset containing 113,287 images and 616,425 sentences, also divided into a training set (29,000 images / 145,000 sentences), a validation set (5,000 images / 25,000 sentences), and a test set (5,000 images / 250,000 sentences).

[0174] On the Flickr30K dataset, the network was trained with an initial learning rate of 0.0002 for 30 epochs, with the learning rate at 0.0002 for the first 15 epochs and decreasing by 0.1 every 15 epochs. On MSCOCO, the network was trained with a learning rate of 0.0005 and a mini-batch size of 32. The network required 15 epochs of training on Flickr30K and 20 epochs on MSCOCO, with training instances randomly transformed in each epoch. For each image, Faster R-CNN was used to detect 36 regions, extracting their 2048-dimensional features. These extracted features were then fed into a fully connected layer to generate a 1024-dimensional representation. The initial one-hot vectors of word embeddings were transformed into 300-dimensional vectors and then fed into a bidirectional GRU that produced the 1024-dimensional representation. The entire network was trained on PyTorch using the Adam optimizer and an NVIDIA 3070Ti GPU.

[0175] Recall@K is a commonly used evaluation metric, often used to quantitatively assess the performance of image-text retrieval. R@K represents the percentage of correct results retrieved from the first K matching results. Since users primarily focus on the first and first few search results in actual retrieval work, this invention selects two metrics, R@1 and R@5, to evaluate the performance of the proposed method. Furthermore, the rsum value is calculated by summing the recall values ​​to display the overall retrieval performance. The formula for calculating rsum is shown below:

[0176] rsum=R@K(i2t)+R@K(t2i),K=1,5

[0177] The comparison results are shown in Tables 1 and 2:

[0178] Table 1 shows the comparison results on the Flickr30K dataset.

[0179]

[0180] Table 2 shows the comparison results on the MSCOCO dataset.

[0181]

[0182] The proposed FARA method was experimentally compared with various state-of-the-art methods on the Flickr30K and MSCOCO datasets. Table 1 shows the comparison results of the proposed method with existing methods on Flickr30K. Clearly, the proposed method outperforms other methods. Compared to IMRAM, the proposed method improves Recall@1 by 0.4% and 0.6%, respectively. IMRAM uses an attention mechanism to filter salient regions as shared semantic exploration for image and text alignment relationships. This method results in the proposed method having a slightly lower Recall@5 than IMRAM, but overall, the proposed method outperforms all other methods in both rmean and rsum metrics. The proposed method utilizes filtered attention to eliminate irrelevant alignments and then redistributes attention to salient regions and words. On the COCO dataset, as shown in Table 2, compared with BFAN, which also performs bidirectional attention filtering, the proposed method shows a significant improvement in overall performance, specifically improving Recall@1 by 2.4% and 2.1% in text-to-image and image-to-text searches, respectively. This also demonstrates the effectiveness of the adaptive regulator proposed in improving the attention distribution of salient features, and also shows the superiority of the method in various datasets.

[0183] To further explore the effectiveness of the filtering attention module and the adaptive adjustment module, an ablation experiment was conducted on Flickr30K. Experiments were performed on models without filtering attention and without the adaptive adjuster. The experimental results are shown in Table 3. The results show that both filtering attention and the adaptive adjuster play a positive role in retrieval, improving performance to some extent in both image-to-text and text-to-image retrieval. Under the complete network structure, the experimental results are superior to those of the aforementioned structures, fully demonstrating that the method of this invention also learns finer-grained region-word correspondences.

[0184] Table 3 Ablation experiments on the Flickr30K dataset.

[0185]

[0186] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0187] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A text-image retrieval method based on semantic filtering and adaptive adjustment, characterized in that, Includes the following steps: An adaptive attention filtering model is constructed, which includes a feature representation module, a filtering attention module, and an adaptive adjustment module. Based on the feature representation module, features of the input image and the input sentence are extracted respectively to obtain the image feature representation of each image region and the text feature representation of each word; Based on the filtering attention module, image-text matching is performed from the text to image direction and the image to text direction. Irrelevant image region feature representations and word feature representations are filtered at the same time to obtain a global image region-word relevance representation. Based on the adaptive adjustment module, image region-word alignment is guided in a cyclical manner, and the alignment weights between image regions and words are gradually optimized and updated to obtain the global similarity score between the image and the text. In the filtering attention module, for the text-to-image direction, each word is fixed as a shared semantic, each word is matched with a relevant image region, and irrelevant image regions are filtered out to obtain the relevant image region features of each word; for the image-to-text direction, each image region is fixed as a shared semantic, each image region is matched with a relevant word, and irrelevant words are filtered out to obtain the relevant word features of each image region. Based on the filtered relevant image region features and the filtered relevant word features, a global image region-word relevance representation is obtained; For the text-to-image direction, the filtering attention module's steps for matching each word to a relevant image region include: For each word's text features, cosine similarity is calculated between it and each image region to obtain the similarity value of a word to different image regions. The obtained similarity values ​​are normalized to the [0,1] interval by using the softmax activation function, thus completing the pre-assignment of attention scores; Compare the relative importance between two image regions. If the pre-assigned attention score of the image region being compared is greater than that of the image region being compared with, then the score of the image region being compared is greater than 0, and it is considered a relevant image region. Otherwise, it is considered an irrelevant image region. Based on the calculation of function H, the scores of image regions with scores less than 0 are set to 0, and the scores of image regions with scores greater than 0 are set to 1, thus obtaining the redistributed attention matrix; The image features of each image region are multiplied by the new attention matrix to filter out irrelevant image regions, thus obtaining the relevant image regions for each word; For the image-to-text direction, the filtering attention module matches relevant words for each image region in a similar process to the text-to-image direction.

2. The image and text retrieval method based on semantic filtering and adaptive adjustment according to claim 1, characterized in that, Based on the feature representation module, features of the input image are extracted to obtain image feature representations for different regions, including: Image features of each region of the input image are extracted based on a pre-trained Faster R-CNN model. Image features are mapped to d-dimensional vectors through fully connected layers to generate feature representations of local regions. A self-attention mechanism is applied to local regions. This self-attention mechanism uses the average feature as the query and aggregates the feature representations of all image regions to generate a global image feature representation of the input image.

3. The image and text retrieval method based on semantic filtering and adaptive adjustment according to claim 1, characterized in that, Based on the feature representation module, features of the input text are extracted to obtain the text feature representation of each word, including: The input statement is split into multiple words and the words are embedded into the Bi-GRU model in an orderly manner. The text feature representation of each word is obtained by averaging the forward and backward hidden states at each time step.

4. The image and text retrieval method based on semantic filtering and adaptive adjustment according to claim 1, characterized in that, For the text-to-image direction, the formula for pre-assigning attention scores is: in, This represents the softmax activation function; This represents a scaling factor that further increases the gap between relevant and irrelevant image regions; u i The text features of the i-th word are represented by T; T represents transpose. v represents the transpose of the text feature vector of the i-th word; j The j-th image region represents the image features; m represents the number of words in the input statement; n represents the number of image regions in the input image. The formula for comparing the relative importance of two image regions is: in, This means that for the i-th word, the... The image region for the first Relative attention to each image region; This represents the confidence level of the compared image regions, and the first region being compared... The confidence score of each region is set as its correlation with that of the first region. The relevance of individual words; Reassigned attention matrix The expression is: Wherein, H(v) ij The expression indicates whether the j-th image region selected based on the i-th word is relevant; if relevant, it is 1, and if irrelevant, it is 0. For the i-th word, the matched relevant image regions for: Global image-text relevance is represented as: in, The correlation function represents the correlation score between modes u and v.

5. The image and text retrieval method based on semantic filtering and adaptive adjustment according to claim 4, characterized in that, For the image-to-text direction, the filtering attention module's steps for matching relevant words to each image region include: For the image features of each image region, cosine similarity is calculated between each word to obtain the similarity value of a certain image region for different words; The obtained similarity values ​​are normalized to the [0,1] interval by using the softmax activation function, thus completing the pre-assignment of attention scores; Compare the relative importance between two words. If the pre-assigned attention score of the word being compared is greater than that of the word being compared, then the score of the word being compared is greater than 0, and it is considered a relevant word; otherwise, it is considered an irrelevant word. Based on the calculation of function H, the scores of words with scores less than 0 are set to 0, and the scores of words with scores greater than 0 are set to 1, thus obtaining the redistributed attention matrix; The text features of each word are multiplied by the new attention matrix, and irrelevant words are filtered out to obtain the relevant word features for each image region.

6. The image and text retrieval method based on semantic filtering and adaptive adjustment according to claim 5, characterized in that, For the image-to-text direction, the formula for pre-assigning attention scores is: The formula for comparing the relative importance of two words is: in, This represents the relative attention of the i-th word to the t-th word for the j-th image region; Indicates the confidence level of the compared words; Reassigned attention matrix The expression is: in, This indicates whether the i-th word matched for the j-th image region is relevant. A value of 1 indicates that it is relevant, while a value of 0 indicates that it is irrelevant. For the j-th image region, the matched word for: Global image-text relevance is represented as: in, This indicates the similarity between a region in an image and the corresponding word in a sentence.

7. The image and text retrieval method based on semantic filtering and adaptive adjustment according to claim 1, characterized in that, The overall objective function of the adaptive attention filtering model is: Where S() represents the similarity function, which is used to calculate the similarity score between two modalities; , Represents positive examples of image region-word pairs. , Representing difficult examples of words and difficult examples of image regions; This represents a marginal parameter controlling the difference between positive and negative pairs; positive pairs should have higher similarity than negative pairs. The function... Equivalent to .

Images