Cross-modal retrieval method and system based on mixed attention mechanism

Abstract

The invention discloses a cross-modal retrieval method and system based on a mixed attention mechanism, and the method comprises the steps: carrying out the feature extraction of an image and a corresponding text based on ConvNeXt and BERT models, and carrying out the multi-modal feature aggregation according to the extracted features, so as to obtain a corresponding coarse-grained feature; performing cross-layer alignment on the output of the middle layer of the BERT and the output of the top layer of the ConvNeXt based on the coarse-grained features, and performing hierarchical alignment on the output of the top layer of the BERT and the output of the top layer of the ConvNeXt to obtain an image-text alignment result; and based on an image-text alignment result, carrying out cross-modal retrieval by adopting a hierarchical reasoning strategy of coarse-grained recall and fine-grained retrieval. According to the method, key information is extracted through an effective semantic filtering mechanism, more accurate cross-modal retrieval is realized, and the retrieval accuracy and efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of cross-modal retrieval, and particularly to a cross-modal retrieval method and system based on a hybrid attention mechanism. Background Art

[0002] In the prior art, cross-modal image-text retrieval methods mainly focus on achieving the matching between images and texts through direct correspondence relationships. In a 2019 study, Wu et al. proposed a method that aligns image regions and sentence words by locally correlating visual semantics, and integrates the semantic similarity between relevant region-word pairs to measure the overall image-text correlation. This method has made certain progress in local correlation, but may have limitations when dealing with more complex semantic interactions because it does not fully consider the deeper semantic relationships between images and texts.

[0003] In a 2021 study, Diao et al. modeled image and text data as a graph structure, utilized edge information transmission, and inferred local and global alignment relationships through a graph convolutional neural network to perceive similarity. Although this method has also shown good performance, it may cause misunderstandings or incorrect semantic inductions in complex or ambiguous scenarios, indicating that the model does not truly understand the semantic content.

[0004] There are also studies that use two-stream networks to process image and text data separately and fuse them at a deep layer of the network to achieve cross-modal retrieval. These methods can capture visual and language features respectively, but the lack of effective interaction in the early stage may lead to the loss of semantic information and affect the final retrieval performance.

[0005] In recent years, the attention mechanism has also been widely applied to cross-modal retrieval tasks to enhance the model's ability to focus on key information. However, most existing methods only use basic attention mechanisms and do not fully utilize attentions at different levels and directions to achieve deeper semantic alignment. Although these methods have improved the retrieval accuracy to a certain extent, they cannot completely solve the semantic difference problem in cross-modal retrieval.

[0006] In the cross-modal image-text retrieval task, due to the inherent semantic differences between visual content and text descriptions, it is difficult to achieve accurate retrieval matching. Traditional retrieval methods often only focus on the direct correspondence between visual elements in images and text keywords, ignoring the more complex semantic interactions between the two, which may lead to misunderstandings or incorrect semantic inductions in complex or ambiguous scenarios. In the field of cross-modal image-text retrieval, many methods, when dealing with the correspondence between images and texts, only focus on the matching between direct visual elements and text keywords, ignoring deeper semantic interactions and prone to misunderstandings or incorrect semantic inductions. Existing technologies also have deficiencies in integrating local details and global context information. Some methods may not be able to effectively utilize the global information of images to assist in the understanding of local features, or lack balance in dealing with global and local information, resulting in the retrieval model being unable to comprehensively understand the image content. Especially when dealing with images containing multiple objects and scenes, this imbalance between local and global information is particularly obvious. Existing technologies lack an effective semantic filtering mechanism. During the cross-modal retrieval process, non-critical details and noise may interfere with the model's capture and understanding of key semantic information. Many existing methods do not provide an effective mechanism to filter out this irrelevant information, affecting the accuracy of retrieval. In terms of the balance between retrieval efficiency and accuracy, existing technologies also face challenges. Although some methods have made progress in improving retrieval accuracy, this often comes at the cost of retrieval efficiency, especially when dealing with large-scale datasets.

[0007] Existing methods lack sufficient robustness to maintain stable retrieval performance when faced with noisy data, incomplete information, or abnormal situations, which limits the effectiveness and reliability of existing technologies in practical applications. Summary of the Invention

[0008] The present invention aims to solve at least one of the technical problems in the related technologies to some extent.

[0009] To this end, the present invention proposes a cross-modal retrieval method based on a hybrid attention mechanism, aiming to solve the limitations of existing technologies in dealing with cross-modal image-text retrieval tasks. The present invention utilizes a hybrid attention mechanism, including local relationship guidance and key semantic filtering, as well as cross-layer alignment technology, to optimize the cross-modal semantic alignment process.

[0010] Another object of the present invention is to propose a cross-modal retrieval system based on a hybrid attention mechanism.

[0011] To achieve the above object, on the one hand, the present invention proposes a cross-modal retrieval method based on a hybrid attention mechanism, including:

[0012] Extract features of images and corresponding texts based on the ConvNeXt and BERT models, and perform multi-modal feature aggregation based on the extracted features to obtain corresponding coarse-grained features;

[0013] Based on the coarse-grained features, perform cross-layer alignment between the intermediate layer output of BERT and the top layer output of ConvNeXt, and perform hierarchical alignment between the top layer output of BERT and the top layer output of ConvNeXt to obtain an image-text alignment result;

[0014] Perform cross-modal retrieval based on the image-text alignment result by adopting a hierarchical inference strategy of coarse-grained recall and fine-grained retrieval.

[0015] The cross-modal retrieval method based on the hybrid attention mechanism in the embodiments of the present invention may further have the following additional technical features:

[0016] In an embodiment of the present invention, extracting features of images and corresponding texts based on the ConvNeXt and BERT models, and performing multi-modal feature aggregation based on the extracted features to obtain corresponding coarse-grained features includes:

[0017] Obtain images and corresponding texts;

[0018] Respectively use the ConvNeXt and BERT models to extract local and global features of images and texts;

[0019] Perform fine-grained alignment on the local features and perform aggregation processing on the global features to finally obtain coarse-grained image features and text features.

[0020] In an embodiment of the present invention, performing cross-layer alignment between the intermediate layer output of BERT and the top layer output of ConvNeXt based on the coarse-grained features includes:

[0021] Based on the image-text retrieval task, calculate the similarity matrix of all word-region pairs;

[0022] Normalize the similarity matrix, and use the normalized similarity matrix to calculate the word context vector related to each image region to obtain the sentence vector obtained for each image region;

[0023] Aggregate all region-words based on the sentence vector to measure the similarity between the entire image and the sentence, so as to align the intermediate layer output of BERT and the top layer output of ConvNeXt across layers.

[0024] In an embodiment of the present invention, performing hierarchical alignment between the top layer output of BERT and the top layer output of ConvNeXt includes:

[0025] For the image-text retrieval task, attention weights are pre-assigned to each image region-text word pair;

[0026] By calculating the difference between the pre-assigned attentions of different words to measure the relevance of each word to other words, after finding other words related to the word, re-assign attention weights to each image region-text word pair;

[0027] Based on the re-assigned attention weights, calculate the weighted combination of all relevant words as the text shared semantic segment corresponding to the image region, and finally obtain the similarity between the image and the text, so as to hierarchically align the top-level output of BERT and the top-level output of ConvNeXt.

[0028] To achieve the above object, on the other hand, the present invention proposes a cross-modal retrieval system based on a hybrid attention mechanism, including:

[0029] A multi-modal feature aggregation module, which is used to extract features of images and corresponding texts based on the ConvNeXt and BERT models, and perform multi-modal feature aggregation according to the extracted features to obtain corresponding coarse-grained features;

[0030] An image alignment module, which is used to perform cross-layer alignment of the intermediate layer output of BERT and the top-level output of ConvNeXt based on the coarse-grained features, and hierarchically align the top-level output of BERT and the top-level output of ConvNeXt to obtain an image-text alignment result;

[0031] A modal retrieval hierarchical inference module, which is used to perform cross-modal retrieval based on the image-text alignment result by adopting a hierarchical inference strategy of coarse-grained recall and fine-grained retrieval.

[0032] The cross-modal retrieval method and system based on the hybrid attention mechanism according to the embodiments of the present invention deeply integrate local details and global context information, introduce a key semantic information filtering mechanism, and optimize the cross-modal semantic alignment process. The present invention pays attention to the context guiding relationship between the local and the global, extracts key information through an effective semantic filtering mechanism, realizes more accurate cross-modal retrieval, and improves the accuracy and efficiency of retrieval.

[0033] Additional aspects and advantages of the present invention will be given in part in the following description, become apparent in part from the following description, or be understood through the practice of the present invention. Description of the Drawings

[0034] The above and / or additional aspects and advantages of the present invention will become apparent and easy to understand from the following description of the embodiments in conjunction with the drawings, where:

[0035] Figure 1It is a logic diagram of a cross-modal retrieval method based on a hybrid attention mechanism according to an embodiment of the present invention;

[0036] Figure 2 It is a schematic diagram of a multi-modal retrieval driven by hybrid attention according to an embodiment of the present invention;

[0037] Figure 3 It is a flowchart of a cross-modal retrieval method based on a hybrid attention mechanism according to an embodiment of the present invention;

[0038] Figure 4 It is a schematic diagram of an implementation framework of a cross-layer alignment network driven by hybrid attention according to an embodiment of the present invention;

[0039] Figure 5 It is a structural diagram of a cross-modal retrieval system based on a hybrid attention mechanism according to an embodiment of the present invention. Detailed implementation manners

[0040] It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments may be combined with each other. The present invention will be described in detail below with reference to the drawings and in conjunction with the embodiments.

[0041] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0042] The cross-modal retrieval method and system based on a hybrid attention mechanism proposed according to an embodiment of the present invention will be described below with reference to the drawings.

[0043] The problems to be solved by the present invention can be formally defined as follows:

[0044] The input is a batch of images and texts. The present invention defines the visual representation as V = {v m , v h , v a}, where v m , v h are the features after being mapped by the fully connected layer in the intermediate layer and the top layer downsampling stage of the ConvNeXt model respectively, representing the local and global semantic information of the image respectively, and satisfying Here, n k represents the number of regional features, and v aIt refers to the coarse-grained feature vector obtained by processing the features output by the global average pooling module of the ConvNeXt model through the carefully designed visual aggregator of the present invention, and its dimension satisfies

[0045]

[0046] Similar to the extraction method of visual features, the present invention defines the text feature representation as C = {c m , c h , c a}, where c m and c h respectively refer to the outputs of the middle layer and the top layer of the BERT model, representing the local and global semantic information of the text respectively. Its dimension satisfies where n c represents the number of word features, and d represents the dimension of the feature in space. c a is the coarse-grained aggregation feature obtained by further processing through the text aggregator designed by the present invention,

[0047] The cross-modal image-text retrieval problem is to construct a method to maximize S(I i , T i ), where (I i , T i ) is the matching image-text pair, and S is the similarity calculated by the cross-modal retrieval method. Identify and match the keywords in the text with the visual features in the image, understand the relationship between the local regions inside the image, and extract and strengthen the core semantic matching information in the image and text, minimizing the distance between positive samples while maximizing the distance between negative samples of different modalities within the same batch, so as to achieve efficient and accurate cross-modal semantic alignment.

[0048] The present invention realizes effective semantic disambiguation while ignoring non-critical details and background interference, improves the retrieval efficiency, and optimizes the cross-modal semantic alignment process by designing a cross-layer alignment technology based on a hybrid attention mechanism. Its basic idea is to explore the guiding relationship existing between the global and local segments, use different attention mechanisms at different levels to achieve semantic disambiguation, and at the same time effectively filter non-critical details and noise to help the model deeply understand cross-modal semantic information. And a global contrast divergence loss is introduced as the training loss of the present invention, fully considering the distribution characteristics of image-text positive and negative sample pairs to ensure higher-quality semantic alignment. The whole calculation is carried out under a framework process. The overall framework process is as Figure 1 shown.

[0049] When an input given image is received, it is first converted into a vector form of H×W×C, where H and W represent the height and width resolutions of the image respectively, and C represents the number of color channels. An image instance is represented as The text T consists of a sequence of words {t 1 , t 2 ,..., t n}, where n represents the total number of words in the sentence. A text instance is represented as where C i represents the i-th word.

[0050] The present invention includes two stages: local relationship guidance and key semantic filtering. In the first stage, cross-attention is used to align the keyword-based query with the entire image and align the query of a specific region with the text, enhancing the association recognition between keywords and visual elements and the understanding between regions. In the second stage, global image-text alignment is achieved through unidirectional focus attention, focusing on core semantic matching and ignoring irrelevant details. As Figure 2 shown is an example of a hybrid attention-driven multimodal retrieval of the present invention.

[0051] Figure 3 is a flowchart of a cross-modal retrieval method based on a hybrid attention mechanism according to an embodiment of the present invention. As Figure 3 shown, the method includes:

[0052] S1, perform feature extraction on the image and the corresponding text based on the ConvNeXt and BERT models, and perform multimodal feature aggregation according to the extracted features to obtain corresponding coarse-grained features.

[0053] It can be understood that the present invention uses the keywords in the text as query conditions for a specific image and its corresponding text, and uses the whole image as an aid to guide the model to identify the matching relationship between the keywords and the corresponding visual features in the image. Using the local image region as the query condition and combining the overall text information as an aid, further guide the model to deeply understand the mutual connection between local regions inside the image.

[0054] Specifically, the embodiment of the present invention obtains the image and the corresponding text; respectively uses the ConvNeXt and BERT models to extract the local and global features of the image and the text; performs fine-grained alignment on the local features and performs aggregation processing on the global features, and finally obtains the coarse-grained image features and text features.

[0055] Specifically, for a given image and its corresponding text description, first extract the outputs of the intermediate and top layers from the BERT and ConvNeXt models. This process extracts the feature maps of multiple stages of ConvNeXt and introduces corresponding fully connected layers for each feature map to map the features of each region into a d-dimensional space. The image is converted into a vector form and then input into the ConvNeXt model to extract the local and global features of the image. The image representation is defined as V = {v h , v h , v a}. Where v m , v h are the features after mapping through the fully connected layer in the intermediate and top downsampling stages of the ConvNeXt model respectively, representing the local and global semantic information of the image, satisfying Here, n k represents the number of regional features. The text feature representation is defined as C = {c m , c h , c a}, where c m and c h are the outputs of the intermediate and top layers of the BERT model respectively, representing the local and global semantic information of the text. Their dimensions satisfy where n c represents the number of word features, and d represents the dimension of the feature in the space. Define and as the representations of the image and text instances respectively. Here, H and W represent the height and width resolutions of the image, and C represents the number of color channels, usually set to 3. Where C i represents the i-th word. The image encoder (ConvNeXt) and text encoder (BERT) are used to extract the local and global features of the image and text respectively. The formulas are as follows: [V l , V cls = ConvNeXt(I), [C l , C cls = BERT(T). and represent local features, and represent global features. The value of d v is 512 or 1024, and the value of d c is 768. Next, the output features V l and C l are subjected to fine-grained alignment, while V cls and C cls are processed by aggregators with a similar structure. The image aggregator is f v(), the text aggregator is f c >(). Each aggregator includes three fully connected (FC) blocks, and each block contains a feed-forward network (FFN) and a LeakyReLU (LR) activation function. The aggregated image and text features v are finally obtained in this step a and c a . v a refers to the coarse-grained image feature vector obtained by processing the features output by the global average pooling module of the ConvNeXt model through the image aggregator, and its dimension satisfies c a is the coarse-grained text aggregation feature obtained by further processing the embedding vector of the [CLS] token output by BERT through the text aggregator

[0056] v a = f v (V cls )

[0057] c a = f c (C cls )

[0058] S2. Based on the coarse-grained features, cross-layer alignment is performed between the intermediate layer output of BERT and the top layer output of ConvNeXt, and hierarchical alignment is performed between the top layer output of BERT and the top layer output of ConvNeXt to obtain the image-text alignment result

[0059] It can be understood that through the above steps, the model masters the ability to synthesize these image information to construct a semantic structure that fits the text description, such as the spatial position of objects in the image and their relationship attributes. In order to ensure the interaction and fusion of text and image information, a cross-attention mechanism is adopted. The present invention implements global alignment between images and texts through a unidirectional focal attention mechanism, achieving the purpose of extracting and strengthening the core semantic matching information in images and texts. Effectively implementing semantic disambiguation while ignoring non-critical details and background interference makes the model more accurately understand and match the core information of images and texts

[0060] Such as Figure 4 shown, the present invention designs a cross-layer alignment mechanism driven by hybrid attention, using ConvNeXt and BERT as the visual encoder and text encoder respectively. For a given image and corresponding text description, after extracting the intermediate layer and top layer outputs of the BERT and ConvNeXt models, the intermediate layer output c of BERT m is cross-layer aligned with the top layer output v of ConvNeXt h to implement the cross-attention alignment mechanism. Then the top layer output c of BERT hand the top - layer output v of ConvNeXt h are hierarchically aligned to implement a one - way focus attention mechanism for image - text and text - image.

[0061] In an embodiment of the present invention, based on the image - text retrieval task, a similarity matrix of all word - region pairs is calculated; the similarity matrix is normalized, and the word context vectors related to each image region are calculated using the normalized similarity matrix to obtain the sentence vectors corresponding to each image region; based on the sentence vectors, all region - words are aggregated to measure the similarity between the entire image and the sentence, so as to achieve cross - layer alignment between the intermediate - layer output of BERT and the top - layer output of ConvNeXt.

[0062] Specifically, define c m ={c 1 , c 2 ,..., c n}, where n represents the number of words in the given text. Define v h ={v 1 , v 2 ,..., v k}, representing the number of image regions. For image - text retrieval, first calculate the similarity matrix of all word - region pairs:

[0063]

[0064] where s ij represents the similarity between the i - th image region and the j - th word. Then normalize the similarity matrix:

[0065]

[0066] where [x] + = max(x, 0), and use the normalized similarity matrix to calculate the word context vectors related to each image region:

[0067]

[0068] Furthermore, obtain the sentence vectors corresponding to each image region:

[0069]

[0070] Then aggregate all region - words to measure the similarity between the entire image I and the sentence T. For text - image cross - modal retrieval, it is necessary to calculate and the similarity of c j and perform an aggregation operation. Similarly, the top - layer output c of BERT hWith the middle-level output v of ConvNeXt m A similar formula is also followed when performing cross-layer stacked cross-attention alignment. In short, the cross-layer alignment from text words to the entire global image can help the model learn the matching relationship between text words and corresponding elements in the image, while the cross-layer alignment from image regions to the entire text guides the model to deeply understand the mutual connections between local regions within the image, making full use of the text context. In addition, using the stacked cross-attention mechanism in the alignment stage can gradually refine and synthesize information at different information levels, strengthening the complementarity between vision and language, which is particularly important for processing various features from details to the global level, helping the model understand how these local information are combined into the overall semantic process, and being more in line with the cognitive intuition of humans to understand complex information.

[0071] Subsequently, the middle-level output c of BERT m is aligned with the top-level output v of ConvNeXt h to achieve a cross-layer alignment mechanism through a cross-attention alignment process.

[0072] In an embodiment of the present invention, based on the image-text retrieval task, attention weights are pre-allocated for each image region-text word pair; the correlation between each word and other words is measured by calculating the difference between the pre-allocated attention of different words. After finding other words related to the word, attention weights are re-allocated for each image region-text word pair; the weighted combination of all related words is calculated based on the re-allocated attention weights as the text shared semantic segment corresponding to the image region, and finally the similarity between the image and the text is obtained, so as to hierarchically align the top-level output of BERT and the top-level output of ConvNeXt.

[0073] Specifically, the top-level output c of BERT h and the top-level output v of ConvNeXt h are hierarchically aligned. The present invention realizes a one-way focus attention mechanism for image-text and text-image. For the image-text retrieval task, first, attention weights are pre-allocated for each image region-text word pair:

[0074]

[0075] where w ij is the pre-allocated attention weight between the i-th region of the image and the j-th word of the text, σ is the softmax activation function, α is a scaling factor used to increase the gap between relevant and irrelevant regions (a scalar), and its value is set to 20. Then, the correlation between each word and other words is measured by calculating the difference between the pre-allocated attention of different words:

[0076]

[0077] where R is the correlation function, is the indicator function, when the input F(x) is greater than 0, the value is 1, otherwise 0.

[0078]

[0079] f(c ij ,c it ) = w ij -w it

[0080]

[0081] F(c ij ) is the correlation scoring function with respect to the j-th word based on the i-th image region as the query context, f(c ij ,c it ) is the pre-allocated attention difference between the j-th word in the text and the t-th compared word, g(c it ) is the pre-confidence of the t-th compared word. After finding other words related to the j-th word, re-allocate the attention weights for each image region-text word pair:

[0082]

[0083] Based on the above formula, the re-allocated attention weights can be obtained, and then calculate the weighted combination of all relevant words as the text shared semantic segment corresponding to the i-th image region Finally, the similarity between the image I and the text T is obtained:

[0084]

[0085] The present invention focuses the focal attention on a single direction of image-text or text-image. Specifically, using focal attention in the top alignment stage of BERT and ConvNeXt aims to solve the possible semantic induction bias in the previous cross-layer cross-attention alignment process, help the model focus on the features most critical to the retrieval task, eliminate ambiguity and reduce the attention to unimportant information at the same time.

[0086] S3, perform cross-modal retrieval based on the image-text alignment result by adopting a hierarchical inference strategy of coarse-grained recall and fine-grained retrieval.

[0087] It can be understood that the present invention adopts a hierarchical inference strategy of coarse-grained recall and fine-grained retrieval to improve the retrieval efficiency of image and text matching, ensure the consistency of image and text pairs within and across modalities, thereby improving the accuracy and robustness of the model. Finally, the model can more effectively complete the cross-modal semantic alignment task.

[0088] Specifically, the present invention combines this coarse-grained recall with a fine-grained ranking method based on the attention mechanism, not only achieving good accuracy but also effectively improving the retrieval efficiency. The coarse-grained aggregated features v of vision and text have been completed in step S1 a and c a are coarsely aligned, and then combined with the fine-grained retrieval method of the cross-layer alignment mechanism driven by the aforementioned hybrid attention for multi-task learning. During the retrieval process, first, the coarse-grained retrieval method is used to quickly recall the top K samples (where K is much smaller than the number of target retrieval modalities in the dataset), and then the cross-layer alignment framework (including local relationship guidance and key semantic filtering) based on the hybrid attention mechanism is used to re-rank these K samples, achieving efficient and accurate cross-modal retrieval.

[0089] The present invention adopts a hierarchical inference strategy of coarse-grained recall and fine-grained retrieval, quickly narrowing the search scope through coarse-grained recall, and then precisely ranking the candidate samples through fine-grained retrieval, thereby achieving efficient and accurate cross-modal retrieval. Ensuring higher-quality semantic alignment, guiding the model to better learn the semantic correspondence between images and texts during training, and improving the accuracy and robustness of the model.

[0090] According to the cross-modal retrieval method based on the hybrid attention mechanism of the embodiments of the present invention, by introducing the hybrid attention mechanism and cross-layer alignment technology, the performance of cross-modal image-text retrieval is significantly improved. The adopted feature extraction and multi-modal feature aggregation methods enable the model to extract rich semantic information from images and texts and effectively aggregate them, improving the model's ability to understand the semantic content of images and texts and enhancing the accuracy and robustness of retrieval. Adopting a hierarchical inference strategy, through the combination of coarse-grained recall and fine-grained retrieval, the retrieval efficiency is significantly improved, enabling the model to quickly recall candidate samples related to the query and precisely rank these candidate samples through fine-grained retrieval, achieving efficient and accurate cross-modal retrieval. Through the hybrid attention mechanism, cross-layer alignment technology, feature extraction and multi-modal feature aggregation, and hierarchical inference strategy, the accuracy, efficiency, and robustness of cross-modal image-text retrieval are comprehensively improved, providing an effective solution to the semantic difference problem in cross-modal retrieval.

[0091] To implement the above embodiments, as Figure 5As shown in the figure, in this embodiment, a cross-modal retrieval system 10 based on a hybrid attention mechanism is further provided, including:

[0092] A multi-modal feature aggregation module 100, configured to extract features of an image and its corresponding text based on the ConvNeXt and BERT models, and perform multi-modal feature aggregation according to the extracted features to obtain corresponding coarse-grained features;

[0093] An image alignment module 200, configured to perform cross-layer alignment between the intermediate layer output of BERT and the top layer output of ConvNeXt based on the coarse-grained features, and perform hierarchical alignment between the top layer output of BERT and the top layer output of ConvNeXt to obtain an image-text alignment result;

[0094] A modal retrieval hierarchical inference module 300, configured to perform cross-modal retrieval based on the image-text alignment result by adopting a hierarchical inference strategy of coarse-grained recall and fine-grained retrieval.

[0095] Furthermore, the multi-modal feature aggregation module 100 is further configured to:

[0096] Obtain an image and its corresponding text;

[0097] Respectively use the ConvNeXt and BERT models to extract local and global features of the image and text;

[0098] Perform fine-grained alignment on the local features and perform aggregation processing on the global features to finally obtain coarse-grained image features and text features.

[0099] Furthermore, performing cross-layer alignment between the intermediate layer output of BERT and the top layer output of ConvNeXt based on the coarse-grained features includes:

[0100] Based on the image-text retrieval task, calculate the similarity matrix of all word-region pairs;

[0101] Normalize the similarity matrix, and use the normalized similarity matrix to calculate the word context vector related to each image region to obtain the sentence vector obtained for each image region;

[0102] Aggregate all region-words based on the sentence vector to measure the similarity between the entire image and the sentence, so that the intermediate layer output of BERT is cross-layer aligned with the top layer output of ConvNeXt.

[0103] Furthermore, performing hierarchical alignment between the top layer output of BERT and the top layer output of ConvNeXt includes:

[0104] Based on the image-text retrieval task, pre-assign attention weights to each image region-text word pair;

[0105] The relevance between each word and other words is measured by calculating the difference in pre-allocated attention of different words. After finding other words related to the word, the attention weights are re-allocated for each image region-text word pair.

[0106] Based on the re-allocated attention weights, the weighted combination of all relevant words is calculated as the text shared semantic segment corresponding to the image region, and finally the similarity between the image and the text is obtained, so that the top-level outputs of BERT and ConvNeXt are hierarchically aligned.

[0107] According to the cross-modal retrieval system based on the hybrid attention mechanism of the embodiments of the present invention, by introducing the hybrid attention mechanism and the cross-layer alignment technology, the performance of cross-modal image-text retrieval is significantly improved. The feature extraction and multi-modal feature aggregation methods adopted enable the model to extract rich semantic information from images and texts and effectively aggregate them, improving the model's ability to understand the semantic content of images and texts and enhancing the accuracy and robustness of retrieval. The hierarchical inference strategy, through the combination of coarse-grained recall and fine-grained retrieval, significantly improves the retrieval efficiency, enabling the model to quickly recall candidate samples related to the query and accurately rank these candidate samples through fine-grained retrieval, achieving efficient and accurate cross-modal retrieval. Through the hybrid attention mechanism, cross-layer alignment technology, feature extraction and multi-modal feature aggregation, and hierarchical inference strategy, the accuracy, efficiency, and robustness of cross-modal image-text retrieval are comprehensively improved, providing an effective solution to the semantic difference problem in cross-modal retrieval.

[0108] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0109] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" can explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "a plurality" is at least two, such as two, three, etc., unless otherwise specifically defined.

Claims

1. A cross-modal retrieval method based on a hybrid attention mechanism, characterized in that: include: Based on the ConvNeXt and BERT models, the features of the image and the corresponding text are extracted, and multimodal feature aggregation is performed based on the extracted features to obtain the corresponding coarse-grained features; Based on the coarse-grained features, the intermediate layer output of BERT is cross-layer aligned with the top layer output of ConvNeXt, and the top layer output of BERT is hierarchically aligned with the top layer output of ConvNeXt to obtain an image-text alignment result; Based on the image-text alignment results, cross-modal retrieval is performed by adopting a hierarchical reasoning strategy of coarse-grained recall and fine-grained retrieval.

2. The method according to claim 1, characterized in that Based on the ConvNeXt and BERT models, the image and the corresponding text features are extracted, and multimodal feature aggregation is performed based on the extracted features to obtain the corresponding coarse-grained features, including: Get the image and the corresponding text; ConvNeXt and BERT models are used to extract local and global features of images and texts respectively; The local features are fine-grained aligned and the global features are aggregated to finally obtain coarse-grained image features and text features.

3. The method according to claim 1, characterized in that Based on the coarse-grained features, the intermediate layer output of BERT is aligned with the top layer output of ConvNeXt, including: Based on the image-text retrieval task, the similarity matrix of all word-region pairs is calculated; The similarity matrix is ​​normalized, and the normalized similarity matrix is ​​used to calculate the word context vector associated with each image region to obtain the sentence vector obtained relative to each image region; Based on the sentence vector, all region-words are aggregated to measure the similarity between the entire image and the sentence, so that the intermediate layer output of BERT is aligned with the top layer output of ConvNeXt across layers.

4. The method according to claim 1, characterized in that: Hierarchically align the top-level output of BERT with the top-level output of ConvNeXt, including: Based on the image-text retrieval task, attention weights are pre-assigned to each image region-text word pair; The relevance of each word to other words is measured by calculating the difference in pre-assigned attention of different words. After finding other words related to the word, the attention weight is reassigned to each image region-text word pair. Based on the reallocated attention weights, a weighted combination of all relevant words is calculated as the shared semantic fragment of the text corresponding to the image area, and finally the similarity between the image and the text is obtained, so that the top-level output of BERT and the top-level output of ConvNeXt are hierarchically aligned.

5. A cross-modal retrieval system based on a hybrid attention mechanism, characterized in that: include: The multimodal feature aggregation module is used to extract features of images and corresponding texts based on the ConvNeXt and BERT models, and to perform multimodal feature aggregation based on the extracted features to obtain corresponding coarse-grained features; An image alignment module, used to perform cross-layer alignment of the intermediate layer output of BERT with the top layer output of ConvNeXt based on the coarse-grained features, and hierarchically align the top layer output of BERT with the top layer output of ConvNeXt to obtain an image-text alignment result; The modality retrieval hierarchical reasoning module is used to perform cross-modality retrieval based on the image-text alignment results by adopting a hierarchical reasoning strategy of coarse-grained recall and fine-grained retrieval.

6. The system according to claim 5, characterized in that The multimodal feature aggregation module is also used for: Get the image and the corresponding text; ConvNeXt and BERT models are used to extract local and global features of images and texts respectively; The local features are fine-grained aligned and the global features are aggregated to finally obtain coarse-grained image features and text features.

7. The method according to claim 5, characterized in that Based on the coarse-grained features, the intermediate layer output of BERT is aligned with the top layer output of ConvNeXt, including: Based on the image-text retrieval task, the similarity matrix of all word-region pairs is calculated; The similarity matrix is ​​normalized, and the normalized similarity matrix is ​​used to calculate the word context vector associated with each image region to obtain the sentence vector obtained relative to each image region; Based on the sentence vector, all region-words are aggregated to measure the similarity between the entire image and the sentence, so that the intermediate layer output of BERT is aligned with the top layer output of ConvNeXt across layers.

8. The system according to claim 5, characterized in that Hierarchically align the top-level output of BERT with the top-level output of ConvNeXt, including: Based on the image-text retrieval task, attention weights are pre-assigned to each image region-text word pair; The relevance of each word to other words is measured by calculating the difference in pre-assigned attention of different words. After finding other words related to the word, the attention weight is reassigned to each image region-text word pair. Based on the reallocated attention weights, a weighted combination of all relevant words is calculated as the shared semantic fragment of the text corresponding to the image area, and finally the similarity between the image and the text is obtained, so that the top-level output of BERT and the top-level output of ConvNeXt are hierarchically aligned.

Images