Gradual semantic aggregation and structured cognitive enhancement-based image-text matching method

Abstract

The invention discloses an image-text matching method based on progressive semantic aggregation and structured cognitive enhancement, and relates to the technical field of image-text matching, comprising the following steps: S1, multi-modal feature extraction; s2, introducing a hierarchical self-enhanced semantic perception pooling mechanism; s3, introducing a structured knowledge auxiliary learning module; s4, introducing a cross-modal semantic alignment mechanism; s5, introducing a multi-scale semantic association matrix; according to the method, through multi-level semantic enhancement and fine-grained level self-adaptive feature aggregation, a significant word field is enhanced, and consistency and unification between local representation and global representation are achieved; extracting structured semantics between entities from an image title corpus, mining high-order relationships in long and short sentences, and expanding concepts missed in text global embedding by utilizing the priori knowledge; a cross-modal relation interaction mechanism is introduced, semantic embedding is enhanced, and more consistent semantic understanding is promoted; a multi-scale semantic association matrix is introduced, and a more comprehensive and accurate similarity measurement matrix is generated.

Description

Technical Field

[0001] The present invention relates to the technical field of image-text matching, and in particular to an image-text matching method based on progressive semantic aggregation and structured cognitive enhancement. Background Art

[0002] Image-text matching refers to finding matching texts through images or searching for semantically similar images given text descriptions. This is a fundamental task to bridge the semantic gap between two heterogeneous modalities. Connecting these two modes can facilitate a large number of multimodal tasks, such as cross-modal retrieval, image or video captioning, and text-to-image generation. Although exciting progress has been made in recent years, it is still challenging to accurately and effectively learn cross-modal embeddings and semantic similarity between images and texts due to the inconsistency and heterogeneity in representation and distribution of images and texts. To address this challenge, the Visual Semantic Embedding (VSE) framework is proposed to learn a unified joint embedding space. The general paradigm can be summarized into three steps: 1) Use a separate encoder to extract local features of image and text; 2) Design feature aggregation strategies or fine-grained interactions for semantic-aware learning and map them to a shared representation space; 3) Compute the semantic similarity between cross-modal features. Existing work mainly focuses on the exploration of the second step. Recent work studies pooling strategies. They design different pooling functions to aggregate local features and obtain overall embeddings, and then check the matching scores between the two modes. Another work dedicated to studying fine-grained feature interactions focuses on modeling the relationship between all image regions and text domains and local feature interactions to infer the correlation between image and text instances. Compared with independent embedding methods that only focus on self-modal interactions, modeling inter-modal relationships usually has better performance. Therefore, introducing cross-modal interactions in interactive embedding models has become a research trend. Most methods in this branch mainly use cross-attention networks and their variants to model cross-modal semantic associations. In order to fully explore semantic associations, there is also a category of methods that explore structured information to guide image and text representation learning. Most of these methods obtain certain prior knowledge from the outside, such as scene graphs. By introducing prior knowledge of entity semantic information and entity association information, the understanding of semantic information within the modality is enhanced to a certain extent, and the accuracy of image-text matching is improved. For metric learning, image-text matching strategies usually use dimensional correspondence to construct cross-modal similarity, that is, typical cosine similarity and weighted similarity, to improve the distinguishability between different sample pairs and the correlation between the same sample pairs.

[0003] Previous aggregation methods mainly focus on intra-modal fine-grained feature interactive learning, and mechanically measure the overall image-text relevance through global representation. In the aggregation process, the influence of global semantics is ignored, and the macroscopic contribution of each local region to the overall image semantics is not fully considered. Figure 1As shown in the figure, the traditional method uses uniform weight pooling and treats image regions and text words with equal weights (a=0.1). This simplistic treatment makes it difficult to distinguish key information from minor details, significantly reducing the matching accuracy. The mean pooling strategy only pays equal attention to the internal correlation between each region and assigns equal attention scores. Without considering the macro contribution, non-core fragment-level regions are aggregated indiscriminately, and the consideration of inter-fragment correlation at the regional level is not comprehensive enough, which interferes with the measurement of overall correlation. Although methods based on local feature fusion can capture fine-grained semantic information, they often fail to extract key structural semantics in sentences. This limits the model's ability to understand and express complex semantic relationships.

[0004] In view of this, a graph-text matching method based on progressive semantic aggregation and structured cognitive enhancement is provided to overcome the above problems. Summary of the invention

[0005] The purpose of the present invention is to provide a graph-text matching method based on progressive semantic aggregation and structured cognitive enhancement to solve the problems raised in the above background technology.

[0006] In order to solve the above technical problems, the present invention provides a method for image-text matching based on progressive semantic aggregation and structured cognitive enhancement, comprising the following steps:

[0007] S1. Multimodal feature extraction: including region-aware image representation and context-sensitive text word encoding, extracting visual region or text word features as input;

[0008] S2. Introducing a hierarchical self-enhanced semantic-aware pooling mechanism: Aggregating visual region or text word features to generate global feature embedding through self-guided enhancement modules and semantic-aware modules;

[0009] S3, introduces structured knowledge-assisted learning module: It includes three key steps to construct a structured representation rich in semantic information: extracting triple phrases, building a neighborhood semantic association graph, and generating instance-level structured embedding to inject prior knowledge into the global semantic embedding;

[0010] S4. Introducing a cross-modal semantic alignment mechanism: Using an asynchronous method to explore deep interactions between and within modalities of visual and textual embedding vectors;

[0011] S5. Introducing multi-scale semantic association matrix: Constructing a multi-dimensional semantic association matrix based on the hierarchical matching paradigm of multiple perspectives.

[0012] Furthermore, in region-aware image representation, given an image, a bottom-up attention network is adopted, and Faster-RCNN is used to extract the top K salient regions, and the regional features are obtained through ResNet-101. The extracted regional features are converted into a specific dimension by combining a multi-layer perceptron and a fully connected layer. In context-sensitive text word encoding, given a text containing N words, the BERT model is used to mark the input text as subword units, and a fully connected layer is applied to convert the extracted text word features into a specific dimension.

[0013] Furthermore, in S2, a dimension-based sorting method is used to sort the input features, and the self-guided enhanced pooling operator extracts significant elements from the sorted feature vector. SEPO overcomes the limitations of traditional maximum pooling and K-means pooling by adopting a smooth weight allocation mechanism. In the semantic perception module, the self-enhanced global features are used as the query matrix, and the original sorted features are input into the self-attention mechanism as keys and values. The aggregation coefficient matrix learned by the attention mechanism is used to adaptively quantify the semantic correlation between each fragment-level representation and the self-guided enhanced global representation to generate a semantically-aware weight vector. The two global feature vectors are fused using the adaptive weight learning method of the gating mechanism to obtain the final visual aggregate embedding and text aggregate embedding.

[0014] Furthermore, in S3, an end-to-end relation extraction model REBEL is adopted, which takes the original text as input and outputs a linearized triple sequence, transforming the relation extraction task into a generation task; in the knowledge enhanced encoder, BERT-base is used to obtain the embedding of each entity, and a graph convolutional network is introduced to optimize the embedding representation of the node through an iterative neighborhood information aggregation process, integrating the global embedding of the text with the structured knowledge embedding, and using a specific formula to integrate the two embeddings.

[0015] Furthermore, in S4, a multi-head cross-attention mechanism is used to capture the relational interactions within different modalities, and the semantic dependencies between modalities are processed through a joint-modality multi-head self-attention mechanism. The relations obtained in the first two stages are nonlinearly transformed through a feedforward neural network. The hard negative triplet loss is combined with the hard negative sample mining strategy, and the triplet loss is introduced and extended to enhanced visual-text embedding.

[0016] Furthermore, in S5, we first use the global representation of images and texts to calculate the similarity and construct a global semantic association matrix. Then, we introduce a multidimensional semantic association matrix based on a hierarchical matching paradigm from multiple perspectives, perform hierarchical iterative aggregation on the fragment-level representation, and obtain a dynamic multidimensional enhanced representation. Based on the multidimensional enhanced representation, we calculate the semantic similarity of word region pairs, infer the instance-level correlation of visual text from two directions, unify different semantic representations in the same semantic space through mean pooling, and introduce a marginal triplet ranking loss based on the most difficult sample to optimize the multidimensional semantic association matrix.

[0017] Furthermore, the total loss of the entire model is defined by a specific formula; in the experimental comparison, evaluation is carried out on two public datasets, specific evaluation indicators are adopted, specific implementation details are set, and comparison is made with the most advanced methods. Ablation experiments are performed to verify the effectiveness of the framework, including analysis of different network structures, namely aggregation modules, structured knowledge extraction, cross-modal alignment, the effectiveness of multi-scale semantic association matrices, and module synergy.

[0018] Furthermore, hyperparameter analysis is performed, including the impact of the number of iterative aggregation structure layers and the knowledge integration coefficient, and cross-dataset migration experiments are conducted to evaluate the generalization ability of the model.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] The hierarchical self-enhanced semantic-aware pooling mechanism effectively strengthens the salient word domain through multi-level semantic enhancement and fine-grained adaptive feature aggregation, achieving consistent unification between local and global representations.

[0021] External knowledge is introduced into the PASE method to extract structured semantics between entities from the image caption corpus, mine high-order relations in long (short) sentences, and use this prior knowledge to expand the missing concepts in the global embedding of text.

[0022] A cross-modal relation interaction mechanism is introduced to better bridge the semantic gap between different modalities. This mechanism collaboratively processes the semantic information of different modalities, enhances semantic embedding, and promotes more consistent semantic understanding.

[0023] This method is based on a dynamic semantic similarity learning framework. It combines fine-grained representation to construct a multi-dimensional semantic relevance matrix, which is then integrated with the global semantic connection matrix to generate a more comprehensive and accurate similarity measurement matrix.

[0024] Extensive experiments are conducted on two public benchmark datasets for image-text retrieval (Flickr30K and MS-COCO) to verify the effectiveness and superiority of our PASE framework. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a diagram illustrating working motivation in the image-text matching method based on progressive semantic aggregation and structured cognitive enhancement of the present invention;

[0026] Figure 2 It is a network framework diagram of the image-text matching method based on progressive semantic aggregation and structured cognitive enhancement of the present invention;

[0027] Figure 3A multi-level semantic enhancement and feature integration diagram in the image-text matching method based on progressive semantic aggregation and structured cognitive enhancement of the present invention;

[0028] Figure 4 A multi-view hierarchical matching strategy diagram in the image-text matching method based on progressive semantic aggregation and structured cognitive enhancement of the present invention;

[0029] Figure 5 It is a parameter analysis diagram of the number of iteration layers in Formula 21 in the image-text matching method based on progressive semantic aggregation and structured cognitive enhancement of the present invention on Flickr30K;

[0030] Figure 6 This is a parameter analysis diagram of the knowledge integration coefficient in Formula 18 in the image-text matching method based on progressive semantic aggregation and structured cognitive enhancement of the present invention on Flickr30K. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] See also Figure 1-Figure 6 , the present invention provides a technical solution:

[0033] See also Figure 1 : Description of work motivation; Figure 1 Middle (a): The traditional method uses uniform weight pooling, which treats image regions and text words with equal weights (a = 0.1). This simplistic treatment makes it difficult to distinguish key information from minor details, significantly reducing matching accuracy. Figure 1 Middle (b): The optimization strategy we proposed introduces a differentiated weight allocation mechanism to assign macro contributions to image regions and text words according to their semantic importance. At the same time, we fully explore semantic associations by extracting structured semantic tags (such as "soccer", "practicedby", and "boys"). This method accurately captures key information, strengthens core semantics while retaining global features, and improves the performance of traditional image-text matching methods in complex scenarios.

[0034] like Figure 2As shown in Figure 3, our proposed framework aims to fully utilize the full-granularity representations and learn their correspondences, providing richer discriminative information to promote comprehensive semantic similarity learning. The framework first extracts visual region or text word features as input, and then aggregates the visual region or text word features through an intra-modality pooler to generate a global feature embedding.

[0035] To deeply explore the high-order semantic associations in sentences, we introduce a structured knowledge-assisted learning module, which includes three key steps to build a structured representation rich in semantic information:

[0036] Extract triple phrases, build neighborhood semantic association graphs, and generate instance-level structured embeddings to inject prior knowledge into global semantic embeddings.

[0037] Subsequently, we learn meaningful correspondences between cross-modal feature embeddings to jointly determine visual-textual semantic similarity.

[0038] Finally, two types of instance-level semantic association matrices are constructed based on the learned global semantic embedding and fine-grained representation.

[0039] Formally, for a given image-text pair (I, T), a set of visual regions represents the image features Instead, text is represented by a set of text word features where n r and n t They represent the number of regions and words respectively, and d is the dimension of the feature vector.

[0040] Figure 2 The proposed network framework is presented, which works in tandem through four core modules. First, the hierarchical self-enhanced semantic-aware pooler adopts a two-stream fine-grained feature fusion mechanism to improve the model expressiveness through multi-level semantic enhancement and adaptive feature fusion. Second, the structured knowledge-assisted learning module parses the original text into structured triple representations, builds a neighborhood semantic association graph, and integrates structured knowledge through a knowledge-enhanced encoder. Then, the cross-modal semantic alignment mechanism uses an asynchronous method to deeply explore the inter-modal and intra-modal relational interactions between visual and textual embedding vectors. Finally, the region-level and instance-level semantic association matrices are constructed based on the learned global semantic representation and the fine-grained features before aggregation. We use the mean pooling method to fuse these two semantic association matrices to generate the final retrieval matrix.

[0041] See also Figure 1-Figure 6 , the image-text matching method based on progressive semantic aggregation and structured cognitive enhancement includes the following steps:

[0042] S1. Multimodal feature extraction: including region-aware image representation and context-sensitive text word encoding, extracting visual region or text word features as input;

[0043] Region-aware image representation:

[0044] To ensure a fair comparison with previous methods, given an image I, the same bottom-up attention network is used; Faster-RCNN is used to extract the top k salient regions; and regional features are obtained through ResNet-101. To ensure matching with the task, a multi-layer perceptron (MLP) and a fully connected layer (FC) are then combined to transform the extracted regional features into v i Dimension;

[0045] v i =W v r i +b v ,#(1)

[0046] W υ represents the weight matrix of the fully connected (FC) layer, and b υ is a learnable bias vector;

[0047] Finally, we obtain a set of regional features as the feature representation of the image, which can be expressed as:

[0048]

[0049] Context-sensitive text word encoding:

[0050] Given a containing n t The input text T of words is tokenized into word-piece tokens using the BERT model, and each token is represented using the learned embedding. Then, n t Text representation Subsequently, a fully connected (FC) layer is applied to transform the extracted text word features into t j Dimension;

[0051] t j =W t e j +b t ,#(3

[0052] W t represents the weight matrix of the fully connected (FC) layer, b t is a learnable bias vector; finally, we obtain a set of word features as text feature representation, which can be expressed as:

[0053]

[0054] Figure 3 Hierarchical Self-Enhanced Semantic-Aware Pooler is an efficient two-stream mechanism for fine-grained feature aggregation; it improves the expressiveness of the model through multi-level semantic enhancement and feature integration;

[0055] S2. Introducing a hierarchical self-enhanced semantic-aware pooling mechanism: Aggregating visual region or text word features to generate global feature embedding through self-guided enhancement modules and semantic-aware modules; specifically:

[0056] To better integrate multimodal fine-grained features, we introduce a hierarchical self-enhanced semantic-aware pooling mechanism, such as Figure 3 As shown in the figure, this efficient two-stream fine-grained feature aggregation mechanism contains two core modules: a self-guided enhancement module and a semantic perception module. In the initial stage of feature aggregation, we introduce a specific sorting mechanism to reorganize the input features of text or images.

[0057] Taking the image branch as an example, assume that the input feature consists of a vector set I = {v1, v2, …, v k} indicates that we use a dimension-based sorting method; the sorting function max k (·) Rearrange the feature vectors; the sorted feature sequence can be expressed as

[0058] in, Corresponding to the vector with the largest dimension value in the original vector set, and so on; this sorting method can be formally expressed as:

[0059]

[0060] Self-booting enhancement module:

[0061] Proposed self-guided enhanced pooling operator:

[0062] Self-guided Enhancement Pooling Operator (SEPO) is a feature-based generalized pooling operator. It extracts significant elements from the sorted feature vector and captures key semantic information more accurately. SEPO overcomes the limitations of traditional maximum pooling and K-means pooling by adopting a smooth weight distribution mechanism. It can be regarded as a stable variant of soft maximum pooling, which enhances the expression of global features while maintaining sensitivity to subtle features. SEPO can be expressed as:

[0063]

[0064] where v serepresents the global representation of self-guided enhancement, Represents the features of the i-th image region after sorting;

[0065]

[0066] here, represents the self-guided enhancement weight vector, that is, the weight parameter of the i-th region feature;

[0067] Semantic perception module:

[0068] The global embedding v obtained by self-guided enhanced pooling se It captures the key semantic information of the image to a certain extent; however, as a static feature fusion method, it lacks the ability to dynamically adjust according to complex contexts; in order to achieve dynamic interactive learning between features and more accurate region-level feature encoding, we propose the Semantic Aware Pooling Operator (SAPO);

[0069] SAPO is a refined feature semantic interaction learning pooling method that aims to achieve secondary enhancement and efficient aggregation of local features. Specifically, it uses the self-enhanced global features as the query matrix (Q) and inputs the original sorted features into the self-attention mechanism as keys (K) and values ​​(V). The aggregation coefficient matrix learned by the attention mechanism can adaptively quantify the semantic relevance between each fragment-level representation and the self-guided enhanced global representation and weight their importance probabilistically. This process achieves semantic enhancement guided by the global representation and effectively captures the intrinsic associations between features:

[0070]

[0071] Through this attention mechanism, SAPO generates a semantically aware weight vector Represents the importance weight of the i-th region feature:

[0072]

[0073] Different from traditional methods, SAPO uses the global representation obtained from self-guided enhanced pooling to guide fine-grained feature fusion and generate a more accurate coefficient matrix for subsequent feature aggregation; the aggregation method can be expressed as:

[0074]

[0075] Among them, v sa represents a semantically aware global representation, while Represents the sorted i-th image region feature;

[0076] In order to enhance the completeness and diversity of the final aggregated features, we need to fuse two d-dimensional global feature vectors v se and v sa ; To avoid the limitation of manual parameter adjustment, we propose an adaptive weight learning method based on a gating mechanism; this method dynamically adjusts the weight coefficients through a lightweight neural network layer to calculate the final visual aggregate embedding v g , and its calculation formula is as follows:

[0077]

[0078] Where τ is a parameter used to adaptively balance v se and v sa The gate value of importance, [·,·] represents the concatenation operation, W v is a learnable weight matrix, b v is the bias term;

[0079] Similarly, the text aggregation embedding t g The calculation formula is as follows:

[0080]

[0081] Through this adaptive fusion mechanism of instance semantics, we obtain the final visual-text converged embedding. and Where m is the batch size;

[0082] S3, introduce structured knowledge assisted learning module: including three key steps to build a structured representation rich in semantic information: extracting triple phrases, building a neighborhood semantic association graph, and generating instance-level structured embedding to inject prior knowledge into the global semantic embedding; specifically:

[0083] Structured triples: To improve the semantic expression ability of the text modality, we proposed the SKL module, which aims to identify and model the complex relationships between entities in the text, extract key structured triple information, enhance the semantic expression of the text modality, and achieve in-depth learning of inter-word dependencies; by explicitly linking the aggregated text instance representation with triple knowledge, a richer and more expressive text representation is generated;

[0084] Traditional text processing methods usually process raw unstructured text directly, often facing the challenges of information redundancy and difficulty in accurately grasping the core semantics; unstructured text usually contains a lot of non-core information, which may interfere with the model's understanding of the central idea of ​​the text; in contrast, the structured knowledge auxiliary module we proposed adopts an entity relationship extraction strategy to extract (subject, relation, object) triple information from the text; this method effectively condenses the text semantics into key semantic units, which helps the model accurately grasp the complex structure between entities and reduce the interference of redundant information;

[0085] To achieve this goal, we adopt an end-to-end relation extraction model REBEL, which uses the BART large-scale pre-trained language model as the infrastructure to transform the relation extraction task into a generation task; specifically, it takes the original text as input and outputs a linearized triple sequence T tri =<subject,relation,object> ; This process can be represented by the following probability model:

[0086]

[0087] Among them, S represents the input sentence, T tri represents the generated triples; represents the i-th element of a triple (subject, relation, or object), and Indicates the elements that have been generated before generating the i-th element; p BART (T tri |S) means that given an input sentence S, the REBEL model generates a triple T tri The model generates each triplet element sequentially through conditional probability, ensuring that the context of previously generated elements is taken into account during the generation process; this approach improves the coherence and accuracy of generated triples;

[0088] The structured representation method transforms complex text into fine and information-dense semantic units; this helps to establish correspondence with salient visual elements in the image, thereby improving cross-modal understanding and matching performance; Table 1 provides specific sentence examples to intuitively illustrate the triple parsing process;

[0089] Knowledge-enhanced encoder: The triplet structured phrases obtained by entity relationship extraction. In order to infer instance-level structured embeddings from these structured triples, we proposed a knowledge-enhanced encoder, which explicitly models the extracted core semantics as the input of the graph convolution module, i.e., the triplet information of subject, relation, and object. This module introduces high-order neighborhood information between triples through the graph structure to simulate deeper semantic associations between instances. Finally, in order to obtain a more comprehensive and rich text representation, we adopted a fusion strategy to integrate the global embedding of the text with the structured knowledge embedding.

[0090] In order to express the triple sequence concisely and intuitively, we adopt the following notation system: Let T tri =

[0091] h i ,r i ,t i )|i∈[1,m] represents the set of structured triples extracted from the text, where h i 、r i and t i Represent the head entity, relation entity, and tail entity of the triple respectively; for each structured triple phrase extracted from the text, we use BERT-base to obtain the embedding of each entity: and

[0092] e x =TripleEmb(x),x∈h,r,t.#(14)

[0093] To deeply capture the intrinsic correlation of structured semantics between instances, we introduced a graph convolutional network (GCN) as the core architecture. In this network structure, each node represents a set of triple entities, and the embedding representation of the node is optimized through an iterative neighborhood information aggregation process, thereby more effectively modeling the correlation of structured semantics between instances.

[0094] The core of this framework is the construction method of the neighborhood semantic association graph of text instances. We choose to use the global representation of text rather than directly using the features of triple entity sets to construct the similarity matrix, thereby generating a more effective semantic association graph. Although the structured triple representation can effectively capture the core semantic relationship, it may ignore some subtle but important semantic information. In contrast, the global representation of text retains a more comprehensive and detailed text semantic structure.

[0095] This method effectively combines the similarity information of global text semantics with the structured triple representation, realizing the organic combination of micro-structured information and macro-semantic features. In the specific implementation, we use cosine similarity to quantify the semantic similarity between text instances and use graph convolutional networks to optimize the structured triple representation. These two key steps can be expressed by the following formulas:

[0096]

[0097] in, and Text embeddings representing aggregations;

[0098]

[0099] here, Represents the structured semantic representation of text at level l, A semantic association graph representing the neighborhood of a text instance, is the learnable weight matrix of layer l, σ(·) refers to the nonlinear activation function (we use ReLU), and is the diagonal element of the degree matrix; after optimization, the embedding representation of each entity is recorded as and

[0100] To obtain the triplet representation of each entity embedding, we adopt the following encoding function:

[0101]

[0102] The knowledge-enhanced encoder integrates structured triples into entity relationship embedding, effectively capturing high-order correlation information between entities;

[0103] However, relying solely on the global representation of text for cross-modal alignment may ignore this structural knowledge; to this end, we propose a method that fuses global embedding and structural knowledge embedding; specifically, we use the following formula to integrate these two embeddings:

[0104]

[0105] in represents the comprehensive text representation, η∈[0,1] is the knowledge integration coefficient, which is used to balance the contribution of the two embeddings; represents global text embedding, capturing the overall semantic information of the text, and m i represents the knowledge embedding extracted from the structured triples;

[0106] Table 1 Triple representation of entity relationship extraction and parsing

[0107]

[0108] S4. Introducing a cross-modal semantic alignment mechanism: Using an asynchronous method to explore deep interactions between visual and textual embedding vectors across modalities and within modalities; specifically:

[0109] An asynchronous method is used to explore the deep interactions between and within modalities of visual and textual embedding vectors; the process includes three consecutive stages: first, a multi-head cross-attention mechanism is used to capture the relational interactions within different modalities and effectively integrate visual and textual information; second, the semantic dependencies between modalities are processed through a joint modality multi-head self-attention mechanism to enrich the representation ability of the joint modality; finally, the relations obtained in the first two stages are nonlinearly transformed through a feedforward neural network to finally obtain an enhanced visual-text representation. and m is the batch size;

[0110] To accelerate convergence and improve the quality of learned embeddings, we adopt the widely used hard negative triplet loss; we introduce triplet loss and combine it with hard negative sample mining strategy to enhance the model's discriminative ability and generalization performance;

[0111]

[0112] Here, α represents the boundary hyperparameter, and s(·,·) is used to compute the cosine similarity between visual and textual embeddings; [·] + ≡max(·,0); for a positive image-text pair (v,t), find the hardest negative image v in the batch - = argmax x≠v s(x,t) and negative text t - = argmax y≠t s(v,y), construct negative image-text pairs and image-negative text pairs, i.e. (v - ,t) and (v,t - ); We use distance weighted sampling for hard negative mining;

[0113] To effectively learn cross-modal representations, we introduce a triplet loss on the original visual-text pairs and extend it to the enhanced visual-text embedding to achieve cross-learning and avoid sample-to-sample interaction in the inference phase; therefore, the total cross-modal alignment loss can be expressed as:

[0114]

[0115] S5. Introducing a multi-scale semantic association matrix: Constructing a multi-dimensional semantic association matrix based on a hierarchical matching paradigm from multiple perspectives; specifically:

[0116] In our study, we first calculated the similarity using the global representation of images and texts and constructed a global semantic association matrix S g As the basic retrieval matrix; although this method performs well in computational efficiency, it also has some inherent limitations; specifically, this non-parametric method (such as applying distance metrics such as cosine similarity) lacks learnable parameters, resulting in low sensitivity to data distribution, making the model mainly focus on the macro-semantic alignment between global feature vectors, while it is difficult to capture and utilize finer-grained semantic correspondences;

[0117] To this end, we introduce a novel multi-dimensional semantic association matrix, constructed based on a hierarchical matching paradigm from multiple perspectives, such as Figure 4 As shown in the figure, taking the Flickr30K dataset as an example, given an image and its corresponding text description, a single image can be parsed from multiple angles to obtain multiple independent but complementary view features. For example, the green view may focus on capturing "people" and "sitting on the sidewalk", while the red view focuses on elements such as "canal" and "ships docked in the canal". This multi-dimensional parsing method can focus on different scene fragments and overcome the problem of semantic information omission that may occur in a single view.

[0118] Figure 4 Multi-view hierarchical matching strategy; the above image can be divided into three overlapping regions, each corresponding to an independent perspective; accordingly, the text description is parsed into different scenes, and a hierarchical matching relationship is established between each perspective;

[0119] This method breaks through the limitations of traditional cross-modal alignment and is no longer limited to static one-dimensional correspondence. Traditional methods mainly focus on the one-dimensional mapping between image regions and text tags, which may ignore the complex semantic associations across multiple perspectives. In contrast, the proposed semantic association matrix has the dual characteristics of dynamics and multidimensionality, which enables it to adaptively capture and learn multi-dimensional enhanced correspondences.

[0120] The dimension of the measurement unit is expanded from a single level to a multi-level structure, which enriches and enhances the information granularity of the measurement unit, thereby improving the retrieval accuracy; by comprehensively utilizing the two instance-level similarity matrices (the efficient global semantic association matrix and the accurate multi-dimensional semantic association matrix), the image-text matching effect is enhanced;

[0121] Inspired by the work of ESL, we use the adaptive dimension selection algorithm proposed by them to perform hierarchical iterative aggregation on the segment-level representation to obtain a dynamic multi-dimensional enhanced representation; the details are as follows:

[0122] For the initial image representation We apply a hierarchical iterative aggregator to obtain a multi-level collection of visual representations: V H = {Vl |l=1,2,…,L}, where And d l =d / 2 (l-1) ; Similarly, for the text modality, we obtain a multi-level text representation set:

[0123] T H ={T l |l=1,2,…,L}, where And d l =d / 2 (l-1) ; k and n t denote the number of image and text segments, d is the dimension of representation, and L is the total number of levels; d l Represents the feature dimension of the lth layer, which decays exponentially with the increase of the layers;

[0124] Using the obtained multi-level representations of the two modalities, we aim to fully align all their correspondences at each level and jointly learn cross-modal semantic similarity; based on the multi-dimensional enhanced representation V H and T H , the semantic similarity of all word region pairs between V and T for:

[0125]

[0126] where α l represents the learnable contribution score of the lth layer, and represents the weight matrix of the lth layer;

[0127] Based on this semantic similarity matrix, we can infer the visual text instance-level relevance from both image-to-text (I2T) and text-to-image (T2I) directions; in I2T, each region is used as a query, and for the i-th region, its relevance score is It can be calculated by aggregating its semantic similarity with all words:

[0128]

[0129] represents the semantic similarity between the ith region and all words, is the corresponding attention weight obtained by softmax, and λ is a learnable softmax temperature-sensitive factor; in T2I, the process of each word as a query is symmetrical, as shown below:

[0130]

[0131] Finally, the overall visual-textual relevance is inferred through the scores of all region-word pairs:

[0132]

[0133] Here, S g Focusing on the overall semantic structure, S l It accurately describes the local semantic correspondence; in order to make full use of these two complementary semantic representations, we unify them in the same semantic space through mean pooling; the final retrieval matrix is ​​defined as S f :

[0134]

[0135] In order to optimize the cross-modal semantic association matrix obtained by multi-level iterative aggregation, we introduce a marginal triplet ranking loss based on the most difficult sample as the objective function, defined as follows:

[0136]

[0137] where γ represents the marginal hyperparameter, S(·,·) is used to calculate the cosine similarity of visual-language embeddings, and [·] + ≡max(·,0); (I,T) is a positive image-text pair;

[0138] Find the hardest negative image I in the batch respectively - = argmax p≠I S(p,T) and the most difficult negative text T - = argmax q≠T S(I,q).

[0139] Objective function:

[0140] The total loss of the entire model is defined as follows:

[0141]

[0142] Experimental comparison:

[0143] Dataset:

[0144] We evaluate the proposed method on two widely used public datasets: Flickr30K and MS-COCO. Each image of these two datasets is associated with five text descriptions. Flickr30K: This dataset contains 31,000 images.

[0145] We use a standard data split, using 29,000 images for training, 1,000 images for validation, and another 1,000 images for testing. MS-COCO: This is a larger dataset with a total of 123,287 images.

[0146] We split it into 113,287 training images, 5,000 validation images, and 5,000 test images. For MS-COCO, we report two test results: performance on the full 5K test set, and the average performance of 5-fold cross validation on the 1K test subset.

[0147] Evaluation Metrics: To evaluate the performance of our proposed method, we use Recall@K (R@K, where K = 1, 5, 10), which is widely used in image-text matching tasks, as the main evaluation metric, which reflects the proportion of correct matches in the top K candidates of the retrieval results.

[0148] In order to comprehensively evaluate the bidirectional retrieval performance of the model, we introduced a comprehensive indicator Rsum, which is defined as the sum of all R@K values ​​in both directions, as shown below:

[0149]

[0150] Implementation details:

[0151] Our model uses FasterR-CNN (based on the ResNet-101 backbone network) and is pre-trained on the VisualGenome dataset to extract image region features. Specifically, we select K=36 regions with the highest confidence, and the dimension of each region feature is 2,048.

[0152] We use the pre-trained BERT-base model to extract 768-dimensional token features for the text modality. The dimension d of the joint embedding space is set to 1,024.

[0153] In order to enhance the semantic representation capability of text modality, we introduce the REBEL relation extraction model to extract key triple information.

[0154] Subsequently, we use the BERT-base model to independently encode entity sets to further enrich the text representation.

[0155] In the training phase, we use the Adam optimizer with an initial learning rate of 5e-4. The batch sizes of the Flickr30K and MS-COCO datasets are set to 128 and 256, respectively. We use a step-by-step decay strategy to reduce the learning rate by a factor of 10 every 15 epochs.

[0156] In addition, the knowledge integration coefficient λ is set to 0.1, and the hyperparameters γ and α of the margin loss are set to 0.2 respectively.

[0157] Table 2. Comparison with state-of-the-art methods on the Flickr30K test set. * indicates the ensemble results of two models, and the best result is shown in bold.

[0158]

[0159]

[0160] Table 3 Comparison with the most advanced methods on the MSCOCO1k and MSCOCO5k test sets. * indicates the combined results of the two models, and the best results are shown in bold

[0161]

[0162]

[0163] Table 4 Ablation experiment

[0164]

[0165] Table 5 Performance comparison of models evaluated on the Flickr30K test set and trained on MSCOCO.

[0166]

[0167]

[0168] Comparison with state-of-the-art methods:

[0169] To verify the effectiveness of our proposed method, we compare our model with the state-of-the-art baseline methods on two public benchmark datasets. These benchmarks include mainstream techniques in image-text matching, including global matching methods and local matching methods.

[0170] The global matching methods include SAEM, CVSE, GPO, MKVSE, USER, HREM, CORA, VSE_2AD, MV-VSE, UARDA, MEMBER, SGRAF, and CODER. The local matching methods include SCAN, IMRAM, GSMN, VSRN++, NAAF, ESL, and CHAN. Among them, GPO and VSE_2AD represent fine-grained feature fusion methods, while MKVSE and CORA utilize knowledge-assisted techniques. All these methods are evaluated on the same test data as our proposed method, ensuring the fairness and reliability of the comparison.

[0171] The full-granularity matching method we proposed combines the advantages of global and local matching methods. It constructs two complementary semantic association matrices: a global semantic association matrix based on global features and a multi-dimensional semantic association matrix based on fine-grained features. This enables the model to capture both macro-semantic relationships and micro-semantic details, ultimately generating a more accurate retrieval matrix.

[0172] Tables 2 and 3 list the quantitative results of our method on two datasets. These results reflect the performance of our method and provide an intuitive comparison with the existing state-of-the-art methods.

[0173] Comparison on Flickr30K: Table 2 shows the comparison of our method with the state-of-the-art algorithms for bidirectional retrieval on the Flickr30K dataset. Our single model achieves 85.8% and 63.8% R@1 performance in sentence and image retrieval, respectively, significantly surpassing existing cutting-edge methods. Compared with the closest competitor CORA, our method improves by 2.1% and 1.5% on R@1 and 5.1% on Rsum, respectively. In addition, we achieve 98.8% and 92.6% on R@10, surpassing all listed comparison methods. These results highlight the powerful ability of our model in handling larger candidate sets.

[0174] Comparison on MS-COCO: Table 3 shows the quantitative evaluation of PASE and all competing methods on the bidirectional retrieval task on the MS-COCO1K and 5K test sets. On the 1K test set, PASE achieves 83.0% and 67.2% R@1 in image-text and text-image retrieval, respectively, surpassing the previous best result with a 1.7% Rsum improvement. On the more challenging 5K test set, PASE's advantage is even more obvious: R@1 in image-text and text-image retrieval reaches 64.4% and 45.0%, respectively, surpassing existing methods. Compared with the current leading methods ESL and CORA, PASE improves Rsum by 3.5% and 6.2%, respectively. These results highlight PASE's superior ability in exact matching and demonstrate its effectiveness and robustness in dealing with large-scale complex datasets.

[0175] Ablation experiment:

[0176] To verify the effectiveness of the PASE framework, we conducted a series of ablation studies to fully explore the impact of different network components on image-text matching performance and training process. In Table 4, “√” means retention, otherwise it means deletion. The best results are shown in bold.

[0177] Different network structures: We conducted a series of comparative experiments on the Flickr30K dataset to evaluate the retrieval performance by building different model structures. To simplify the description, we abbreviate the four core modules as follows: HAP (Hierarchical Self-Enhanced Semantic-Aware Pooler), SKL (Structured Knowledge Assisted Learning Module), CSA (Cross-Modal Semantic Alignment Mechanism), and MSAM (Multi-Scale Semantic Association Matrix). These modules represent the key components of our method.

[0178] Effectiveness of the aggregation module: As shown in Table 4, the comparison between Experiments VI and VII reveals the importance of the HAP module. When traditional max pooling is used instead of HAP, the Rsum of the model decreases by 1.6%. Specifically, after the introduction of HAP, the R@1 indicators of image-to-text (I→T) and text-to-image (T→I) are improved from 83.3% and 63.9% to 84.7% and 64.5%, respectively. This shows that HAP, as an efficient adaptive pooling strategy, significantly enhances the model's ability to perceive fine-grained semantics and effectively integrates local representations.

[0179] Effectiveness of structured knowledge extraction: Learning structured relation graphs is crucial to improving the performance of image and sentence matching. To explore this point in depth, we further compare the results of Experiments V and VII. As shown in Table 4, by incorporating structured relation information into the comprehensive text representation and using it as the input for cross-modal alignment through our proposed knowledge-enhanced encoder, the overall performance of the model is significantly improved.

[0180] After the introduction of SKL, Rsum increased from 522.2 to 524.3, a significant improvement of 2.1%. This significant performance improvement shows that SKL strengthens the language representation ability of text and provides richer and more precise semantic information for cross-modal alignment.

[0181] Effectiveness of cross-modal alignment: To quantify the effect of the CSA module, we compare the results of Experiments IV and VII. As shown in Table 4, after the introduction of CSA, the R@1 of image-to-text (I→T) and text-to-image (T→I) is improved from 82.9% and 63.5% to 84.7% and 64.5%, respectively, an increase of 1.8% and 1.0%, confirming the key role of interactive learning in the image-text matching task.

[0182] The effectiveness of CSA is mainly reflected in its ability to promote deep interaction and semantic alignment between modalities, more effectively learning and capturing the interactive information between modalities, thereby improving embedding learning.

[0183] Effectiveness of multi-scale semantic association matrix: The introduction of the MSAM module has a significant impact on the model performance. Comparing experiments VII and VIII, when only using the global semantic association matrix for image-text retrieval (VII), the R@1 of image to text (I→T) is 84.7%. After the introduction of MSAM (VIII), R@1 is improved by 1.1%, and Rsum increases from 524.3 to 525.2, an improvement of 0.9%. This improvement shows that the MSAM module achieves multi-level semantic expression capabilities through multi-dimensional quantity representation learning, enabling the model to simultaneously capture and integrate different levels of discriminative information, thereby promoting more accurate semantic similarity judgments.

[0184] Module Synergy: When analyzing the experimental results in depth, we found a synergistic effect in terms of module combination. When HAP or SKL is used independently (V and VI), the performance improvement is relatively limited. However, when the two modules are deployed simultaneously (VII), we observe a significant performance leap. This phenomenon reveals a complementary mechanism: there is a deep interaction between the core text semantics extracted by SKL and the global features aggregated by HAP. The SKL module captures the key semantic elements of the text, providing a richer and more precise semantic basis for the HAP module. At the same time, the global features aggregated by the HAP module provide a broader context for the core semantics extracted by SKL, enabling the model to understand the text semantics more comprehensively and deeply.

[0185] Based on the existing modular architecture, the introduction of MSAM (VIII) achieves a significant performance leap. The module's ability to capture subtle semantic associations addresses the limitations of the global semantic association matrix in fine-grained matching. This advantage is particularly evident in image-to-text retrieval tasks, as the rich visual information contained in images requires more precise semantic correspondence. It also demonstrates its excellent ability to integrate the output of previous modules, cleverly realizing a multi-granular, multi-dimensional semantic matching mechanism.

[0186] Hyperparameter Analysis: We investigated the impact of the iterative aggregation structure on performance, focusing on the impact of different iterative layers l on retrieval results. Figure 5 Using the data from , we observe that the retrieval performance improves steadily with the increase in the number of layers. This improvement is attributed to the fact that the increased hierarchical structure provides richer discriminative information for cross-modal semantic similarity learning. Specifically, the multi-layer structure enables the model to capture and integrate semantic information at different levels, resulting in a more comprehensive and accurate similarity measurement.

[0187] In addition, to quantify the impact of the knowledge integration coefficient η on the model performance, we conducted parameter sensitivity analysis, such as Figure 6 The experimental results show that when η is set to 0.1, the model achieves the best performance and the fluctuation of the performance curve is small, indicating that the model is less sensitive to the parameter η and has strong robustness.

[0188] Figure 5 Parameter analysis of the number of iteration layers l in Equation 21 on Flickr30K.

[0189] Figure 6 Parameter analysis of the knowledge integration coefficient η in Equation 18 on Flickr30K.

[0190] Further analysis:

[0191] Generalization: We conducted cross-dataset transfer experiments to evaluate the generalization ability of the model. We first trained the model on the MS-COCO dataset and then conducted zero-shot testing on the Flickr30K dataset. As shown in Table 5, our model significantly outperforms the existing baseline model in this transfer task. This result not only verifies the universal representation ability of our model, but also highlights its robustness in the face of heterogeneous data distributions.

[0192] Summarize:

[0193] A visual semantic adaptive embedding method enhanced by multimodal structured knowledge is proposed for image-text retrieval tasks, that is, a multimodal structured knowledge graph is integrated into image-text retrieval based on adaptive embedding. Specifically, two fragment-level semantic pooling structures are first designed, which can adaptively learn the optimal feature fusion strategy according to the input image and text features, thereby generating a highly discriminative global representation. In addition, an entity relationship extraction strategy is introduced to refine the core semantics of the text, and the strong relationship of the graph convolutional network is combined to further strengthen the structured representation, injecting key concepts and relationship information into the global representation. In order to further improve the matching accuracy, a similarity matrix integrated by hierarchical multi-dimensionality is also constructed. Extensive experiments on two widely used benchmark datasets, MS-COCO and Flickr30K, verify the superiority of our framework. In future work, we tend to use this framework for other vision and language tasks, such as image captioning and text video retrieval.

Claims

1. A graph-text matching method based on progressive semantic aggregation and structured cognitive enhancement, characterized in that: The following steps are involved: S1. Multimodal feature extraction: including region-aware image representation and context-sensitive text word encoding, extracting visual region or text word features as input; S2. Introducing a hierarchical self-enhanced semantic-aware pooling mechanism: Aggregating visual region or text word features to generate global feature embedding through self-guided enhancement modules and semantic-aware modules; S3, introduces structured knowledge-assisted learning module: It includes three key steps to construct a structured representation rich in semantic information: extracting triple phrases, building a neighborhood semantic association graph, and generating instance-level structured embedding to inject prior knowledge into the global semantic embedding; S4. Introducing a cross-modal semantic alignment mechanism: Using an asynchronous method to explore deep interactions between and within modalities of visual and textual embedding vectors; S5. Introducing multi-scale semantic association matrix: Constructing a multi-dimensional semantic association matrix based on the hierarchical matching paradigm of multiple perspectives.

2. The image-text matching method based on progressive semantic aggregation and structured cognitive enhancement as claimed in claim 1, characterized in that: In region-aware image representation, given an image, a bottom-up attention network is used to extract the top K salient regions using Faster-RCNN and ResNet- 101 obtains regional features, combines a multi-layer perceptron and a fully connected layer to convert the extracted regional features into a specific dimension; in context-sensitive text word encoding, given a text containing N words, the BERT model is used to mark the input text into subword units, and a fully connected layer is applied to convert the extracted text word features into a specific dimension.

3. The image-text matching method based on progressive semantic aggregation and structured cognitive enhancement as claimed in claim 2, characterized in that: In S2, a dimension-based sorting method is used to sort the input features, and the self-guided enhanced pooling operator extracts significant elements from the sorted feature vector. SEPO overcomes the limitations of traditional maximum pooling and K-means pooling by adopting a smooth weight assignment mechanism. In the semantic perception module, the self-enhanced global features are used as the query matrix, and the original sorted features are input into the self-attention mechanism as keys and values. The aggregation coefficient matrix learned by the attention mechanism is used to adaptively quantify the semantic relevance between each fragment-level representation and the self-guided enhanced global representation to generate a semantically aware weight vector. The adaptive weight learning method of the gating mechanism is used to fuse the two global feature vectors to obtain the final visual aggregate embedding and text aggregate embedding.

4. The image-text matching method based on progressive semantic aggregation and structured cognitive enhancement as claimed in claim 3, characterized in that: In S3, an end-to-end relation extraction model REBEL is used, which takes the original text as input and outputs a linearized triple sequence, transforming the relation extraction task into a generation task. In the knowledge-enhanced encoder, BERT- The base obtains the embedding of each entity and introduces a graph convolutional network to optimize the embedding representation of the node through an iterative neighborhood information aggregation process. The global embedding of the text is integrated with the structured knowledge embedding, and a specific formula is used to integrate the two embeddings.

5. The image-text matching method based on progressive semantic aggregation and structured cognitive enhancement as claimed in claim 4, characterized in that: In S4, a multi-head cross-attention mechanism is used to capture the relational interactions within different modalities, and the semantic dependencies between modalities are processed through a joint-modality multi-head self-attention mechanism. The relations obtained in the first two stages are nonlinearly transformed through a feedforward neural network. The hard negative triplet loss is combined with the hard negative sample mining strategy, and the triplet loss is introduced and extended to enhanced visual-text embedding.

6. The image-text matching method based on progressive semantic aggregation and structured cognitive enhancement as claimed in claim 5, characterized in that: In S5, we first use the global representation of images and texts to calculate the similarity and construct a global semantic association matrix. Then, we introduce a multidimensional semantic association matrix based on a hierarchical matching paradigm from multiple perspectives, perform hierarchical iterative aggregation on the fragment-level representation, and obtain a dynamic multidimensional enhanced representation. Based on the multidimensional enhanced representation, we calculate the semantic similarity of word region pairs, infer the instance-level correlation of visual text from two directions, unify different semantic representations in the same semantic space through mean pooling, and introduce a marginal triplet ranking loss based on the most difficult sample to optimize the multidimensional semantic association matrix.

7. The image-text matching method based on progressive semantic aggregation and structured cognitive enhancement as claimed in claim 1, characterized in that: The total loss of the entire model is defined by a specific formula; in the experimental comparison, it is evaluated on two public datasets, using specific evaluation indicators, setting specific implementation details, and comparing with the most advanced methods. Ablation experiments are performed to verify the effectiveness of the framework, including analyzing different network structures, namely aggregation modules, structured knowledge extraction, cross-modal alignment, the effectiveness of multi-scale semantic association matrices, and module synergy.

8. The image-text matching method based on progressive semantic aggregation and structured cognitive enhancement as claimed in claim 7, characterized in that: We also conduct hyperparameter analysis including the impact of the number of iterative aggregation structure layers and the knowledge integration coefficient, and perform cross-dataset migration experiments to evaluate the generalization ability of the model.

Images