Cross-modal alignment method and device for multi-modal large language model

Abstract

The invention discloses a cross-modal alignment method and device for a multi-modal large language model, and relates to the technical field of multi-modal alignment, and the method comprises the steps: obtaining an image, carrying out the target detection, and generating a local image fine-grained feature; obtaining a text, extracting text features, extracting a text entity relationship, and obtaining an entity relationship dependency tree as a text entity relationship graph; performing local fine-grained semantic alignment on the local image fine-grained features and the text features; using the scene graph model to obtain a visual relation graph between detection targets in the image; performing graph structure alignment and graph semantic alignment on the text entity relation graph and the visual relation graph; and fusing the semantic alignment features and the graph alignment features to obtain semantic alignment features with rich fine grit, global semantic alignment features and richer relationship alignment features, and inputting the semantic alignment features, the global semantic alignment features and the richer relationship alignment features into a multi-modal large model. According to the method, alignment of different modal data on different feature dimensions is realized, so that the different modal data are accurately associated.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal alignment, and in particular to a cross-modal alignment method and device for a multimodal large language model. Background Art

[0002] Multimodal Large Language Models (MLLM) is an important research direction in the field of artificial intelligence in recent years. It is a large language model that can process and understand multiple types of data (such as text, images, audio, etc.). It not only has the language understanding and generation capabilities of traditional language models, but also can perform more complex and diverse tasks by integrating information from different modalities.

[0003] With the rapid development of artificial intelligence technology, multimodal large language models have become a research hotspot. Multimodal large language models can understand and produce data in multiple modes such as text, images, and audio, providing strong technical support for applications such as intelligent dialogue, intelligent creation, and intelligent recommendation.

[0004] Multimodal data refers to the same object described in different ways (such as perspectives or fields), and each description is called a modality. These data contain multiple modalities such as text, images, audio, and video, and each modality has its own unique representation and semantic information. Multimodal alignment refers to finding instances that point to unified concepts in the real world in different data sources or knowledge bases. In multimodal data, instances may appear in different forms, such as named entities in text, objects or scenes in images, etc. Therefore, the goal of multimodal alignment is to find representations pointing to the same instance in data of different modalities. Multimodal alignment can associate data of different modalities to achieve cross-modal information retrieval, improve the semantic understanding ability of multimodal data, and provide support for tasks such as intelligent question answering and sentiment analysis.

[0005] Multimodal alignment is generally divided into explicit alignment and implicit alignment. The goal of explicit alignment is to find the modal relationship in the data, which is applied to the alignment of speech signals and translated texts, the synchronization of multi-channel sequences, and the positioning of images / videos. Implicit alignment uses the implicit alignment inherent in the modality to improve the performance of the algorithm, which is applied to cross-modal retrieval, automatic visual description generation, and visual question answering.

[0006] The challenges faced by multimodal alignment are as follows: (1) the length of data in different modalities varies greatly; (2) there are semantic (information content) differences between data in different modalities.

[0007] Multimodal alignment includes temporal alignment and semantic alignment. Traditional multimodal alignment is based on the definition of rules to establish the correspondence between modalities; while learning-based alignment uses machine learning and deep learning algorithms to automatically discover and learn the correspondence between different modalities. In addition, multimodal alignment can also be completed using multimodal converters, multimodal sensors, external tool assistance, and data-driven methods. However, if the alignment method is not used properly, it may produce incorrect production results, with limited performance improvement, which in turn leads to high computing and usage costs.

[0008] Multimodal alignment is an area of ​​common concern in different research fields. How to accurately analyze and understand cross-modal data is a task that needs to be solved urgently and is also a milestone on the road to the development of artificial intelligence. Summary of the invention

[0009] In view of the defects in the semantic alignment of different modal data in current information technology applications, the present invention provides a cross-modal alignment method and device of a multimodal large language model, which realizes the alignment of different modal data in different feature dimensions and enables accurate association of different modal data.

[0010] In a first aspect, the present invention provides a cross-modal alignment method for a multimodal large language model. The technical solution adopted to solve the above technical problems is as follows:

[0011] A cross-modal alignment method for a multimodal large language model comprises the following steps:

[0012] S1, extract multimodal data features;

[0013] a) Acquire image data, perform target detection on the image data, and generate fine-grained features of local images;

[0014] b) obtaining text data, extracting text features, and extracting text entity relationships to obtain an entity relationship dependency tree as a text entity relationship graph;

[0015] S2, using the cross attention network, performs local fine-grained semantic alignment on the local image fine-grained features generated in step S1 and the extracted text features;

[0016] S3, using the scene graph model VCTree to obtain a visual relationship graph between the detection targets in the image data of step S1;

[0017] S4, aligning the text entity relationship graph obtained in step S1 and the visual relationship graph obtained in step S3, including graph structure alignment and graph semantic alignment;

[0018] S5. Fuse semantic alignment features and graph alignment features to obtain rich and fine-grained semantic alignment features, as well as global semantic alignment features and richer relation alignment features, and input them into the multimodal large model to complete related tasks.

[0019] Optionally, after executing step S1 and acquiring the image data, the FasterRCNN model is used to perform target detection on the image data to generate fine-grained features of the local image.

[0020] Optionally, execute step S1, after obtaining text data, use the BGE model to extract text features, and use the BERT model to extract text entity relationships.

[0021] Optionally, step S4 is executed to perform the process of graph structure alignment as follows:

[0022] The nodes in the text entity relationship graph and the visual relationship graph are converted into low-dimensional vector representations through graph neural networks or other graph embedding technologies, that is, the node representation vector features are obtained;

[0023] Calculate the similarity between nodes in the text entity relationship graph and the visual relationship graph;

[0024] Select N nodes from all nodes as reference points, calculate the similarity between N reference points and the remaining nodes, and generate a similarity matrix C;

[0025] The node pair with the highest similarity is selected from the similarity matrix C as the aligned nodes, that is, a pair of nodes with the most similar structures in the text entity relationship graph and the visual relationship graph is found.

[0026] Further optionally, step S4 is executed to perform graph semantic alignment, first using the guided attention mechanism to calculate the correlation between the words in the text and the detection targets in the image data to obtain the attention score, and then combining the similarity and attention score during graph structure alignment to calculate the final graph alignment feature.

[0027] In a second aspect, the present invention provides a cross-modal alignment device for a multimodal large language model. The technical solution adopted to solve the above technical problems is as follows:

[0028] A cross-modal alignment device for a multimodal large language model, comprising:

[0029] An image processing module is used to acquire image data, perform target detection on the image data, and generate fine-grained features of local images;

[0030] The text processing module is used to obtain text data, extract text features, and extract text entity relationships to obtain an entity relationship dependency tree as a text entity relationship graph;

[0031] The semantic alignment module is used to perform local fine-grained semantic alignment of local image fine-grained features and text features using a cross-attention network;

[0032] The relationship detection module is used to obtain the visual relationship graph between the detection targets in the image data using the scene graph model VCTree;

[0033] Graph alignment module, used to align the text entity relationship graph and the visual relationship graph in terms of graph structure and graph semantics;

[0034] The fusion input module is used to fuse semantic alignment features and graph alignment features to obtain rich and fine-grained semantic alignment features, as well as global semantic alignment features and richer relation alignment features, which are then input into the multimodal large model to complete related tasks.

[0035] Optionally, the image processing module involved acquires image data, performs target detection on the image data using a FasterRCNN model, and generates fine-grained features of local images.

[0036] Optionally, after the text processing module obtains the text data, it uses the BGE model to extract text features and uses the BERT model to extract text entity relationships.

[0037] Optionally, the graph alignment module involved includes a structural alignment unit and a semantic alignment unit;

[0038] The structural alignment unit performs graph structural alignment on the text entity relationship graph and the visual relationship graph. The specific alignment process is as follows:

[0039] The nodes in the text entity relationship graph and the visual relationship graph are converted into low-dimensional vector representations through graph neural networks or other graph embedding technologies, that is, the node representation vector features are obtained;

[0040] Calculate the similarity between nodes in the text entity relationship graph and the visual relationship graph;

[0041] Select N nodes from all nodes as reference points, calculate the similarity between N reference points and the remaining nodes, and generate a similarity matrix C;

[0042] Select the node pair with the highest similarity from the similarity matrix C as the aligned nodes, that is, find the pair of nodes with the most similar structure in the text entity relationship graph and the visual relationship graph;

[0043] The semantic alignment unit performs graph semantic alignment on the text entity relationship graph and the visual relationship graph. The specific alignment process is as follows:

[0044] First, the guided attention mechanism is used to calculate the correlation between the words in the text and the detection targets in the image to obtain the attention score;

[0045] Then, the similarity and attention score during graph structure alignment are combined to calculate the final graph alignment features.

[0046] Compared with the prior art, the cross-modal alignment method and device of a multimodal large language model of the present invention has the following beneficial effects:

[0047] The present invention enriches feature information by achieving semantic alignment of image targets and texts at a fine granularity, as well as structural alignment of image structures and semantic structures, effectively solving the problem of semantic alignment between data of different modalities in real information applications, and achieving fast and accurate multidimensional alignment in different dimensions, thereby improving the efficiency and accuracy of multimodal large models in processing multimodal information, enabling multimodal large models to process multidimensional and multimodal information in the real world more efficiently and intelligently. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Attached Figure 1 is a flow chart of a method according to Embodiment 1 of the present invention;

[0049] Attached Figure 2 This is a module link block diagram of the second embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the technical solution, the technical problem solved and the technical effect of the present invention more clearly understood, the technical solution of the present invention is clearly and completely described below in conjunction with specific embodiments.

[0051] Embodiment 1:

[0052] Combined with Figure 1 , this embodiment proposes a cross-modal alignment method of a multimodal large language model, which includes the following steps:

[0053] S1. Perform multimodal data feature extraction.

[0054] a) Obtain image data I, use the FasterRCNN model to perform target detection on image data I, and generate local image fine-grained features; this process is expressed as follows:

[0055]

[0056] In the formula, represents the characteristics of the detected target, and g represents the number of detected targets.

[0057] The FasterRCNN model is an efficient target detection model that makes the target detection process faster and more accurate by introducing the Region Proposal Network (RPN).

[0058] b) Obtain text data T, use the BGE model to extract text features, and use the BERT model to extract text entity relationships to obtain an entity relationship dependency tree as a text entity relationship graph. This process is expressed as follows:

[0059] E T =BGE(T),

[0060] RE T =BERT(T),

[0061] In the formula, E T Represents the extraction of text features, RE T Represents the extracted text entity relationship.

[0062] The BGE model, whose full name is BAAI General Embedding, supports more than 100 languages, can handle semantic matching tasks between different languages, and establish effective cross-language semantic connections; the BGE model integrates retrieval functions such as dense retrieval, sparse retrieval, and multi-vector retrieval, and can provide one-stop service for different semantic retrieval scenarios.

[0063] The BERT model, whose full name is Bidirectional Encoder Representations from Transformers, is a pre-trained language representation model based on the Transformer architecture. It is suitable for a variety of NLP tasks, such as text classification, question-answering systems, and named entity recognition.

[0064] S2. Use the cross attention network to perform local fine-grained semantic alignment on the local image fine-grained features generated in step S1 and the extracted text features. This process is expressed as follows:

[0065]

[0066] In the formula, Indicates the characteristics of the detected target; E T Indicates the extraction of text features; SCAN stands for the cross-attention mechanism, which can combine image features and text features to capture the fine-grained semantic relationship between the two; L(I,T) represents the local fine-grained semantic alignment result obtained by the cross-attention network.

[0067] S3, using the scene graph model VCTree to obtain the visual relationship graph between the detection targets in the image data I described in step S1, this process is expressed as follows:

[0068]

[0069] In the formula, Represents the features of the detected target; VCTree represents the scene graph model, which can generate a visual relationship graph based on the detected target features; V(I) represents the visual relationship graph obtained by the VCTree model, which describes the spatial and semantic relationships between the targets in the image.

[0070] The scene graph model VCTree is a deep learning model based on a dynamic tree structure, which is used to capture objects and their relationships in images to generate high-quality scene graphs.

[0071] S4. Align the text entity relationship graph obtained in step S1 and the visual relationship graph obtained in step S3, including graph structure alignment and graph semantic alignment.

[0072] S4.1. The process of graph structure alignment is as follows:

[0073] S4.1.1. Convert the nodes in the text entity relationship graph and the visual relationship graph into low-dimensional vector representations through graph neural networks or other graph embedding techniques, that is, obtain node representation vector features (node ​​embedding);

[0074] S4.1.2. Calculate the similarity between nodes in the text entity relationship graph and the visual relationship graph. The formula is as follows:

[0075]

[0076] Where sim(v,u) represents the similarity between node v in the text entity relationship graph and node u in the visual relationship graph; γ s. is a scaling factor or adjustment parameter used to control the sensitivity or scale in the similarity calculation; d v represents the degree distribution vector of node v in the text entity relationship graph, d u Represents the degree distribution vector of node u in the visual relationship graph; represents the square of the Euclidean distance between the degree distribution vectors of node v and node u; exp represents the exponential function used to convert the distance into a similarity score;

[0077] S4.1.3. Select N nodes from all nodes as reference points, calculate the similarity between the N reference points and the remaining nodes, and generate a similarity matrix C; select the node pair with the highest similarity from the similarity matrix C as the aligned nodes, that is, find a pair of nodes that are most similar in structure in the text entity relationship graph and the visual relationship graph. This process is expressed as follows:

[0078]

[0079] In the formula, node V(I)[i] represents the i-th node in the image data I, node R(T) [j] represents the jth node in the text data T; sim(node V(I) [i],node R(T) [j]) is a function that calculates the similarity between node v and node u. The function obtains the similarity score by calculating the negative exponential of the Euclidean distance between the embeddings of these two nodes; α represents a similarity matrix, α ij Represents the similarity between the i-th node in the text entity relationship graph and the j-th node in the visual relationship graph; |node I |×|node T | represents the size of the similarity matrix α, |node I | is the number of nodes in the text entity relationship graph, |node T | is the number of nodes in the visual relationship graph;

[0080] S4.2, when performing graph semantic alignment, first use the guided attention mechanism to calculate the correlation between the words in the text and the detection targets in the image data to obtain the attention score, and then combine the similarity and attention score during graph structure alignment to calculate the final graph alignment feature. This process involves the following formula:

[0081] K=W k E T +b k ,

[0082]

[0083] V=W v E T +b v ,

[0084]

[0085] Y(I,T)=(α+β)V,

[0086] Where K represents the key matrix converted from the node features in the text entity relationship graph, Q represents the query matrix converted from the node features in the visual relationship graph, V represents the value matrix converted from the node features in the text entity relationship graph, and W k , W q , W v represents the weight matrix, b k 、b q 、b v represents the bias term, E T Represents the node features in the text entity relationship graph, represents the node features in the visual relationship graph; β represents the attention score matrix, which is used to reflect the correlation between the words in the text and the detection targets in the image data; is a scaling factor used to adjust the result of the dot product; α represents the similarity matrix when the graph structure is aligned, which is obtained by calculating the similarity between the node features in the visual relationship graph and the node features in the text entity relationship graph; Y(I,T) represents the fused graph alignment feature, which integrates the local and global semantic information between image data and text data, as well as the correlation between them, providing a rich feature representation for subsequent multimodal tasks

[0087] S5, integrate semantic alignment features and graph alignment features to obtain rich and fine-grained semantic alignment features, as well as global semantic alignment features and richer relation alignment features, and input them into the multimodal large model to complete related tasks. This process is expressed as follows:

[0088] F(I,T)=[L(I,T):Y(I,T)],

[0089] Wherein, F(I,T) represents the fusion function, which is used to receive image data I and text data T as input and output the fused features; L(I,T) represents the local fine-grained semantic alignment features, namely the local semantic alignment features, which focus on the correspondence between local regions or entities in the image data I and the text data T; Y(I,T) represents the fused graph alignment features, namely the global semantic alignment features, which focus on the overall semantic consistency between the entire image data I and the text data T.

[0090] Multimodal Large Language Models (MLLMs) are an advanced artificial intelligence technology that can understand and generate multiple types of data, including text, images, audio, and video. MLLMs provide richer and more accurate understanding and generation capabilities by integrating information from different modalities.

[0091] Embodiment 2:

[0092] Combined with Figure 2 This embodiment proposes a cross-modal alignment device for a multimodal large language model, which includes:

[0093] The image processing module is used to obtain image data, perform target detection on the image data using the FasterRCNN model, and generate fine-grained features of local images;

[0094] The text processing module is used to obtain text data, extract text features using the BGE model, and extract text entity relationships using the BERT model to obtain an entity relationship dependency tree as a text entity relationship graph;

[0095] The semantic alignment module is used to perform local fine-grained semantic alignment of local image fine-grained features and text features using a cross-attention network;

[0096] The relationship detection module is used to obtain the visual relationship graph between the detection targets in the image data using the scene graph model VCTree;

[0097] Graph alignment module, used to align the text entity relationship graph and the visual relationship graph in terms of graph structure and graph semantics;

[0098] The fusion input module is used to fuse semantic alignment features and graph alignment features to obtain rich and fine-grained semantic alignment features, as well as global semantic alignment features and richer relation alignment features, which are then input into the multimodal large model to complete related tasks.

[0099] In this embodiment, the graph alignment module involved includes a structural alignment unit and a semantic alignment unit.

[0100] The structure alignment unit involved performs graph structure alignment on the text entity relationship graph and the visual relationship graph. The specific alignment process is as follows:

[0101] The nodes in the text entity relationship graph and the visual relationship graph are converted into low-dimensional vector representations through graph neural networks or other graph embedding technologies, that is, the node representation vector features are obtained;

[0102] Calculate the similarity between nodes in the text entity relationship graph and the visual relationship graph;

[0103] Select N nodes from all nodes as reference points, calculate the similarity between N reference points and the remaining nodes, and generate a similarity matrix C;

[0104] The node pair with the highest similarity is selected from the similarity matrix C as the aligned nodes, that is, a pair of nodes with the most similar structures in the text entity relationship graph and the visual relationship graph is found.

[0105] The semantic alignment unit involved performs graph semantic alignment on the text entity relationship graph and the visual relationship graph. The specific alignment process is as follows:

[0106] First, the guided attention mechanism is used to calculate the correlation between the words in the text and the detection targets in the image to obtain the attention score;

[0107] Then, the similarity and attention score during graph structure alignment are combined to calculate the final graph alignment features.

[0108] It is necessary to add that:

[0109] The FasterRCNN model is an efficient target detection model that makes the target detection process faster and more accurate by introducing the Region Proposal Network (RPN).

[0110] The BGE model, whose full name is BAAI General Embedding, supports more than 100 languages, can handle semantic matching tasks between different languages, and establish effective cross-language semantic connections; the BGE model integrates retrieval functions such as dense retrieval, sparse retrieval, and multi-vector retrieval, and can provide one-stop service for different semantic retrieval scenarios.

[0111] The BERT model, whose full name is Bidirectional Encoder Representations from Transformers, is a pre-trained language representation model based on the Transformer architecture. It is suitable for a variety of NLP tasks, such as text classification, question-answering systems, and named entity recognition.

[0112] The scene graph model VCTree is a deep learning model based on a dynamic tree structure, which is used to capture objects and their relationships in images to generate high-quality scene graphs.

[0113] Multimodal Large Language Models (MLLMs) are an advanced artificial intelligence technology that can understand and generate multiple types of data, including text, images, audio, and video. MLLMs provide richer and more accurate understanding and generation capabilities by integrating information from different modalities.

[0114] In summary, the cross-modal alignment method and device of a multimodal large language model of the present invention enriches the feature information by realizing the semantic alignment of image targets and texts at a fine granularity, as well as the structural alignment of image structures and semantic structures, and effectively solves the semantic alignment problem between different modal data in real information applications, and realizes fast and accurate multidimensional alignment in different dimensions, thereby improving the efficiency and accuracy of the multimodal large model in processing multimodal information, so that the multimodal large model can process multidimensional and multimodal information in the real world more efficiently and intelligently.

[0115] The above specific examples are used to explain the principles and implementation methods of the present invention in detail. These examples are only used to help understand the core technical content of the present invention. Based on the above specific embodiments of the present invention, any improvements and modifications made by technicians in this technical field without departing from the principles of the present invention should fall within the scope of patent protection of the present invention.

Claims

1. A cross-modal alignment method for a multimodal large language model, characterized in that: The steps include: S1, extract multimodal data features; a) Acquire image data, perform target detection on the image data, and generate fine-grained features of local images; b) obtaining text data, extracting text features, and extracting text entity relationships to obtain an entity relationship dependency tree as a text entity relationship graph; S2, using the cross attention network, performs local fine-grained semantic alignment on the local image fine-grained features generated in step S1 and the extracted text features; S3, using the scene graph model VCTree to obtain a visual relationship graph between the detection targets in the image data of step S1; S4, aligning the text entity relationship graph obtained in step S1 and the visual relationship graph obtained in step S3, including graph structure alignment and graph semantic alignment; S5. Fuse semantic alignment features and graph alignment features to obtain rich and fine-grained semantic alignment features, as well as global semantic alignment features and richer relation alignment features, and input them into the multimodal large model to complete related tasks.

2. The cross-modal alignment method of a multimodal large language model according to claim 1, characterized in that: Execute step S1, after acquiring the image data, use the FasterRCNN model to perform target detection on the image data and generate local image fine-grained features.

3. The cross-modal alignment method of a multimodal large language model according to claim 2, characterized in that: Execute step S1, after obtaining text data, use the BGE model to extract text features, and use the BERT model to extract text entity relationships.

4. The cross-modal alignment method of a multimodal large language model according to claim 1, characterized in that: Execute step S4 to perform graph structure alignment as follows: The nodes in the text entity relationship graph and the visual relationship graph are converted into low-dimensional vector representations through graph neural networks, that is, the node representation vector features are obtained; Calculate the similarity between nodes in the text entity relationship graph and the visual relationship graph; Select N nodes from all nodes as reference points, calculate the similarity between N reference points and the remaining nodes, and generate a similarity matrix C; The node pair with the highest similarity is selected from the similarity matrix C as the aligned nodes, that is, a pair of nodes with the most similar structures in the text entity relationship graph and the visual relationship graph is found.

5. The cross-modal alignment method of a multimodal large language model according to claim 4, characterized in that: Execute step S4. When performing graph semantic alignment, first use the guided attention mechanism to calculate the correlation between the words in the text and the detection targets in the image data to obtain the attention score. Then, combine the similarity and attention score during graph structure alignment to calculate the final graph alignment feature.

6. A cross-modal alignment device for a multimodal large language model, characterized in that: It includes: An image processing module is used to acquire image data, perform target detection on the image data, and generate fine-grained features of local images; The text processing module is used to obtain text data, extract text features, and extract text entity relationships to obtain an entity relationship dependency tree as a text entity relationship graph; The semantic alignment module is used to perform local fine-grained semantic alignment of local image fine-grained features and text features using a cross-attention network; The relationship detection module is used to obtain the visual relationship graph between the detection targets in the image data using the scene graph model VCTree; Graph alignment module, used to align the text entity relationship graph and the visual relationship graph in terms of graph structure and graph semantics; The fusion input module is used to fuse semantic alignment features and graph alignment features to obtain rich and fine-grained semantic alignment features, as well as global semantic alignment features and richer relation alignment features, which are then input into the multimodal large model to complete related tasks.

7. The cross-modal alignment device for a multimodal large language model according to claim 6, characterized in that: The image processing module acquires image data, performs target detection on the image data using the FasterRCNN model, and generates fine-grained features of local images.

8. The cross-modal alignment device for a multimodal large language model according to claim 7, characterized in that: After the text processing module obtains the text data, it uses the BGE model to extract text features and uses the BERT model to extract text entity relationships.

9. The cross-modal alignment device for a multimodal large language model according to claim 6, characterized in that: The graph alignment module includes a structural alignment unit and a semantic alignment unit; The structural alignment unit performs graph structural alignment on the text entity relationship graph and the visual relationship graph. The specific alignment process is as follows: The nodes in the text entity relationship graph and the visual relationship graph are converted into low-dimensional vector representations through graph neural networks, that is, the node representation vector features are obtained; Calculate the similarity between nodes in the text entity relationship graph and the visual relationship graph; Select N nodes from all nodes as reference points, calculate the similarity between N reference points and the remaining nodes, and generate a similarity matrix C; Select the node pair with the highest similarity from the similarity matrix C as the aligned nodes, that is, find the pair of nodes with the most similar structure in the text entity relationship graph and the visual relationship graph; The semantic alignment unit performs graph semantic alignment on the text entity relationship graph and the visual relationship graph. The specific alignment process is as follows: First, the guided attention mechanism is used to calculate the correlation between the words in the text and the detection targets in the image to obtain the attention score; Then, the similarity and attention score during graph structure alignment are combined to calculate the final graph alignment features.

Images