Ship multi-modal image fusion identification method based on graph neural structure alignment

Abstract

The invention particularly relates to a ship multi-modal image fusion recognition method based on graph neural structure alignment, and the method comprises the following steps: 1, obtaining a ship multi-modal image, and constructing a backbone neural network to extract the features of the multi-modal image; 2, constructing graph nodes of the graph neural network, and generating an adjacent edge relationship; 3, for graph structures constructed in different modes, adopting a two-level graph attention mechanism to complete structure alignment; step 4, utilizing an optimal transmission mechanism to realize structure alignment between the infrared and visible light modal diagrams; step 5, feature re-injection is carried out to fuse space coordinates and global information, and the positioning and expression ability of node features is improved; and step 6, training the constructed ship multi-modal image fusion recognition network by adopting local feature alignment loss, graph-level semantic consistency loss and classification supervision loss. According to the method, the problem of alignment errors caused by inconsistency of infrared and visible light modal images is solved, the structure and semantic information of the infrared and optical images are fully fused, the accuracy and robustness of cross-modal target recognition are effectively improved, and the method is suitable for complex scenes such as multi-modal ship recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of multi-modal deep learning and graph neural network fusion, and particularly relates to a ship multi-modal image fusion recognition method based on graph neural structure alignment. BACKGROUND

[0002] With the continuous development of artificial intelligence technology, multi-modal fusion has gradually become a hot research direction in the field of computer vision. In complex environments, single modal perception often faces problems such as illumination changes, occlusion interference, and imaging blur, resulting in a decrease in perception accuracy and robustness. For example, in tasks such as remote sensing monitoring, target recognition, and autonomous driving, visible light images have clear texture details, but their performance degrades significantly in low-light or adverse weather conditions. Infrared images, on the other hand, have the advantages of strong penetration and stable imaging, and can stably obtain target thermal information at night or in rain and fog environments. Therefore, how to effectively fuse information from different modalities, fully leverage their respective advantages, and improve the accuracy and robustness of target recognition has become an important problem that needs to be solved. However, due to the natural differences in perception level, semantic expression, and structure distribution between multi-modal images, simple feature concatenation or linear fusion methods cannot fully exploit the complementary information between modalities, and may even introduce modal conflicts and noise redundancy, affecting the final recognition performance.

[0003] To address the above problems, researchers have proposed a variety of multi-modal fusion strategies. Early methods mostly rely on simple image-level or feature-level fusion, such as pixel superposition, weighted summation, early fusion, and late fusion, but these methods lack a deep understanding of modal relationships. In recent years, with the development of graph neural networks (GNN) and attention mechanisms, researchers have begun to use graph structures to model the spatial semantic relationships within modalities, and have used graph convolution or graph attention mechanisms (GAT) to improve fusion quality. Some works introduce alignment mechanisms, such as feature distance-based, maximum mean discrepancy (MMD), or adversarial learning-based alignment methods, to try to alleviate the problem of inconsistent distribution between modalities. Some research uses the Sinkhorn algorithm in optimal transport theory to construct an optimal matching matrix at the node level between modalities, to enhance the semantic alignment effect between modalities. However, these methods still have some shortcomings, such as using graph structures for local modeling only, ignoring global consistency; lacking a bridging mechanism to achieve cross-modal semantic collaboration; and the alignment results not being effectively fed back to the original feature maps for discrimination, making it difficult to achieve closed-loop fusion of structural information and semantic features.

[0004] Therefore, it is urgent to design a multi-modal fusion framework with structure perception, semantic alignment and feature feedback capability to effectively solve the problems of modal heterogeneity and fusion difficulty. In this paper, a multi-modal fusion method based on structure alignment is proposed by combining graph structure modeling, optimal transport alignment and feature re-injection strategy. The method first extracts local and global structure features within the modal by constructing a dual-channel graph structure of space + semantics; then it realizes cross-modal node alignment by using the Sinkhorn optimal transport mechanism, and establishes a one-to-one mapping relationship at the semantic level; finally, it feeds the aligned features back to the original feature map through the feature re-injection module, improves the collaborative expression ability between modes, and realizes more robust and efficient multi-modal fusion recognition. SUMMARY

[0005] The application provides a ship multi-modal image fusion recognition method based on graph neural structure alignment, which is used to solve the problems of insufficient fusion of infrared and visible light information in multi-modal remote sensing images and difficult semantic alignment, can fully excavate the complementary structure information of infrared and visible light images at different scales, improve the collaborative understanding and recognition accuracy of multi-modal images, and achieve better ship image recognition effect.

[0006] To solve the above technical problems, the technical scheme adopted by the application is as follows: a ship multi-modal image fusion recognition method based on graph neural structure alignment, characterized by comprising the following steps:

[0007] Step one, acquiring ship multi-modal images, constructing a backbone neural network to extract multi-modal image features;

[0008] Step 101: acquiring infrared and visible light ship images, ensuring that each pair of images has a one-to-one correspondence in semantics, and then loading the infrared and visible light images in sequence according to the sample order to provide consistent input for double-branch processing;

[0009] Step 102: constructing a double-branch YOLOv8 network structure for processing infrared and visible light images respectively, which has multi-layer feature extraction capability, and the two branches share the structure but independently extract their own modal features to ensure modal specificity;

[0010] Step 103: inputting the infrared and visible light images into the corresponding branches respectively to extract multi-scale feature representations, and finally selecting the last three layers of output in the network as key features, wherein the feature maps of the infrared modal are denoted as C3_ir, C4_ir and C5_ir, and the feature maps of the visible light modal are denoted as C3_rgb, C4_rgb and C5_rgb, which provide a basis for subsequent graph construction and cross-modal alignment;

[0011] Step two, constructing graph nodes of the graph neural network and generating adjacent edge relationships;

[0012] Step 201: graph node construction: for the multi-layer semantic feature map extracted for the backbone network, let its size be where B is the batch size, C is the number of channels, (H, W) is the length and width size, m e {ir, rgb} represents the modal type, each feature map is divided into g x g equal size regions using grid division, and a graph node representation is extracted for each region, forming N = g 2 graph nodes, N is the total number of nodes, and the features of each grid region are obtained by adaptive pooling operation to obtain region embedding, denoted as where represents the i-th grid region feature of modal m, Pool(·) represents the pooling function, and all node feature representations of modal m are At the same time, to support subsequent spatial edge construction, the normalized coordinate center of each node in the image is recorded as i, j e [0, g-1];

[0013] Step 202: adjacent edge relationship generation: two types of edges are constructed within each modal, which are spatially adjacent edges and modal semantic edges, which together constitute the adjacent tensor for input of graph neural network; wherein the spatially adjacent edges are calculated according to the normalized spatial coordinates p i of the nodes, and the Euclidean distance matrix D i,j between nodes is calculated as i j For each node i, connect its k nearest neighbors in space to form the adjacency matrix A

[0014] Step 203: modal semantic edges use node features to calculate semantic similarity matrix Usually, inner product or cosine similarity is used as a measure, For each node i, keep the top r% connections with the highest similarity to form the semantic adjacency matrix A Finally, two edge matrices are formed for each modal for subsequent explicit modeling of structure information by graph neural network;

[0015] Step three, for the graph structures constructed for different modalities, a two-level graph attention mechanism is used to complete structure alignment;

[0016] Step 301: use the node features extracted in the previous step and the edge relationship tensor to construct a local graph for each image, and for the edge tensor A of each batch graph b, traverse the edge channel e, extract all edge connection pairs (k, n), and construct the edge index matrix E ​​Organize it into a sparse graph structure for the input of the graph attention mechanism;

[0017] Step 302: The local alignment layer updates the features of the nodes within each modality. The original node features are updated using a multi-head graph attention mechanism Adjacency structure modeling is performed to strengthen the node interaction under local topology. The calculation formula is Where d = h·d' is the output dimension, h is the number of attention heads, d' is the output dimension of each head, and GAT is the graph convolution operation with attention mechanism.

[0018] Step 303: To achieve global feature alignment across modalities, a bridge node is introduced First, it is mapped to the same dimension as the graph nodes through a fully connected network, Then the node is concatenated to the graph node set H b , obtaining the enhanced graph Meanwhile, add K edges to the bridge node to connect all nodes in the graph, constructing the global edge set

[0019] Step 304: Global graph attention fusion and graph-level representation generation. The full graph containing the bridge node is input into the second layer of graph convolution network to generate the fused node features Finally, the N+1th node (bridge node) is taken as the graph-level representation Z b , that is All batches of graph embeddings are concatenated to obtain the final output

[0020] Step four, use the optimal transport mechanism to realize the structure alignment between infrared and visible light modal graphs.

[0021] Step 401: Considering the graph node features of the two modalities, the L2 normalization is used for the graph node features of the infrared and visible light modalities respectively to eliminate the scale difference between modalities, obtaining the unit vector Calculate the cosine distance between the two to form the cost matrix Get the cost tensor

[0022] Step 402: To achieve cross-modal structure alignment, the Sinkhorn-Knopp algorithm is introduced to solve the cost matrix Regularized optimal transport; model the optimal transport with entropy regularization Where H(T) = -∑ ij T ij logT ij is the entropy regularization term, T ijdenotes the matching strength between the i-th node in the infrared modality and the j-th node in the visible light modality, and ε is a regularization coefficient, denotes that the transmission matrix satisfies the given marginal constraint, and the optimal transmission matrix T is an inter-modal node mapping matrix obtained by iterative solving under the additive entropy regularization constraint by the Sinkhorn-Knopp algorithm, and is used to measure the matching relationship between nodes in the infrared modality and the visible light modality;

[0023] Step 403: Feature alignment and reconstruction, according to the optimal transmission matrix T, the aligned features of the two modalities are reconstructed, and the reconstructed aligned features are

[0024] Step five, feature re-injection fusion of spatial coordinates and global information, to improve the positioning and expression ability of node features;

[0025] Step 501: In order to enhance the spatial positioning ability of node features, first, the original feature map F is A normalized coordinate map is constructed where 2 represents 2 channels corresponding to horizontal and vertical coordinates, and then mapped to a coordinate feature F with the same channel dimension as the original feature map by 3x3 convolution coord =Conv 3×3 (G), to realize feature coding of coordinate information;

[0026] Step 502: Let the node feature output by the graph neural network be where V=v 2 represents a two-dimensional grid node of v x v, which is converted into a two-dimensional structure and then bilinearly interpolated and upsampled to recover the re-injection graph node feature with the same size as the original feature map

[0027] Step 503: Add the original feature F orig and the coordinate convolution feature F coord , then perform 1x1 convolution and apply Sigmoid activation to generate a spatial attention weight map used for weighted control of the enhanced feature map A w =σ(Conv 1×1 (F orig +F coord ));

[0028] Step 504: Use the weighting mechanism to fuse the re-injection node feature with the original feature map F orig , and do global weighted enhancement with the bridging feature to obtain the final structure-aware feature map where denotes element-wise multiplication, F bridgeBy converting the features into (B, C, 1, 1)-dimensional features, the local spatial features and the global context information are effectively fused, and the richness and recognition ability of the feature expression are improved;

[0029] Step six, local feature alignment loss, graph-level semantic consistency loss and classification supervision loss are used to train the constructed ship multi-modal image fusion recognition network;

[0030] Step 601: The multi-modal alignment loss is for the semantic alignment between local regions in different modalities, and an optimal transport loss function based on Sinkhorn optimization is designed, assuming that the graph node features extracted in two modalities are respectively and The cost matrix is defined as the cosine distance The Sinkhorn algorithm is used to solve the balanced optimal transport matrix And the multi-modal alignment loss is defined as This loss encourages the correct matching of graph nodes with similar semantics between different modalities, and improves the consistency of local structures;

[0031] Step 602: The graph-level representation consistency loss is the graph-level embedding extracted by the graph attention network which represents the overall structure semantics in the modal, and its alignment loss is set as the mean square error form This loss term encourages semantic consistency between the two modalities from the graph structure level, effectively suppresses the propagation of low-level heterogeneity errors, and improves the consistency of global embedding expression;

[0032] Step 603: Under the premise of fusing multi-modal alignment, graph embedding consistency and basic classification supervision, the final training target function can be represented as where, represents the cross-entropy classification loss, and λ1 and λ2 are balance weight coefficients, which are usually adjusted through the validation set, and this joint loss design enables the model to consider the local and global modal alignment and expression consistency under the supervision guidance, effectively improving the multi-modal recognition performance.

[0033] Compared with the prior art, the present application has the following advantages:

[0034] The present application aims to overcome the shortcomings of existing multi-modal fusion methods, and proposes a ship multi-modal image fusion recognition method based on graph neural structure alignment. By constructing a graph structure, multi-level semantic alignment and structure feature re-injection, the semantic space of infrared and visible light features is precisely aligned, and the model is guided to learn the consistent representation across modalities. This method not only improves the modeling ability of structural differences between different modalities, but also significantly enhances the expression efficiency of the fused features, and improves the robustness and recognition accuracy of the model in the multi-modal scene.

[0035] The technical solutions of the present application will be further described in detail below with reference to the drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 A flow chart of a ship multi-modal image fusion recognition method based on graph neural structure alignment; DETAILED DESCRIPTION

[0037] The method of the present application will be further described in detail below with reference to the drawings and embodiments of the present application.

[0038] It should be noted that the embodiments and features in the embodiments in the present application can be combined with each other without conflict. The present application will be described in detail below with reference to the drawings and embodiments.

[0039] It should be noted that the terms used herein are only for describing specific embodiments, and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should also be understood that when the terms "comprise" and / or "include" are used in the specification, there is a presence of a feature, step, operation, device, component and / or combinations thereof.

[0040] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0041] For purposes of the description hereinafter, spatially relative terms, such as "above", "below", "up", "down", "right", "left", "vertical", "horizontal", "top", "bottom", "lateral", "longitudinal", "upper", "lower", "over", "under", "in", "out", "side", "front", "rear", and the like, can be used where appropriate to describe an element's or feature's relationship to another element or feature as illustrated in the figures. The spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. The devices can be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly. The terms "first", "second", "third", etc., do not necessarily denote any ordinal, chronological or other sequence unless specified herein. The terms "first", "second", "third", etc., are mainly used to distinguish between two or more elements or steps.

[0042] Reference Figure 1 As shown, in the example embodiment, a ship multi-modal image fusion recognition method based on graph neural structure alignment is provided, which can specifically include the following steps:

[0043] Step one, acquire ship multi-modal image, construct backbone neural network to extract multi-modal image features;

[0044] Step 101: Acquire infrared and visible light ship images, ensure that each pair of images has a one-to-one correspondence in semantics, then load the infrared and visible light images in sequence according to the sample order, and provide consistent input for double-branch processing;

[0045] Step 102: Construct a double-branch YOLOv8 network structure for processing infrared and visible light images, respectively. The network has multi-layer feature extraction capability, and the two branches share the structure but independently extract their own modal features to ensure that modal specificity is preserved;

[0046] Step 103: Input the infrared and visible light images into the corresponding branches respectively to extract multi-scale feature representations, and finally select the last three layers of the network as the key features, where the feature maps of the infrared modality are denoted as C3_ir, C4_ir, and C5_ir, and the feature maps of the visible light modality are denoted as C3_rgb, C4_rgb, and C5_rgb, providing a basis for subsequent graph construction and cross-modal alignment;

[0047] Step two, construct the graph nodes of the graph neural network and generate the adjacency edge relationship;

[0048] Step 201: Graph node construction: for the multi-layer semantic feature maps extracted by the backbone network, let their size be where B is batch size, C is channel number, (H, W) is height and width size, m e {ir, rgb} represents modality type, each feature map is divided into g x g equal size regions by grid partition, and one graph node representation is extracted from each region to form N = g 2 graph nodes, N is the total number of nodes, and the features of each grid region are obtained by adaptive pooling operation to obtain region embedding, denoted as where represents the i-th grid region feature of modality m, Pool(·) represents the pooling function, and all node feature representations of modality m are At the same time, in order to support subsequent spatial edge construction, the normalized coordinate center of each node in the image is recorded as

[0049] Step 202: Adjacent edge relationship generation: two types of edges are constructed within each modality, which are spatial adjacent edges and modality semantic edges, and together constitute the adjacent tensor for input of graph neural network; wherein the spatial adjacent edge is calculated according to the normalized spatial coordinates p i of the nodes D i,j = ‖p i -p j ‖2, for each node i, connecting its k nearest neighbors in space to form the adjacent matrix

[0050] Step 203: Modality semantic edge uses node feature to calculate semantic similarity matrix Generally, inner product or cosine similarity is used as a measure, for each node i, the top r% connections with the highest similarity are retained to form the semantic adjacent matrix Finally, two edge matrices are formed for each modality for subsequent explicit modeling of structure information by graph neural network;

[0051] Step three, for the graph structure constructed by different modalities, a two-level graph attention mechanism is used to complete structure alignment;

[0052] Step 301: Use the node features extracted in the previous step and the edge relationship tensor to construct a local graph for each image, and for the edge tensor of the graph b in each batch, traverse the edge channel e, extract all edge connection pairs (k, n), and construct an edge index matrix organize it into a sparse graph structure for input of graph attention mechanism;

[0053] Step 302: The local alignment layer updates the features of the nodes within each modality, using a multi-head graph attention mechanism to update the original node features Adjacency structure modeling is performed to strengthen the interaction between nodes under local topology, and the calculation formula is where d = h-d' is the output dimension, h is the number of attention heads, d' is the output dimension of each head, and GAT is a graph convolution operation with attention mechanism.

[0054] Step 303: In order to realize the global feature alignment across modalities, a bridge node is introduced First, it is mapped to the same dimension as the graph nodes through a fully connected network, Then the node is concatenated to the graph node set H b , obtaining the enhanced graph At the same time, K edges are added to the bridge node, connecting all nodes in the graph, and a global edge set is constructed

[0055] Step 304: Global graph attention fusion and graph-level representation generation, input the full graph containing the bridge node into the second layer graph convolution network, and generate the fused node features Finally, the N+1th node (bridge node) is taken as the graph-level representation Z b , that is The graph embeddings of all batches are concatenated to obtain the final output

[0056] Step four, use the optimal transport mechanism to realize the structure alignment between infrared and visible light modality graphs;

[0057] Step 401: Considering the graph node features of the two modalities, the graph node features of the infrared and visible light modalities are normalized by L2 to eliminate the scale difference between modalities, obtaining unit vectors The cosine distance between the two is calculated to form the cost matrix The cost tensor is obtained

[0058] Step 402: In order to realize the cross-modality structure alignment, the Sinkhorn-Knopp algorithm is introduced to solve the cost matrix Regularized optimal transport; through optimal transport modeling with entropy regularization where H(T) = -∑ ij T ij logT ij is the entropy regularization term, T ij represents the matching strength between the i-th node in the infrared modality and the j-th node in the visible light modality, and ε is the regularization coefficient, The optimal transmission matrix T is obtained by the Sinkhorn-Knopp algorithm under the additive entropy regularization constraint, and is used to measure the matching relationship between nodes of the infrared modal and the visible light modal;

[0059] Step 403: feature alignment and reconstruction, according to the optimal transmission matrix T, the aligned feature reconstructed by the two modalities is

[0060] Step five, feature re-injection fusion space coordinates and global information, improve the positioning and expression ability of node features;

[0061] Step 501: in order to enhance the spatial positioning ability of node features, first for the original feature map Construct a normalized coordinate map Where 2 represents 2 channels corresponding to horizontal and vertical coordinates, and then mapped to the coordinate feature F with the same channel dimension as the original feature map through 3*3 convolution coord = Conv 3×3 (G), to realize the feature coding of coordinate information;

[0062] Step 502: let the node feature output by the graph neural network be Where V=v 2 Represents a two-dimensional grid node of v*v, which is converted into a two-dimensional structure and then bilinearly interpolated and upsampled to recover the re-injection graph node feature with the same size as the original feature map

[0063] Step 503: add the original feature F orig And the coordinate convolution feature F coord After 1*1 convolution and sigmoid activation, the spatial attention weight map is generated Used to control the enhanced feature map A w =σ(Conv 1×1 (F orig +F coord ));

[0064] Step 504: use the weighted mechanism to fuse the re-injection node feature With the original feature map F orig And the bridging feature Do global weighted enhancement to get the final structure-aware feature map Where ⊙ represents element-wise multiplication, F bridge Through the conversion operation to the feature with the dimension of (B, C, 1, 1), the local spatial feature and the global context information are effectively fused, and the richness and recognition ability of the feature expression are improved;

[0065] Step six, using local feature alignment loss, graph-level semantic consistency loss and classification supervision loss, training the constructed ship multi-modal image fusion recognition network;

[0066] Step 601: The multi-modal alignment loss is for the semantic alignment between local regions in different modalities. An optimal transport loss function based on Sinkhorn optimization is designed. Assuming that the graph node features extracted in two modalities are respectively and The cost matrix is defined as the cosine distance The Sinkhorn algorithm is used to solve the balanced optimal transport matrix And the multi-modal alignment loss is defined as This loss encourages correct matching between graph nodes with similar semantics in different modalities, and improves the consistency of local structure;

[0067] Step 602: The graph-level representation consistency loss is the graph-level embedding extracted by the graph attention network The alignment loss of the graph-level embedding is set as the mean square error form This loss term encourages semantic consistency from the graph structure level in two modalities, effectively suppresses the propagation of low-level heterogeneity error, and improves the consistency of global embedding expression;

[0068] Step 603: Under the premise of combining multi-modal alignment, graph embedding consistency and basic classification supervision, the final training objective function can be represented as Where, The cross-entropy classification loss is represented as λ1 and λ2 are balance weight coefficients, which are usually adjusted through the validation set. This joint loss design enables the model to consider both local and global modal alignment and expression consistency under the supervision of the guide, effectively improving the multi-modal recognition performance.

[0069] The above is only an embodiment of the present application, and does not limit the present application. Any simple modification, change and equivalent structure change according to the technical essence of the present application to the above embodiment are still within the protection scope of the technical solution of the present application.

Claims

1. A ship multimodal image fusion recognition method based on graph neural structure alignment, characterized by: The following steps are involved: Step 1: Obtain multimodal images of ships and construct a backbone neural network to extract multimodal image features; Step 101: Obtain infrared and visible light ship images, ensuring that each pair of images has a one-to-one semantic correspondence, and then load the infrared and visible light images in sequence according to the sample order to provide consistent input for the dual-branch processing; Step 102: Construct a dual-branch YOLOv8 network structure for processing infrared and visible light images respectively. The network has multi-layer feature extraction capabilities. The two branches share the same structure but independently extract features of each modality to ensure that modality specificity is preserved. Step 103: Input the infrared and visible light images into the corresponding branches respectively, extract multi-scale feature representations, and finally select the outputs of the last three layers in the network as key features. The feature maps of the infrared modality are denoted as C3_ir, C4_ir, and C5_ir, and the feature maps of the visible light modality are denoted as C3_rgb, C4_rgb, and C5_rgb, which provide a basis for subsequent graph construction and cross-modal alignment. Step 2: Construct the graph nodes of the graph neural network and generate adjacent edge relationships; Step 201: Graph node construction: For the multi-layer semantic feature graph extracted by the backbone network, set its size to Where B is the batch size, C is the number of channels, (H, W) is the length and width, m∈{ir,rgb} represents the modality type, and each feature map is divided into g×g equal-sized regions using a grid partitioning method. A graph node is extracted from each region to form N=g 2 graph nodes, N is the total number of nodes, and the features of each grid area are embedded by adaptive pooling operation, which is recorded as in represents the i-th grid region feature of mode m, Pool(·) represents the pooling function, so all node features of mode m are expressed as At the same time, to support the subsequent construction of spatial edges, the normalized coordinate center of each node in the image is recorded as Step 202: Generate adjacency edge relationships: Construct two types of edges within each modality, namely spatial adjacency edges and modality semantic edges, which together constitute the adjacency tensor. Used as input for graph neural networks; where the spatial neighboring edges are based on the normalized spatial coordinates p of the nodes i , calculate the Euclidean distance matrix between nodes D i,j =‖p i -p j ‖2, for each node i, connect its k nearest neighbors in space to form an adjacency matrix Step 203: Modal semantic edges use node features to calculate semantic similarity matrix Usually inner product or cosine similarity is used as a metric. For each node i, retain the top r% connections with the highest similarity to form a semantic adjacency matrix Finally, each mode forms two edge matrices Used for subsequent graph neural network to explicitly model structural information; Step 3: For graph structures constructed from different modalities, a two-level graph attention mechanism is used to complete structural alignment; Step 301: Utilize the node features extracted in the previous step Edge relationship tensor Construct a local graph for each image, and the edge tensor of the graph b in each batch Traverse the edge channel e, extract all edge connection pairs (k,n), and construct the edge index matrix Organize it into a sparse graph structure for graph attention mechanism input; Step 302: The local alignment layer updates the features of the nodes within each modality and uses the multi-head graph attention mechanism to update the original node features. Adjacency structure modeling is performed to strengthen node interaction under the local topological structure. The calculation formula is: Where d = h·d′ is the output dimension, h is the number of attention heads, d′ is the output dimension of each head, and GAT is a graph convolution operation with attention mechanism; Step 303: In order to achieve cross-modal global feature alignment, a bridge node is introduced First, map it to the same dimension as the graph node through a fully connected network. Then the node is spliced ​​into the graph node set H b , and get the enhanced graph At the same time, add K edges to the bridge node, connect all nodes in the graph, and construct a global edge set Step 304: Global graph attention fusion and graph-level representation generation. Input the full graph including the bridge nodes into the second-layer graph convolutional network to generate fused node features. Finally, take the N+1th node (bridge node) as the graph-level representation Z b ,Right now All batches of graph embeddings are concatenated into the final output Step 4: Use the optimal transmission mechanism to achieve structural alignment between infrared and visible light modal images; Step 401: Considering the graph node features of the two modalities, the graph node features of the infrared modality and the visible light modality are respectively normalized using L2 to eliminate the scale difference between the modalities, and the unit vector is obtained. Calculate the cosine distance between the two to form a cost matrix Get the cost tensor Step 402: In order to achieve cross-modal structural alignment, the Sinkhorn-Knopp algorithm is introduced to the cost matrix Solve the regularized optimal transport problem; Model the optimal transport problem with entropy regularization Where H(T) = -∑ ij T ij logT ij is the entropy regularization term, T ij represents the matching strength between the i-th node in the infrared modality and the j-th node in the visible light modality, ε is the regularization coefficient, It means that the transfer matrix satisfies the given marginal constraints. The optimal transfer matrix T is the inter-modal node mapping matrix obtained by iteratively solving the Sinkhorn-Knopp algorithm under the additive entropy regularization constraint. It is used to measure the matching relationship of each node pair between the infrared mode and the visible light mode. Step 403: Feature alignment and reconstruction. According to the optimal transmission matrix T, the alignment features of the two modes are reconstructed. The reconstructed alignment features are Step 5: Feature re-injection integrates spatial coordinates and global information to improve the positioning and expression capabilities of node features; Step 501: In order to enhance the spatial positioning capability of node features, firstly, the original feature map Constructing a normalized coordinate map Where 2 means that the two channels correspond to the horizontal and vertical coordinates respectively, and then the 3×3 convolution is mapped to the coordinate feature F with the same channel dimension as the original feature map. coord =Conv 3×3 (G) to achieve feature encoding of coordinate information; Step 502: Assume that the node features output by the graph neural network are Where V = v 2 Represents a v×v two-dimensional grid node, converts it into a two-dimensional structure, performs bilinear interpolation upsampling, and restores the re-injection graph node feature to the same size as the original feature map Step 503: The original feature F orig Convolution feature F with coordinates coord After addition, a 1×1 convolution is performed and Sigmoid activation is applied to generate a spatial attention weight map Used to perform weighted control on the enhanced feature map A w =σ(Conv 1×1 (F orig +F coord )); Step 504: Use a weighting mechanism to inject node features With the original feature map F orig Fusion and bridging features Perform global weighted enhancement to obtain the final structure-aware feature map where ⊙ represents element-wise multiplication, F bridge By converting the feature into (B, C, 1, 1) dimension, the local spatial features and global context information are effectively integrated to improve the richness of feature expression and recognition ability; Step 6: Use local feature alignment loss, image-level semantic consistency loss, and classification supervision loss to train the constructed ship multimodal image fusion recognition network; Step 601: Multimodal alignment loss is designed to optimize the transfer loss function based on Sinkhorn optimization for semantic alignment between local regions under different modalities. Assume that the graph node features extracted under the two modalities are and Define the cost matrix as cosine distance Solving the balanced optimal transmission matrix using the Sinkhorn algorithm And define the multimodal alignment loss This loss encourages semantically similar graph nodes to be correctly matched between different modalities, improving the consistency of local structure; Step 602: Graph-level representation consistency loss is used to extract the graph-level embeddings from the graph attention network. Represents the overall structural semantics within the modality, and sets its alignment loss to the mean square error form This loss term encourages the two modalities to obtain semantic consistency from the graph structure level, effectively suppresses the propagation of low-level heterogeneity errors, and improves the consistency of global embedding expression; Step 603: Under the premise of integrating multimodal alignment, graph embedding consistency and basic classification supervision, the final training objective function can be expressed as in, represents the cross entropy classification loss, λ1 and λ2 are balance weight coefficients, which are usually adjusted by the validation set. This joint loss design enables the model to take into account both local and global modality alignment and expression consistency under supervision, effectively improving multimodal recognition performance.

Images