Transform multi-mode multi-vision scene identification method and system based on cross guidance fusion

Abstract

The invention discloses a Transform multi-mode multi-vision scene recognition method and a Transform multi-mode multi-vision scene recognition system based on cross guidance fusion. The method comprises the following steps: firstly, constructing an aviation and ground image data set, and extracting dual-mode global features and local image marks by adopting a double-flow vision Transform; through an instance-level multi-modal alignment module, performing consistency constraint on global features on an instance level by using bidirectional contrast learning; meanwhile, a fine-grained alignment module of Token attention is introduced, key areas most relevant to tasks are screened from local image marks through a cross attention mechanism, and weighted comparison alignment is carried out; and finally, fusing the aligned global features, and inputting the fused global features into a classifier to complete scene recognition. According to the invention, through a'global-local 'double-layer cross guide alignment mechanism, the accuracy and robustness of multi-modal fusion identification in a complex scene are significantly improved.

Description

Technical Field

[0001] This invention relates to the field of multimodal remote sensing scene recognition technology, and more specifically, to a Transformer multimodal multi-vision scene recognition method and system based on cross-guided fusion. Background Technology

[0002] With the rapid development of remote sensing technology, the acquisition of multimodal and multi-view data (such as aerial and ground imagery) has become increasingly convenient, providing a rich data foundation for intelligent Earth observation missions. By fusing complementary information from different perspectives and modalities, scene recognition performance in complex scenarios can be significantly improved, which has important application value in fields such as environmental monitoring, urban planning, and military reconnaissance. However, the significant differences in perspective between aerial and ground images, complex scene backgrounds, and sensor-introduced noise make effectively mining and fusing cross-modal complementary features a key scientific problem for improving the robustness and accuracy of scene recognition.

[0003] While existing technologies have made some progress in processing multimodal and multi-view remote sensing image fusion, they still suffer from the following significant drawbacks and technical bottlenecks:

[0004] Shallow fusion strategies lead to insufficient cross-modal correlation mining: Most existing methods employ simple and direct fusion strategies, such as feature-level concatenation or element-wise summation, mechanically combining features from aerial and ground images in post-processing. These methods essentially treat multimodal features as independent, parallel information flows, lacking explicit modeling of deep intermodal interactions and semantic complementarity. Because they fail to effectively explore the dynamic correlations between different modal feature channels, the model struggles to understand the semantic correspondence between "a region in the aerial view" and "which part in the ground view," resulting in a fused representation that remains merely a superficial information overlay, failing to truly achieve cross-modal knowledge complementarity and synergistic enhancement.

[0005] Coarse-grained global alignment struggles to handle dramatic changes in perspective and spatial misalignment: Aerial and ground images naturally exhibit dramatic changes in perspective (e.g., top-view vs. side-view), scale differences, and spatial misalignment, resulting in drastically different visual representations of the same target in the two modalities. Existing methods typically align only global features (e.g., image-level feature vectors), attempting to match the entire aerial image with the entire ground image in feature space. This coarse-grained global alignment ignores fine-grained correspondences in local regions. When complex scenes contain background interference, target occlusion, or extreme perspective changes, a large amount of task-irrelevant noise mixed in the global features (e.g., sky, vegetation, building backgrounds) severely interferes with the alignment process, preventing the model from accurately locating truly discriminative complementary regions across modalities, thus limiting the representational power of fused features.

[0006] Lack of task-oriented fine-grained interaction mechanisms: In multimodal remote sensing scenarios, the information most relevant to the recognition task is often only reflected in key local areas of the image (such as airport runways or school playgrounds), while most image areas are redundant background. However, existing methods lack task knowledge guidance during multimodal interaction and fail to dynamically filter and strengthen important local features according to the needs of downstream recognition tasks. Specifically, although traditional cross-modal attention mechanisms can achieve feature interaction between modalities, in complex remote sensing scenarios, they are easily interfered with by large areas of similar backgrounds, diverting attention to common non-target areas (such as common green spaces and roads) rather than core areas with strong discriminative power for specific scene categories. This task-decoupled interaction makes it difficult for models to extract truly helpful fine-grained cross-modal cues from massive amounts of data for classification decisions.

[0007] Poor robustness to noise and weakly aligned data: Real-world multimodal remote sensing data often contains complex factors such as sensor noise, illumination variations, and seasonal changes, resulting in weak alignment characteristics in multimodal data pairs. Most existing methods assume that multimodal data has achieved ideal spatial or semantic alignment; however, their performance drops sharply when processing such weakly aligned data. Due to the lack of effective noise filtering mechanisms, specific noise from individual modalities is easily transmitted and amplified into the joint feature space during the fusion process. This not only fails to achieve complementary enhancement but also introduces interference information, weakening the model's generalization ability and making it difficult to guarantee recognition accuracy in scenarios with small samples or limited data. Summary of the Invention

[0008] The main objective of this invention is to provide a Transformer multimodal and multi-view scene recognition method and system based on cross-guided fusion, which at least solves the problems of high noise and low alignment accuracy in multimodal data fusion in the prior art, and achieves a significant improvement in the performance of highly robust feature fusion and scene recognition of multimodal and multi-view images in complex remote sensing scenarios.

[0009] To achieve the above objectives, a Transformer multimodal and multi-vision scene recognition method and system based on cross-guided fusion is provided.

[0010] In a first aspect, the present invention provides a Transformer multimodal and multi-vision scene recognition method based on cross-guided fusion, the recognition method comprising:

[0011] Construct a pairwise heterogeneous dataset for multimodal and multi-view remote sensing scene recognition. The pairwise heterogeneous dataset includes aerial images of the target area and ground images paired with the aerial images.

[0012] A dual-stream visual Transformer network is used to extract features from paired heterogeneous datasets to obtain the first global features and first local image labels of aerial images, as well as the second global features and second local image labels of ground images.

[0013] Based on the instance-level multimodal alignment method, the first global feature and the second global feature are aligned, the first contrast loss and the second contrast loss are calculated based on the first global feature and the second global feature, and the total contrast loss is calculated using the first contrast loss and the second contrast loss.

[0014] The fine-grained alignment method based on token attention calculates the similarity score between the first global feature and the second global feature through a cross-attention mechanism. Based on the similarity score, it selects the most relevant third local image tag from the first local image tag and the most relevant fourth local image tag from the aerial image and the ground image, respectively. Through contrastive learning, it performs weighted alignment of the third local image tag and the fourth local image tag and calculates the total fine-grained loss.

[0015] The aligned first and second global features are fused to obtain the fused features. The single-modal classification loss is calculated. The fused features are processed by a classifier and combined with the single-modal classification loss to obtain the total classification loss.

[0016] A total objective function is constructed based on the total contrast loss, total fine-grained loss, and total classification loss. A multimodal multi-vision scene classifier is trained using the total objective function. The fused features are then input into the multimodal multi-vision scene classifier to obtain the multimodal multi-vision scene recognition results.

[0017] Specifically, based on the instance-level multimodal alignment method, the first global feature and the second global feature are aligned, a first contrastive loss and a second contrastive loss are calculated based on the first global feature and the second global feature, and the total contrastive loss is calculated using the first contrastive loss and the second contrastive loss, including:

[0018] The instance-level multimodal alignment method is used to calculate the first contrast loss between the first global feature and the second global feature within the same batch, when the first global feature is used as the anchor point; the formula for calculating the first contrast loss is:

[0019]

[0020] in, Indicates the first comparative loss. Represents cosine similarity. Indicates the first global feature. Indicates the second global feature. Indicates the first temperature parameter. Indicates the number of samples in the batch. Indicates the sample index within the same batch;

[0021] An instance-level multimodal alignment method is used to calculate the second contrast loss between the second global feature and the first global feature within the same batch, when the second global feature is used as the anchor point; the formula for calculating the second contrast loss is:

[0022]

[0023] in, This indicates the second comparative loss.

[0024] Specifically, the total contrast loss is calculated using the first contrast loss and the second contrast loss, including:

[0025] The total contrast loss is obtained by weighted summing of the first contrast loss and the second contrast loss. The formula for calculating the total contrast loss is as follows:

[0026]

[0027] in, This represents the total comparative loss.

[0028] Specifically, the fine-grained alignment method based on token attention calculates the similarity score between the first global feature and the second global feature through a cross-attention mechanism, including:

[0029] Using a cross-attention mechanism, a first global feature is used as a query to interact with a second local image label to generate a first common attention feature; the first similarity score between the first common attention feature and the second local image label is calculated.

[0030] The second global feature is used as a query to interact with the first local image label using a cross-attention mechanism to generate a second common attention feature; the second similarity score between the second common attention feature and the first local image label is calculated.

[0031] Specifically, based on similarity scores, the most relevant third local image markers in the aerial image and the most relevant fourth local image markers in the ground image are selected from the first and second local image markers, respectively, including:

[0032] The top k second local image labels, ranked from highest to lowest based on the first similarity score, are selected as the fourth local image labels.

[0033] The top k first local image labels are selected from high to low based on the second similarity score as the third local image labels.

[0034] Specifically, the third local image label and the fourth local image label are weighted and aligned through contrastive learning, and the total fine-grained loss is calculated, including:

[0035] Based on the contribution weights of the third and fourth local image labels to the first and second global features in the last self-attention layer, the selected third and fourth local image labels are weighted and summed to obtain the first fine-grained loss of the aerial image and the second fine-grained loss of the ground image, respectively.

[0036] The first fine-grained loss is calculated using the following formula:

[0037]

[0038] in, This represents the first fine-grained loss. Indicates the third local image marker. Indicates the fourth local image marker. Indicates the second temperature parameter;

[0039] The second fine-grained loss is calculated using the following formula:

[0040]

[0041] in, This represents the second fine-grained loss.

[0042] Specifically, the method involves weighted alignment of the third and fourth local image labels through contrastive learning, calculating the total fine-grained loss, and also includes:

[0043] The first and second fine-grained losses are bidirectionally aligned through contrastive learning to obtain the total fine-grained loss. The formula for calculating the total fine-grained loss is as follows:

[0044]

[0045] in, This represents the total fine-grained loss.

[0046] Specifically, the aligned first and second global features are fused to obtain fused features, including:

[0047] The first and second global features are projected onto a normalized multimodal feature space using a mapping function, and then summed and aggregated to obtain the fused features. The formula for calculating the fused features is as follows:

[0048]

[0049] in, Indicates fusion characteristics, and Represents a mapping function. Indicates the first global feature. This represents the second global feature.

[0050] Specifically, a total objective function is constructed based on the total contrastive loss, the total fine-grained loss, and the total classification loss, including:

[0051] The total objective function is obtained by weighted summing of the total classification loss, total contrast loss, and total fine-grained loss.

[0052] Secondly, this invention provides a Transformer multimodal multi-vision scene recognition system based on cross-guided fusion. The recognition system is applied to the recognition method of the first aspect, and the recognition system includes:

[0053] The data acquisition module is used to construct paired heterogeneous datasets for multimodal and multi-view remote sensing scene recognition. The paired heterogeneous datasets include aerial images of the target area and ground images paired with the aerial images.

[0054] The feature extraction module is connected to the data acquisition module. The feature extraction module is used to extract features from pairs of heterogeneous datasets using a dual-stream visual Transformer network to obtain the first global features and the first local image label of the aerial image, as well as the second global features and the second local image label of the ground image.

[0055] The instance-level alignment module is connected to the feature extraction module. The instance-level alignment module is used to align the first global feature and the second global feature based on the instance-level multimodal alignment method, calculate the first contrast loss and the second contrast loss based on the first global feature and the second global feature, and calculate the total contrast loss using the first contrast loss and the second contrast loss.

[0056] The fine-grained alignment module, connected to the instance-level alignment module, is used for a token-attention-based fine-grained alignment method. It calculates the similarity score between the first global feature and the second global feature through a cross-attention mechanism, and selects the most relevant third local image label from the first local image label and the most relevant fourth local image label from the ground image from the first local image label and the second local image label, respectively. Through contrastive learning, the third local image label and the fourth local image label are weighted and aligned, and the total fine-grained loss is calculated.

[0057] The feature fusion module is connected to the fine-grained alignment module. The feature fusion module is used to fuse the aligned first global features and second global features to obtain fused features, calculate the single-modal classification loss, process the fused features through a classifier and combine them with the single-modal classification loss to obtain the total classification loss.

[0058] The classification and recognition module is connected to the feature fusion module. The classification and recognition module is used to construct an overall objective function based on the overall contrast loss, overall fine-grained loss and overall classification loss. The overall objective function is used to train a multimodal and multi-vision scene classifier. The fused features are input into the multimodal and multi-vision scene classifier to obtain the multimodal and multi-vision scene recognition results.

[0059] This application provides a Transformer multimodal and multi-vision scene recognition method and system based on cross-guided fusion. This method constructs paired aerial and ground image datasets, employs a dual-stream visual Transformer network to extract global features and local image labels from both modalities, and innovatively designs an instance-level multimodal alignment module and a fine-grained alignment module using token attention. The former aligns global features from different modalities at the instance level using bidirectional contrastive loss, while the latter uses a cross-attention mechanism to select the most relevant key regions from local image labels and performs weighted alignment based on contribution weights. Finally, the aligned dual-modal global features are fused and input into a classifier for scene recognition. This method, through instance-level and fine-grained dual alignment mechanisms, effectively mines deep semantic relationships and local complementary information across modalities, significantly improving the robustness and recognition accuracy of multimodal fusion in complex remote sensing scenes. Attached Figure Description

[0060] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0061] Figure 1 A flowchart illustrating a Transformer multimodal and multi-vision scene recognition method based on cross-guided fusion provided for this application;

[0062] Figure 2 This is an example diagram of the data collected in the Transformer multimodal multi-vision scene recognition method based on cross-guided fusion provided in this application;

[0063] Figure 3 This is a schematic diagram of the overall framework of a Transformer multimodal and multi-vision scene recognition method based on cross-guided fusion provided in this application;

[0064] Figure 4 A connection diagram of a Transformer multimodal and multi-vision scene recognition system based on cross-guided fusion provided for this application;

[0065] Figure 5A schematic diagram of the confusion matrices for the Baseline and CGFT models on the AiRound dataset;

[0066] Figure 6 This is a schematic diagram of the confusion matrices for the Baseline and CGFT models on the CV-BrCT dataset. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0068] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein.

[0069] In this invention, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0070] This application provides a Transformer-based multimodal and multi-vision scene recognition method and system based on cross-guided fusion. The method first constructs paired aerial and ground image datasets, and extracts global features and local image labels for both modalities using a dual-stream vision Transformer network. Then, an instance-level multimodal alignment module is employed, utilizing bidirectional contrastive learning to constrain the consistency of global features from different modalities at the instance level. Simultaneously, a fine-grained alignment module with token attention is introduced, dynamically selecting the most task-relevant key regions from local image labels through a cross-attention mechanism, and performing weighted contrast alignment based on attention weights. Finally, the aligned dual-modal global features are fused and input into a classifier to complete scene recognition. This invention effectively solves the feature misalignment problem caused by drastic changes in perspective and background interference in multimodal remote sensing images by constructing a "global-local" two-level cross-guided alignment mechanism. It achieves accurate modeling of deep semantic associations across modalities and full mining of discriminative complementary information, significantly improving the accuracy and robustness of multimodal fusion recognition in complex scenes.

[0071] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0072] Figure 1 A flowchart illustrating a Transformer multimodal and multi-vision scene recognition method based on cross-guided fusion provided in this application is shown below. Figure 1As shown in this embodiment, a Transformer multimodal and multi-view scene recognition method based on cross-guided fusion is provided. This method includes: constructing paired heterogeneous datasets for multimodal and multi-view remote sensing scene recognition, where each dataset includes aerial images of the target area and ground images paired with the aerial images; using a two-stream visual Transformer network to extract features from the paired heterogeneous datasets to obtain first global features and first local image labels for the aerial images, and second global features and second local image labels for the ground images; aligning the first and second global features based on an instance-level multimodal alignment method, calculating a first contrast loss and a second contrast loss based on the first and second global features, and calculating the total contrast loss using the first and second contrast losses; and employing a fine-grained alignment method based on token attention. The similarity score between the first and second global features is calculated using a cross-attention mechanism. Based on the similarity score, the most relevant third local image marker in the aerial image and the most relevant fourth local image marker in the ground image are selected from the first and second local image markers, respectively. The third and fourth local image markers are weighted and aligned through contrastive learning, and the total fine-grained loss is calculated. The aligned first and second global features are fused to obtain fused features, and the single-modal classification loss is calculated. The fused features are processed by a classifier and combined with the single-modal classification loss to obtain the total classification loss. A total objective function is constructed based on the total contrastive loss, the total fine-grained loss, and the total classification loss. A multimodal multi-vision scene classifier is trained using the total objective function. The fused features are input into the multimodal multi-vision scene classifier to obtain the multimodal multi-vision scene recognition result.

[0073] This application provides a Transformer multimodal and multi-vision scene recognition method based on cross-guided fusion. The method first constructs a paired heterogeneous dataset containing aerial images of the target area and paired ground images. A dual-stream visual Transformer network is used to extract features from the dual-modal images, obtaining the first global feature and first local image label for the aerial images, and the second global feature and second local image label for the ground images. Then, based on an instance-level multimodal alignment method, the first and second global features are aligned at the instance level through bidirectional contrastive learning, and the total contrastive loss is calculated. Simultaneously, based on a token attention-based fine-grained alignment method, a cross-attention mechanism is used to calculate the similarity score between the dual-modal global features and local image labels, selecting the most relevant third and fourth local image labels, and achieving fine-grained alignment through weighted contrastive learning, calculating the total fine-grained loss. Finally, the aligned dual-modal global features are fused to obtain fused features, which are then processed by a classifier and combined with a single-modal classification loss to construct the overall objective function. This is used to train a multimodal and multi-vision scene classifier to output the final recognition result.

[0074] Figure 2 This is an example diagram of the data collected in the Transformer multimodal multi-vision scene recognition method based on cross-guided fusion provided in this application, such as... Figure 2 As shown, this is an aerial photograph and its corresponding ground image collected in a Transformer multimodal multi-vision scene recognition method based on cross-guided fusion.

[0075] This method uses two multimodal, multi-view remote sensing scene recognition datasets, named AiRound and CVBrCT. Each dataset contains different modalities of data from different viewpoints. Some examples of the aforementioned datasets are shown below. Figure 2 As shown. The dataset details are as follows:

[0076] The AiRound dataset contains 11,753 images from 11 remote sensing scene categories. The images in this dataset were acquired from three unique sensors with different modalities: RGB images from aerial and ground-based perspectives, and multispectral image data from the Sentinel-2 satellite. Notably, this dataset utilizes only aerial and ground-based image pairs.

[0077] The CV-BrCT dataset contains over 24,000 image pairs, divided into nine remote sensing scene categories. Each image pair includes images of the same target taken from both aerial and ground-based perspectives. All images in this dataset have been resized to 500×500 pixels.

[0078] The constructed pairwise heterogeneous dataset is represented as follows:

[0079] ;

[0080] ;

[0081] in, and These are the labels for the corresponding images. and These represent the number of samples in the two datasets, respectively. A batch of input multimodal data empty pairs is represented as follows: ,in , This refers to the batch size. The goal is to learn a scene recognition model. ,in This represents the learnable parameters of the model, which are used to accurately identify test data by combining air and ground planes.

[0082] Figure 3 This is a schematic diagram of the overall framework of a Transformer multimodal and multi-vision scene recognition method based on cross-guided fusion provided in this application, as shown below. Figure 3 The diagram illustrates the overall framework of a Transformer-based multimodal and multi-vision scene recognition method based on cross-guided fusion. This method employs cross-guided representation alignment to enhance multimodal feature fusion and recognition performance. This alignment is achieved through a two-stage adjustment strategy. First, instance-level contrastive learning is used to align the global label embeddings of aerial-ground image pairs. In the second stage, fine-grained alignment is enhanced by learning cross-modal similarity, which emphasizes important image sub-regions of the corresponding modality. This strategy ensures robust alignment by learning fine-grained inter-subject relationships between the two modalities, thereby improving recognition performance. Finally, multimodal features are fused to learn the joint representation for the final prediction.

[0083] This method has the following significant advantages: First, by constructing a dual alignment mechanism at both the instance level and fine-grained level, it ensures the overall consistency of global features across modalities while accurately mining the deep semantic relationships in key local areas, effectively solving the feature misalignment problem caused by drastic changes in perspective between aerial and ground images. Second, by introducing a weighted filtering strategy based on token attention, it can dynamically focus on the local image regions most relevant to the recognition task, significantly suppressing the interference of complex background noise on the fusion process and improving the robustness of the model in weakly aligned data scenarios. Third, through the synergistic optimization of bidirectional contrastive learning and cross-attention mechanism, it achieves efficient capture and fusion of complementary information across modalities, achieving recognition accuracy superior to existing methods in complex remote sensing scene recognition tasks, and maintaining stable performance even under small sample training conditions.

[0084] Specifically, based on the instance-level multimodal alignment method, the first global feature and the second global feature are aligned. A first contrast loss and a second contrast loss are calculated based on the first and second global features. The total contrast loss is then calculated using the first and second contrast losses, including: calculating the first contrast loss between the first global feature and the second global feature within the same batch, when the first global feature is used as the anchor point, using the instance-level multimodal alignment method; the formula for calculating the first contrast loss is:

[0085]

[0086] in, Indicates the first comparative loss. Represents cosine similarity. Indicates the first global feature. Indicates the second global feature. Indicates the first temperature parameter. Indicates the number of samples in the batch. This represents the sample index within the same batch; using an instance-level multimodal alignment method, the second contrast loss between the second global feature and the first global feature is calculated within the same batch when the second global feature is used as the anchor point; the formula for calculating the second contrast loss is:

[0087]

[0088] in, This indicates the second comparative loss.

[0089] This application provides a Transformer multimodal and multi-vision scene recognition method based on cross-guided fusion. In the instance-level multimodal alignment module, this method employs a bidirectional contrastive learning strategy to perform cross-modal alignment of the first global features of aerial images and the second global features of ground images: First, using the first global features of aerial images as anchors, the similarity is calculated with the second global features of all ground images in the same batch to construct a first contrastive loss, which maximizes the similarity of positive sample pairs (truthfully paired aerial-ground images) and minimizes the similarity of negative sample pairs; then, using the second global features of ground images as anchors, the similarity is calculated with the first global features of all aerial images in the same batch to construct a second contrastive loss; finally, the first contrastive loss and the second contrastive loss are weighted and summed to obtain the total contrastive loss used to constrain the consistency of cross-modal global features.

[0090] Instance-Wise Multi-Modal Alignment (IMA): First, a two-stream network architecture (e.g., ViT) is used to extract features. For each modality, two modalities encode image class labels (e.g., ...). and ) is through an aerial image encoder and ground image encoder The obtained representations are global representations. Furthermore, the encoder generates image tag embeddings. and For the two modes, among which, .

[0091] Mutual information loss is used to bring related embedding pairs closer together and push unrelated embedding pairs apart in each aerial-ground image pair by comparing pairs of global features with corresponding alternative modalities:

[0092]

[0093] in, Indicates the first comparative loss. Represents cosine similarity. Indicates the first global feature. Indicates the second global feature. Indicates the first temperature parameter. Indicates the number of samples in the batch. This indicates the sample index within the same batch.

[0094] Similarly, to ensure symmetrical alignment between the ground representation and the airborne modes, ground image features are compared. Features of aerial images To further minimize loss:

[0095]

[0096] in, This indicates the second comparative loss.

[0097] This method first employs a bidirectional contrastive learning mechanism, simultaneously applying symmetry constraints in both the aerial-to-ground and ground-to-aerial directions, ensuring the integrity and balance of cross-modal alignment and avoiding modal bias issues that may arise from unidirectional alignment. Second, it introduces negative samples at the batch level for contrastive learning, effectively widening the distance between samples of different categories in the joint feature space and enhancing the discriminative power of the features. Finally, by maximizing the similarity of positive sample pairs, it significantly reduces the semantic gap between aerial and ground images caused by dramatic changes in perspective, laying a solid foundation for global consistency in subsequent fine-grained alignment and feature fusion, and effectively improving the overall performance of multimodal fusion.

[0098] Specifically, the total contrast loss is calculated using the first contrast loss and the second contrast loss, including: weighted summing of the first contrast loss and the second contrast loss to obtain the total contrast loss. The formula for calculating the total contrast loss is as follows:

[0099]

[0100] in, This represents the total comparative loss.

[0101] This application provides a Transformer multimodal and multi-vision scene recognition method based on cross-guided fusion. This method achieves accurate alignment of global features across modalities through a mathematically formalized contrastive loss function: First, a first contrastive loss is constructed. First global features of aerial images Using the anchor point, calculate its second global feature relative to all ground images within the same batch. The cosine similarity, and through temperature parameters Adjusting the smoothness of the similarity distribution to make positive sample pairs ( With the corresponding The exponential similarity of the negative sample pairs is maximized within the sum of their exponential similarities; a second contrastive loss is symmetrically constructed. The second global feature of the ground image Using the anchor point, calculate its first global feature relative to all aerial images within the same batch. The cosine similarity is used to achieve alignment constraints from the ground to the flight direction; finally, the contrast losses in the two directions are summed with equal weights and the average is taken to obtain the total contrast loss. It is used to guide the model to learn a consistent global feature representation across modalities.

[0102] The overall objective of IMA is a combination of the above losses, which can be expressed as:

[0103]

[0104] in, This represents the total comparative loss.

[0105] In this way, the alignment between real aerial and ground image pairs is enhanced, and the aerial-ground correlation of either modality is ensured.

[0106] This method first uses a contrastive loss function of the form InfoNCE to explicitly bring positive sample pairs closer together and push negative sample pairs further apart within a batch. This enables efficient learning of a discriminative cross-modal feature space, allowing aerial and ground images of the same scene to cluster in the feature space while separating images from different scenes. Secondly, temperature parameters... The introduction of this feature provides adjustability to the similarity distribution, enabling flexible control over the model's focus on difficult negative samples based on data characteristics, thereby enhancing training stability and convergence efficiency. Finally, a bidirectional symmetric loss design is adopted to ensure balanced alignment strength in both the air-to-ground and ground-to-air directions, avoiding alignment bias caused by the model favoring a particular modality. This lays a symmetric and robust global feature foundation for subsequent multimodal fusion.

[0107] Specifically, the fine-grained alignment method based on token attention calculates the similarity score between a first global feature and a second global feature through a cross-attention mechanism, including: using the first global feature as a query and interacting with a second local image tag using the cross-attention mechanism to generate a first common attention feature; calculating a first similarity score between the first common attention feature and the second local image tag; using the second global feature as a query and interacting with the first local image tag using the cross-attention mechanism to generate a second common attention feature; and calculating a second similarity score between the second common attention feature and the first local image tag.

[0108] This application provides a Transformer multimodal and multi-vision scene recognition method based on cross-guided fusion. This method achieves deep interaction and similarity modeling of cross-modal local features through a cross-attention mechanism: First, the first global feature of the aerial image is used as a query and cross-attention interaction is performed with all second local image markers of the ground image to generate a first common attention feature that integrates the global semantics of the aerial image and the local context of the ground image, and the similarity score between this feature and each second local image marker is calculated; symmetrically, the second global feature of the ground image is used as a query and interacts with all first local image markers of the aerial image to generate a second common attention feature, and its similarity score with each first local image marker is calculated, thereby constructing a bidirectional cross-modal local feature similarity measurement system.

[0109] In the ViT model, class tokens act as global descriptors for images, encoding discriminative information during training. To enhance the model's ability to align discriminative features, key token embeddings are aligned across modalities by comparing attention-weighted image regions with another modality in the joint multimodal feature space.

[0110] First, cross-attention (CA) is used to analyze the global features of aerial images. Add all markers embedded in the corresponding ground image To calculate common attention features :

[0111]

[0112] in, Indicates embedding The elements of the sequence, The cross-attention operation can be represented as:

[0113]

[0114] in, This represents the cross-attention mechanism. , and This represents a learnable projection function. This represents the feature dimension embedded in each token.

[0115] Next, the features of interest in the aerial images are calculated. Embedded with all markers in the corresponding ground image The similarity score.

[0116] This method first uses a cross-attention mechanism to guide local feature interaction with global features, dynamically capturing the complex cross-modal semantic correspondence between aerial and ground images, effectively solving the problem of global-local feature misalignment caused by drastic changes in perspective. Second, the process of generating common attention features integrates the global semantics of the query modality and the local details of the key-value modality, enabling the similarity score to simultaneously reflect global discriminative information and local structural features, thus improving the semantic accuracy of fine-grained alignment. Third, the bidirectional symmetrical design ensures the integrity of similarity measurement in both aerial-to-ground and ground-to-aerial directions, avoiding modal bias that may be caused by unidirectional modeling. Finally, the calculated similarity score provides a reliable ranking basis for subsequent key region selection, enabling the model to accurately locate the most discriminative complementary local regions between modalities.

[0117] Specifically, based on similarity scores, the most relevant third local image markers in the aerial image and the most relevant fourth local image markers in the ground image are selected from the first local image markers and the second local image markers, respectively. This includes: selecting the top k second local image markers from high to low based on the first similarity score as the fourth local image markers; and selecting the top k first local image markers from high to low based on the second similarity score as the third local image markers.

[0118] This application provides a Transformer multimodal and multi-vision scene recognition method based on cross-guided fusion. This method uses bidirectional similarity scores calculated by cross-attention mechanism to screen key local markers: First, according to the first similarity score (similarity between global aerial features and local ground markers) sorted from high to low, the second local image markers of the top k ground images are selected as the third local image markers most relevant to the aerial images; symmetrically, according to the second similarity score (similarity between global aerial features and local aerial markers) sorted from high to low, the first local image markers of the top k aerial images are selected as the fourth local image markers most relevant to the ground images, thereby constructing a candidate set of cross-modal local key regions.

[0119] Based on the generated similarity choose The top k most relevant embeddings are denoted as In a similar way, it can be obtained Most relevant embeddings .

[0120] This method first uses a similarity score ranking mechanism to accurately locate the local region that best matches the global semantics of another modality, effectively filtering out a large amount of background noise irrelevant to the task and achieving attention focus from coarse-grained global to fine-grained local. Second, the adjustability of the k value provides a flexible selection mechanism that can adaptively control the number of key regions selected according to the complexity of different datasets, balancing alignment accuracy and computational efficiency. Third, the bidirectional symmetrical selection strategy ensures that the aviation and ground modalities retain their most discriminative local features, laying an equal and complementary feature foundation for subsequent weighted alignment. Finally, the selection process is entirely based on differentiable similarity calculation, which can be seamlessly embedded into the end-to-end training framework and jointly optimized with contrastive loss, so that the selected key regions are continuously refined as the model is trained.

[0121] Specifically, the third and fourth local image labels are weighted and aligned through contrastive learning, and the total fine-grained loss is calculated. This includes: weighting and summing the selected third and fourth local image labels based on their contribution weights to the first and second global features in the last self-attention layer, to obtain the first fine-grained loss for the ground image and the second fine-grained loss for the aerial image, respectively; the first fine-grained loss is calculated using the following formula:

[0122]

[0123] in, This represents the first fine-grained loss. Indicates the fourth local image marker. Indicates the third local image marker. The second temperature parameter is represented; the second fine-grained loss is calculated using the following formula:

[0124]

[0125] in, This represents the second fine-grained loss.

[0126] This application provides a Transformer multimodal and multi-vision scene recognition method based on cross-guided fusion. This method achieves accurate alignment of cross-modal local key features through a bidirectional contrastive loss function: firstly, a first fine-grained loss is constructed. Weighted local marker features selected from aerial images As an anchor point, its weighted local marker features are calculated compared with those selected from all ground images within the same batch. Cosine similarity between them, and through temperature parameters Adjusting the similarity distribution to ensure that positive sample pairs (those selected from real paired images) are... With the corresponding The exponential similarity of the negative sample pairs is maximized as a percentage of the sum of their exponential similarities; a second fine-grained loss is symmetrically constructed. Weighted local labeling features of ground images As an anchor point, its weighted local labeling features are calculated relative to all aerial images within the same batch. The cosine similarity between them enables fine-grained alignment constraints from the ground to the air direction.

[0127] This method first focuses on the top-k most relevant local labels selected from the cross-attention mechanism, extending contrastive learning from global features to key local regions. This achieves accurate alignment of fine-grained semantic units across modalities, effectively solving the problem of local feature misalignment caused by drastic changes in perspective between aerial and ground images. Second, attention contribution weights are introduced to perform a weighted summation of the selected local labels, giving higher importance to more discriminative image sub-regions during alignment, enhancing the task orientation and noise resistance of fine-grained alignment. Third, a bidirectional symmetrical loss design is adopted to ensure balanced alignment strength of local features in both aerial-to-ground and ground-to-aerial directions, avoiding alignment bias caused by model favoring one modality. Finally, temperature parameters... The introduction of this feature provides adjustability to the similarity distribution, enabling flexible control over the model's attention to difficult local negative samples based on data characteristics, thereby further improving the robustness and convergence efficiency of fine-grained alignment.

[0128] Specifically, the third local image label and the fourth local image label are weighted and aligned through contrastive learning, and the first fine-grained loss, the second fine-grained loss and the total fine-grained loss are calculated. It also includes: the first fine-grained loss and the second fine-grained loss are bidirectionally aligned through contrastive learning to obtain the total fine-grained loss. The formula for calculating the total fine-grained loss is as follows:

[0129]

[0130] in, This represents the total fine-grained loss.

[0131] This application provides a Transformer multimodal and multi-vision scene recognition method based on cross-guided fusion. This method sums the fine-grained losses in two directions with equal weights and takes the average to obtain the total fine-grained loss. It is used to guide the model in learning consistent and discriminative representations of key local regions across modalities.

[0132] According to the China and The weighted sums of these values ​​are compared bidirectionally based on their varying importance in identification, leading to a total fine-grained loss: The total fine-grained loss is calculated using the following formula:

[0133]

[0134] in, This represents the total fine-grained loss.

[0135] Specifically, the aligned first and second global features are fused to obtain fused features. This includes: projecting the first and second global features onto a normalized multimodal feature space using a mapping function, and then summing and aggregating the results to obtain the fused features. The formula for calculating the fused features is as follows:

[0136]

[0137] in, Indicates fusion characteristics, and Represents a mapping function. Indicates the first global feature. This represents the second global feature.

[0138] This application provides a Transformer multimodal and multi-vision scene recognition method based on cross-guided fusion. The method first inputs the aligned first global features of the aerial image and the second global features of the ground image into a mapping function, and projects them into a normalized shared multimodal feature space to eliminate the dimensional differences and distribution shifts between the two modal features. Then, the projected two modal features are summed and aggregated element by element to generate fused features that simultaneously contain global semantic information from both the aerial and ground perspectives, providing a joint representation basis for subsequent classification and recognition.

[0139] To fuse the multimodal features of aerial and ground images, we first project global representations of both modalities. and Through mapping function and They are then transformed into a normalized multimodal feature space. Next, the... Fusion features of individual sample pairs Aggregation can be achieved by summing the projected representations:

[0140]

[0141] in, Indicates fusion characteristics, and Represents a mapping function. Indicates the first global feature. This represents the second global feature.

[0142] After fusion, fusion features By classifier The process enables the model to utilize complementary features for final label prediction. The model is optimized using cross-entropy (CE) classification loss.

[0143]

[0144] in, Indicates the first Predicted label for each sample, This represents the total number of samples.

[0145] Furthermore, the CE loss was optimized to learn discriminative global features for each modality:

[0146] ;

[0147] ;

[0148] in, and Representing the predicted labels for different modalities, and This represents the corresponding classifier.

[0149] The final classification objective can be expressed as:

[0150]

[0151] in, and This represents the weight parameters of each component.

[0152] This method first projects bimodal features onto a normalized shared feature space using a mapping function, effectively eliminating the inconsistency in feature distribution between aerial and ground images due to differences in imaging mechanisms and viewpoints, thus laying a solid numerical foundation for subsequent summation and aggregation. Second, by employing summation and aggregation instead of simple feature concatenation, it avoids an explosive increase in feature dimensions, reduces the number of model parameters and computational complexity, and improves training and inference efficiency. Finally, instance-level and fine-grained dual cross-modal alignment is completed before feature fusion, ensuring that the projected bimodal features have a good semantic correspondence in the shared space. Summation and aggregation can achieve organic complementarity of information rather than simple superposition, significantly improving the representational and discriminative power of the fused features.

[0153] Specifically, a total objective function is constructed based on the total contrast loss, the total fine-grained loss, and the total classification loss, including: obtaining the total objective function by weighted summation of the total classification loss, the total contrast loss, and the total fine-grained loss.

[0154] This application provides a Transformer multimodal and multi-vision scene recognition method based on cross-guided fusion. This method weights and sums the total classification loss, total contrast loss, and total fine-grained loss: the total classification loss is used to constrain the model's classification accuracy for fused features and single-modal features, the total contrast loss is used to guide instance-level cross-modal global feature alignment, and the total fine-grained loss is used to guide token-level cross-modal local key region alignment. By assigning corresponding weight coefficients to each loss term, the model can simultaneously optimize classification and discrimination performance, global consistency, and local semantic association during training, and finally learn a multimodal joint representation that has both discriminative power and alignment accuracy.

[0155] The training objective of the model consists of three parts, including , and classification loss By optimizing this objective, the two losses can complement each other to promote the learning of effective multimodal representations, thereby improving recognition performance. The overall objective can be expressed as:

[0156]

[0157] in, Describe the overall objective function. and This represents the weighted parameters for each component.

[0158] This method first achieves collaborative learning between the discrimination and alignment tasks by jointly optimizing the classification loss and two contrastive losses. This allows the model to spontaneously enhance the semantic consistency of cross-modal features while pursuing classification accuracy, avoiding performance bias caused by single-task optimization. Second, the total contrastive loss and total fine-grained loss constrain cross-modal alignment from two dimensions: global instance level and local token level, respectively, forming a complementary supervision signal with coarse and fine granularities. This effectively solves the problem of multi-scale feature misalignment caused by drastic changes in perspective between aerial and ground images. Third, adjustable weight coefficients provide a flexible trade-off mechanism, which can adaptively adjust the contribution intensity of each loss term according to the characteristics of different datasets or the focus of different tasks, enhancing the model's generalization ability and scene adaptability. Finally, the multi-loss joint optimization framework has good scalability, making it easy to introduce more auxiliary supervision signals or constraints in the future to further improve the performance ceiling of multimodal fusion recognition.

[0159] Figure 4 A connection diagram of a Transformer multimodal multi-vision scene recognition system based on cross-guided fusion provided in this application is shown below. Figure 3 As shown, this embodiment provides a Transformer multimodal multi-vision scene recognition system based on cross-guided fusion. The recognition system includes:

[0160] The data acquisition module is used to construct paired heterogeneous datasets for multimodal and multi-view remote sensing scene recognition. The paired heterogeneous datasets include aerial images of the target area and ground images paired with the aerial images.

[0161] The feature extraction module is connected to the data acquisition module. The feature extraction module is used to extract features from pairs of heterogeneous datasets using a dual-stream visual Transformer network to obtain the first global features and the first local image label of the aerial image, as well as the second global features and the second local image label of the ground image.

[0162] The instance-level alignment module is connected to the feature extraction module. The instance-level alignment module is used to align the first global feature and the second global feature based on the instance-level multimodal alignment method, calculate the first contrast loss and the second contrast loss based on the first global feature and the second global feature, and calculate the total contrast loss using the first contrast loss and the second contrast loss.

[0163] The fine-grained alignment module, connected to the instance-level alignment module, is used for a token-attention-based fine-grained alignment method. It calculates the similarity score between the first global feature and the second global feature through a cross-attention mechanism, and selects the most relevant third local image label from the first local image label and the most relevant fourth local image label from the ground image from the first local image label and the second local image label, respectively. Through contrastive learning, the third local image label and the fourth local image label are weighted and aligned, and the total fine-grained loss is calculated.

[0164] The feature fusion module is connected to the fine-grained alignment module. The feature fusion module is used to fuse the aligned first global features and second global features to obtain fused features, calculate the single-modal classification loss, process the fused features through a classifier and combine them with the single-modal classification loss to obtain the total classification loss.

[0165] The classification and recognition module is connected to the feature fusion module. The classification and recognition module is used to construct an overall objective function based on the overall contrast loss, overall fine-grained loss and overall classification loss. The overall objective function is used to train a multimodal and multi-vision scene classifier. The fused features are input into the multimodal and multi-vision scene classifier to obtain the multimodal and multi-vision scene recognition results.

[0166] This application provides a Transformer multimodal and multi-vision scene recognition system based on cross-guided fusion. The system comprises a data acquisition module, a feature extraction module, an instance-level alignment module, a fine-grained alignment module, a feature fusion module, and a classification and recognition module connected sequentially. The data acquisition module constructs a paired heterogeneous dataset containing paired aerial and ground images. The feature extraction module uses a dual-stream visual Transformer network to extract the first global feature and first local image label from the aerial image, and the second global feature and second local image label from the ground image. The instance-level alignment module performs instance-level alignment of the first and second global features based on contrastive learning and calculates the total contrastive loss. The fine-grained alignment module uses a cross-attention mechanism to select the most relevant third and fourth local image labels from the two modalities, performs fine-grained alignment through weighted contrastive learning, and calculates the total fine-grained loss. The feature fusion module fuses the aligned dual-modal global features to obtain fused features, which are then processed by a classifier and combined with a single-modal classification loss to obtain the total classification loss. The classification and recognition module constructs a total objective function based on the total classification loss to train a multimodal and multi-vision scene classifier, and finally inputs the fused features to output the scene recognition result.

[0167] The system's modules are clearly defined in function and division of labor, and are sequentially connected to form a complete data processing pipeline. From data input to feature extraction, multi-level alignment, feature fusion, and final classification and recognition, the process logic is rigorous and easy to implement and maintain. Secondly, the instance-level alignment module and the fine-grained alignment module form a cascaded structure. First, global comparative learning is used to narrow the cross-modal semantic gap, and then fine alignment is achieved through local key region screening. The complementary supervision mechanism of coarse and fine granularity effectively solves the problem of multi-scale feature misalignment caused by drastic changes in perspective between aerial and ground images. Thirdly, the connection relationship between modules reflects the unidirectional transmission of data flow and the progressive refinement of features, ensuring that high-quality features after alignment are transmitted to the fusion module, avoiding information degradation and noise accumulation. Finally, the system adopts an end-to-end trainable architecture. The loss functions of each module are jointly optimized through the overall objective function, realizing collaborative learning of cross-modal alignment and classification and recognition tasks, significantly improving the robustness and accuracy of multi-modal fusion recognition in complex remote sensing scenarios.

[0168] The effectiveness of our proposed method was validated through experiments on two widely used benchmarks: AiRound and CV-BrCT. Comparisons with several state-of-the-art fusion recognition methods demonstrate that our method achieves superior recognition performance.

[0169] All experiments were conducted using the PyTorch framework on a workstation equipped with four RTX 4090 GPUs. Images from the multimodal dataset were uniformly resized to 224×224 pixels to fit the model's input size. In the experiments, ViT (VisionTransformer) was chosen as the image encoder, and weights pre-trained on ImageNet were utilized for each task. The AdamW optimizer was used to fine-tune all models, with a weight decay of 0.0005. The optimizer's betas parameters were 0.9 and 0.999, respectively. The initial learning rate was set to η0 = 8 × 10⁻⁵. During training, a cosine learning rate decay strategy was used to gradually reduce the learning rate.

[0170] Parameters λ1 and λ2 were both set to 0.3. In the total loss function Ltotal, the weights α and β were both fixed at 0.5 in the experiments. The hyperparameter k in TFMA was set to 2. Finally, considering both network architecture and performance, the total number of training epochs for each task was set to 50, and the minimum batch size for bimodal training was set to 64.

[0171] The final experimental results are shown in Table 1. Table 1 compares the results of the multimodal scene recognition task on the AIROUND and CV-BRCT datasets using different comparison methods. The OA (%) datasets at different training ratios (20%, 50%, 80%) are presented. Bold and underlined text indicate the best and second-best results.

[0172] Table 1. Comparison of recognition results under different training ratios (20%, 50%, 80%)

[0173]

[0174] The results were compared with several other methods on several challenging multimodal, multi-view remote sensing scene recognition datasets, including MVCNN, CBV-WCF, and TEB-Suq. Additionally, several standard networks based on different fusion strategies of CNN and Transformer were included, such as VGG16, ResNet50, DenseNet, SENet, SKNet, SMT, and VIT. Overall accuracy (OA) was chosen as the criterion for recognition performance. Furthermore, a confusion matrix was proposed to represent the degree of confusion for each class. Notably, the comparison results include recognition performance based on the original segmentation dataset (training to testing ratio of 4:1) and more challenging limited data scenarios, where the training data ratio was set to 50% and 20%, respectively, with the remainder used for testing.

[0175] Experimental results in a single-modal scenario: To comprehensively evaluate this method, experimental results on a single-modal model are first presented as a benchmark for evaluating the proposed multimodal fusion strategy. The experimental results are shown in Table 2. The experimental results show that when relying solely on single-modal data, the recognition performance has significant limitations, highlighting the urgent need for multimodal fusion.

[0176] Table 2. Comparison of unimodal recognition performance on CV-BrCT and AiRound datasets

[0177]

[0178] Experimental Results on the Airound Dataset: Table 1 presents the comparison results for the Airound dataset. Compared to other methods, the proposed CGFT model achieves the highest OA (Objective Automated Assessment) of 94.77%, outperforming the second-best method, TEB-Suq, by 2.61%. Furthermore, this method significantly improves recognition performance compared to single-modal models. This advantage is attributed to its strength in complementary learning from different modalities. By leveraging the combined advantages of multiple modalities, this method effectively addresses the limitations of single-modal methods in dealing with ambiguity in complex scenes. The superior recognition performance demonstrates that existing fusion models struggle to resolve ambiguities caused by changes in object perspective and mode, limiting their effectiveness in complex multi-view scenarios. Moreover, even with only 50% and 20% of the training data, this method still achieves the highest OA, at 87.76% and 91.85%, respectively. This highlights the robustness of CGFT to limited training data and its applicability to complex real-world scenes. It is worth noting that its performance with 50% training data matches that of existing methods that rely on 80% training data, highlighting its efficiency in learning discriminative features and cross-modal alignment.

[0179] Figure 5 This is a schematic diagram of the confusion matrices for the Baseline and CGFT models on the AiRound dataset.

[0180] To evaluate the classification performance on this dataset, the confusion matrix of the baseline and the proposed CGFT method are further presented, such as... Figure 5 As shown in the diagram. In the experiment, "VIT-B+Concate" was used as the baseline method, representing a VIT model with simple feature concatenation for multimodal fusion. From the results, it can be observed that the most significant misclassification occurred between the "stadium" and "park" samples, which may be due to the high visual similarity between the samples.

[0181] Experimental results on the CV-BrCT dataset: For the CV-BrCT dataset, our proposed method outperforms all comparable methods in the multimodal fusion recognition task. The experimental results are shown in Table 1. When the proportion of training data is set to 80%, 50%, and 20%, respectively, the proposed CGFT model achieves state-of-the-art performance of 91.12%, 86.04%, and 82.93%. The proposed multi-channel alignment strategy effectively suppresses channel-specific noise while emphasizing the differentiation of cross-channel knowledge. This highlights CGFT's ability to explore cross-modal correlations and achieve effective feature fusion of multimodal remote sensing scene images.

[0182] Figure 6This is a schematic diagram of the confusion matrices for the Baseline and CGFT models on the CV-BrCT dataset.

[0183] The confusion matrix on the CV-BrCT dataset is as follows: Figure 6 As shown, the most severe misclassification occurred between the "hospital" and "school" samples. Specifically, the CGFT model was observed to outperform the baseline method, particularly in tasks requiring fine-grained semantic understanding.

[0184] Comprehensive ablation studies were conducted to demonstrate the effectiveness of different components in the CGFT model, primarily the proposed instance-by-instance multimodal alignment (IMA) module and the token attention fine-grained alignment (TAMA) module. In these experiments, each module undergoing ablation was individually removed to evaluate their impact on overall recognition performance.

[0185] Impact of IMA: The ablation experiment results of the IMA module are shown in Table 3. The purpose of the IMA module is to reduce the differences between paired global representations, enabling the model to learn more task-related knowledge from different channels. It was observed that applying ℒIMA loss resulted in a 1.1% and 2.5% improvement in recognition performance on the two datasets, respectively, compared to the baseline method. This indicates that cross-channel alignment can improve multi-channel information fusion by effectively capturing target information from different perspectives. Furthermore, compared to the model using only ℒTAMA loss, IMA achieved performance improvements ranging from 0.12% to 1.84%.

[0186] Table 3. Ablation Experiment Results

[0187]

[0188] Impact of TAMA: Table 3 presents the ablation study of the TAFA contrastive loss. The TAFA module aims to explore cross-modal relationships at the key token level, facilitating more effective fine-grained multimodal alignment. Simultaneously, TAFA further guides the model to focus on the most relevant cross-modal complementarity details. Specifically, experiments show that the model trained based on the TAFA objective function significantly outperforms the baseline method in overall accuracy (OA), achieving a substantial performance improvement. This result validates the effectiveness of incorporating the TAFA contrastive loss into the model. Furthermore, ablation analysis demonstrates that including fine-grained multimodal alignment contributes to effective feature fusion, particularly in complex remote sensing image scenarios requiring fine-grained semantic understanding.

[0189] Impact of Different k Values: Ablation experiments were also conducted on k values ​​varying between 1 and 3 in TAFA. These experiments aimed to examine the impact of different k values ​​on the overall recognition performance of the model. As shown in Table 3, different k values ​​affect fine-grained contrastive multimodal alignment; the model achieves optimal recognition performance when k is set to 2. A decrease in recognition performance was observed when using larger or smaller k values, which may be due to insufficient exploration of fine-grained relevant label embeddings. This indicates that the model encounters challenges in effectively coordinating different modalities.

Claims

1. A Transformer multimodal and multi-vision scene recognition method based on cross-guided fusion, characterized in that, The identification method includes: Construct a pairwise heterogeneous dataset for multimodal, multi-view remote sensing scene recognition, wherein the pairwise heterogeneous dataset includes aerial images of the target area and ground images paired with the aerial images; A dual-stream visual Transformer network is used to extract features from the paired heterogeneous datasets to obtain the first global features and first local image labels of the aerial images, as well as the second global features and second local image labels of the ground images; Based on the instance-level multimodal alignment method, the first global feature and the second global feature are aligned, and the first contrast loss and the second contrast loss are calculated based on the first global feature and the second global feature. The total contrast loss is calculated using the first contrast loss and the second contrast loss. The fine-grained alignment method based on token attention calculates the similarity score between the first global feature and the second global feature through a cross-attention mechanism. Based on the similarity score, it selects the most relevant third local image tag in the aerial image and the most relevant fourth local image tag in the ground image from the first local image tag and the second local image tag, respectively. Through contrastive learning, it performs weighted alignment of the third local image tag and the fourth local image tag and calculates the total fine-grained loss. The aligned first global feature and second global feature are fused to obtain a fused feature. The single-modal classification loss is calculated. The fused feature is processed by a classifier and combined with the single-modal classification loss to obtain the total classification loss. A total objective function is constructed based on the total contrast loss, the total fine-grained loss, and the total classification loss. A multimodal multi-vision scene classifier is trained using the total objective function. The fused features are then input into the multimodal multi-vision scene classifier to obtain the multimodal multi-vision scene recognition result.

2. The Transformer multimodal and multi-vision scene recognition method based on cross-guided fusion according to claim 1, characterized in that, The instance-level multimodal alignment method aligns the first global feature and the second global feature, calculates a first contrast loss and a second contrast loss based on the first global feature and the second global feature, and calculates the total contrast loss using the first contrast loss and the second contrast loss, including: The instance-level multimodal alignment method is used to calculate the first contrast loss between the first global feature and the second global feature within the same batch, when the first global feature is used as the anchor point; the formula for calculating the first contrast loss is: ； in, This represents the first comparative loss. Represents cosine similarity. This represents the first global feature. This represents the second global feature. Indicates the first temperature parameter. This indicates the number of samples in the batch. Indicates the sample index within the same batch; The instance-level multimodal alignment method is used to calculate the second contrast loss between the second global feature and the first global feature within the same batch, when the second global feature is used as the anchor point; the formula for calculating the second contrast loss is: ； in, This represents the second comparative loss.

3. The Transformer multimodal and multi-vision scene recognition method based on cross-guided fusion according to claim 2, characterized in that, The calculation of the total contrast loss using the first contrast loss and the second contrast loss includes: The total contrast loss is obtained by weighted summing of the first contrast loss and the second contrast loss. The formula for calculating the total contrast loss is as follows: ； in, This represents the total contrast loss.

4. The Transformer multimodal and multi-vision scene recognition method based on cross-guided fusion according to claim 1, characterized in that, The fine-grained alignment method based on token attention calculates the similarity score between the first global feature and the second global feature through a cross-attention mechanism, including: Using the cross-attention mechanism, the first global feature is used as a query to interact with the second local image marker to generate a first common attention feature; the first similarity score between the first common attention feature and the second local image marker is calculated. Using the cross-attention mechanism, the second global feature is used as a query to interact with the first local image label to generate a second common attention feature; the second similarity score between the second common attention feature and the first local image label is calculated.

5. The Transformer multimodal and multi-vision scene recognition method based on cross-guided fusion according to claim 4, characterized in that, The step of filtering the most relevant third local image marker in the aerial image and the most relevant fourth local image marker in the ground image from the first local image marker and the second local image marker based on the similarity score includes: Based on the first similarity score, the top k second local image markers from high to low are selected as the fourth local image markers; The top k first local image markers, selected from high to low based on the second similarity score, are used as the third local image markers.

6. The Transformer multimodal and multi-vision scene recognition method based on cross-guided fusion according to claim 1, characterized in that, The step of weighted alignment of the third local image label and the fourth local image label through contrastive learning, and calculation of the total fine-grained loss, includes: Based on the contribution weights of the third local image marker and the fourth local image marker to the first global feature and the second global feature in the last self-attention layer, the selected third local image marker and the fourth local image marker are weighted and summed to obtain the first fine-grained loss of the aerial image and the second fine-grained loss of the ground image, respectively. The first fine-grained loss is calculated using the following formula: ； in, This represents the first fine-grained loss. This indicates the third local image marker. This represents the fourth local image marker. Indicates the second temperature parameter; The second fine-grained loss is calculated using the following formula: ； in, This represents the second fine-grained loss.

7. The Transformer multimodal and multi-vision scene recognition method based on cross-guided fusion according to claim 1, characterized in that, The step of weighted alignment of the third local image label and the fourth local image label through contrastive learning, and calculation of the total fine-grained loss, further includes: The first fine-grained loss and the second fine-grained loss are bidirectionally aligned through comparative learning to obtain the total fine-grained loss, and the formula for calculating the total fine-grained loss is as follows; ； in, This represents the total fine-grained loss.

8. The Transformer multimodal and multi-vision scene recognition method based on cross-guided fusion according to claim 1, characterized in that, The step of fusing the aligned first global feature and the second global feature to obtain the fused feature includes: The first global feature and the second global feature are projected onto a normalized multimodal feature space using a mapping function, and then summed and aggregated to obtain the fused feature. The calculation formula for the fused feature is as follows: ； in, This indicates the fusion feature. and Represents the mapping function, This represents the first global feature. This represents the second global feature.

9. The Transformer multimodal and multi-vision scene recognition method based on cross-guided fusion according to claim 1, characterized in that, The construction of the overall objective function based on the total contrast loss, the total fine-grained loss, and the total classification loss includes: The total objective function is obtained by weighted summing of the total classification loss, the total contrast loss, and the total fine-grained loss.

10. A Transformer multimodal multi-vision scene recognition system based on cross-guided fusion, wherein the recognition system is applied to the recognition method according to any one of claims 1 to 9, and the recognition system comprises: The data acquisition module is used to construct the paired heterogeneous dataset for the multimodal multi-view remote sensing scene recognition. The paired heterogeneous dataset includes the aerial image of the target area and the ground image paired with the aerial image. The feature extraction module is connected to the data acquisition module. The feature extraction module is used to extract features from the paired heterogeneous datasets using the dual-stream visual Transformer network to obtain the first global feature and the first local image label of the aerial image, and the second global feature and the second local image label of the ground image. An instance-level alignment module is connected to the feature extraction module. The instance-level alignment module is used to align the first global feature and the second global feature based on the instance-level multimodal alignment method, calculate the first contrast loss and the second contrast loss based on the first global feature and the second global feature, and calculate the total contrast loss using the first contrast loss and the second contrast loss. A fine-grained alignment module, connected to the instance-level alignment module, is used in the token-attention-based fine-grained alignment method. It calculates the similarity score between the first global feature and the second global feature through the cross-attention mechanism, and based on the similarity score, selects the most relevant third local image marker from the first local image marker and the most relevant fourth local image marker from the second local image marker. It then performs weighted alignment of the third local image marker and the fourth local image marker through contrastive learning, and calculates the total fine-grained loss. The feature fusion module is connected to the fine-grained alignment module. The feature fusion module is used to fuse the aligned first global feature and the second global feature to obtain the fused feature, calculate the single-modality classification loss, process the fused feature through the classifier and combine it with the single-modality classification loss to obtain the total classification loss. The classification and recognition module is connected to the feature fusion module. The classification and recognition module is used to construct the total objective function based on the total contrast loss, the total fine-grained loss and the total classification loss, train the multimodal multi-vision scene classifier using the total objective function, and input the fused features into the multimodal multi-vision scene classifier to obtain the multimodal multi-vision scene recognition result.

Images