An interactive multi-modal classification method and apparatus

Abstract

The application discloses an interactive multi-modal classification method and device, and relates to the technical field of remote sensing image processing. The method is used for solving the problem that the existing semi-supervised multi-modal classification method is difficult to simultaneously realize efficient utilization of multi-modal complementary information, unified representation of local and overall features, and iterative updating of high-quality pseudo labels. The method takes a hyperspectral image and a synthetic aperture radar image registered in the same area as input, and under the condition that a small amount of labeled samples and a large amount of unlabeled samples jointly participate, realizes multi-modal remote sensing image classification through shared double-branch convolution feature extraction, sample set exclusive private feature extraction, double-flow scale interactive encoding, local-global collaborative fusion, multi-output classification and graph discriminative pseudo label iterative updating.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, and more specifically to an interactive multimodal classification method and apparatus. Background Technology

[0002] Hyperspectral images can provide continuous and detailed spectral information, which has significant advantages in the identification of ground object material properties. Synthetic aperture radar images can provide ground object scattering characteristics, structural features and geometric information. The two are highly complementary. Joint classification of hyperspectral images and synthetic aperture radar images is an important technical direction for improving the accuracy and reliability of ground object identification in complex scenes.

[0003] Existing joint classification methods using hyperspectral and synthetic aperture radar mostly employ deep learning frameworks for feature extraction and fusion, and combine semi-supervised learning strategies to alleviate the problem of scarce labeled samples. However, existing methods still have significant drawbacks: multimodal features are simply concatenated or enhanced along a single path, failing to achieve efficient cross-modal interaction and local-global feature fusion; public and private features lack effective decoupling constraints, resulting in high feature redundancy; pseudo-label generation often relies on fixed thresholds or simple label propagation, without incorporating sample distribution reliability, boundary-aware screening, and class adaptation strategies, easily introducing low-quality pseudo-labels, leading to insufficient classification stability and generalization ability under small sample conditions.

[0004] In summary, existing semi-supervised multimodal classification methods struggle to simultaneously achieve efficient utilization of multimodal complementary information, unified representation of local and global features, and high-quality pseudo-label iterative updates. Therefore, this invention proposes a corresponding technical solution to address these issues. Summary of the Invention

[0005] This invention provides an interactive multimodal classification method and apparatus to address the problem that existing semi-supervised multimodal classification methods struggle to simultaneously achieve efficient utilization of multimodal complementary information, unified representation of local and global features, and high-quality pseudo-label iterative updates.

[0006] This invention provides an interactive multimodal classification method, comprising:

[0007] The registered initial spectral image and the initial synthetic aperture radar image are subjected to dimensionality reduction and standardization. The sample positions are divided according to the shared label map, and hyperspectral image patches and synthetic aperture radar image patches are extracted with the corresponding positions as the center to construct a labeled training sample set, an unlabeled sample set, and a labeled test sample set.

[0008] The hyperspectral image patch and the synthetic aperture radar image patch are shared by a dual-branch convolutional network to obtain dual-modal common features respectively; the labeled training sample set is processed by a first private convolutional branch and the unlabeled sample set is processed by a second private convolutional branch to obtain the exclusive private features of the corresponding sample set respectively.

[0009] Based on the bimodal common features and global interactive fusion features, the final bimodal fusion features and the classification results of the three branches are obtained sequentially;

[0010] The labeled training sample set and the unlabeled sample set are jointly optimized based on bimodal common features, corresponding sample set exclusive private features, global interactive fusion features, final bimodal fusion features and classification results of three branches, and five types of loss functions are jointly optimized to construct a strongly constrained semi-supervised training mechanism.

[0011] Based on the final dual-modal fusion features, a sample nearest neighbor graph is constructed. The distribution reliability of the unlabeled sample set is identified by a graph discriminator. A boundary-aware strategy is used to select high-value candidate samples from the unlabeled sample set. The high-value candidate samples that reach the class adaptation threshold and meet the confidence condition are assigned pseudo-labels. The high-value candidate samples with pseudo-labels are iteratively added to the labeled training sample set to form a semi-supervised pseudo-label update mechanism.

[0012] This invention provides an interactive multimodal classification device, comprising:

[0013] The first construction unit is used to perform dimensionality reduction and standardization processing on the registered initial spectral image and the initial synthetic aperture radar image, divide the sample positions according to the shared label map, and extract hyperspectral image blocks and synthetic aperture radar image blocks with the corresponding positions as the center to construct a labeled training sample set, an unlabeled sample set, and a labeled test sample set.

[0014] The first obtaining unit is used to obtain dual-modal common features from the hyperspectral image patch and the synthetic aperture radar image patch through a shared dual-branch convolutional network; the labeled training sample set is obtained through a first private convolutional branch and the unlabeled sample set is obtained through a second private convolutional branch to obtain the exclusive private features of the corresponding sample set.

[0015] The second obtaining unit is used to obtain the final bimodal fusion feature and the classification results of the three branches in sequence based on the bimodal common features and global interactive fusion features;

[0016] The second construction unit is used to jointly optimize five types of loss functions based on the labeled training sample set and the unlabeled unlabeled sample set, and construct a strongly constrained semi-supervised training mechanism.

[0017] The update unit is used to construct a sample nearest neighbor graph based on the final dual-modal fusion features, identify the distribution reliability of the unlabeled sample set through a graph discriminator, select high-value candidate samples in the unlabeled sample set using a boundary-aware strategy, assign pseudo-labels to the high-value candidate samples that have reached the class adaptation threshold and meet the confidence conditions, and iteratively add the high-value candidate samples with pseudo-labels to the labeled training sample set to form a semi-supervised pseudo-label update mechanism.

[0018] This invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform any of the above-described interactive multimodal classification methods.

[0019] This invention provides a computer device, which includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs any of the interactive multimodal classification methods described above.

[0020] This invention provides an interactive multimodal classification method and apparatus, based on a semi-supervised classification method for dual-modal remote sensing images. Through standardized sample construction, multi-branch feature extraction, dual-stream interactive fusion, joint loss constraints, and pseudo-label iterative updates, it can effectively improve the classification accuracy, stability, and generalization ability of hyperspectral and synthetic aperture radar images under conditions of a small number of labeled samples, and has the following beneficial effects:

[0021] 1) By performing dimensionality reduction and standardization on the registered images and extracting bimodal image patches, labeled training sample sets, unlabeled sample sets, and labeled test sample sets are constructed. The data preprocessing is standardized, and the sample structure is unified, providing a stable and reliable data foundation for subsequent network training. A shared dual-branch convolutional network is used to extract bimodal common features, and the first and second private convolutional branches are used to obtain the exclusive private features of the corresponding sample sets. This enables parallel learning of common discriminative information and sample-specific information, reducing feature redundancy and improving the expressive power and discriminative power of multimodal features.

[0022] 2) Based on the common features of the two modes, cross-modal interaction is performed to obtain global interactive fusion features, and further, the final two-modal fusion features and the classification results of the three branches are obtained, realizing the synergistic utilization of local features and global correlation information, and fully exploring the complementary advantages of spectral and scattering information. This embodiment of the invention optimizes five types of loss functions based on multiple features and outputs, constructing a strongly constrained semi-supervised training mechanism, effectively improving the model's discrimination ability and training stability under small sample conditions.

[0023] 3) Based on the final dual-modal fusion features, the embodiments of the present invention construct a sample nearest neighbor graph. Through graph discriminator, boundary awareness strategy and class adaptive threshold, high-value candidate samples are screened and pseudo-labels are assigned. Reliable samples are iteratively added to the labeled training sample set to form an efficient semi-supervised pseudo-label update mechanism, which significantly improves the utilization rate of unlabeled samples and avoids low-quality pseudo-labels interfering with training. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of an interactive multimodal classification method provided in an embodiment of the present invention;

[0026] Figure 2 A schematic diagram of a semi-supervised multimodal classification network structure for hyperspectral and synthetic aperture radar image classification provided in an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram of an interactive multimodal classification device provided in an embodiment of the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Figure 1 This is a schematic diagram of an interactive multimodal classification method provided in an embodiment of the present invention; the following is in conjunction with... Figure 1 This article provides a detailed description of an interactive multimodal classification method provided by an embodiment of the present invention. For example... Figure 1As shown, the method includes the following steps:

[0030] Step 101: Dimensionality reduction and standardization are performed on the registered initial spectral image and the initial synthetic aperture radar image. The sample positions are divided according to the shared label map, and hyperspectral image blocks and synthetic aperture radar image blocks are extracted with the corresponding positions as the center to construct a labeled training sample set, an unlabeled sample set, and a labeled test sample set.

[0031] Step 102: The hyperspectral image patch and the synthetic aperture radar image patch obtain dual-modal common features through a shared dual-branch convolutional network; the labeled training sample set obtains exclusive private features for the corresponding sample set through the first private convolutional branch and the unlabeled sample set obtains exclusive private features for the corresponding sample set through the second private convolutional branch.

[0032] Step 103: Based on the bimodal common features and global interactive fusion features, the final bimodal fusion features and the classification results of the three branches are obtained sequentially;

[0033] Step 104: The labeled training sample set and the unlabeled sample set are jointly optimized based on the bimodal common features, the exclusive private features of the corresponding sample set, the global interactive fusion features, the final bimodal fusion features and the classification results of the three branches, and a strong constraint semi-supervised training mechanism is constructed.

[0034] Step 105: Construct a sample nearest neighbor graph based on the final dual-modal fusion features, identify the distribution reliability of the unlabeled sample set using a graph discriminator, select high-value candidate samples from the unlabeled sample set using a boundary-aware strategy, assign pseudo-labels to the high-value candidate samples that reach the class adaptation threshold and meet the confidence conditions, and iteratively add the high-value candidate samples with pseudo-labels to the labeled training sample set to form a semi-supervised pseudo-label update mechanism.

[0035] The main body of this method is computer equipment, which reads hyperspectral images, synthetic aperture radar images and label images, and performs the entire process of data preprocessing, sample construction, network training and classification reasoning.

[0036] Before step 101, labeled samples and unlabeled samples are represented by the following formulas:

[0037] (1-1)

[0038] (1-2)

[0039] in, This indicates that there are labeled samples. Indicates the first A hyperspectral image patch, Indicates the first A synthetic aperture radar image patch, Indicates category label , Indicates the number of categories. This indicates the number of labeled samples; Indicates unlabeled samples. Indicates the first A hyperspectral image patch, Indicates the first A synthetic aperture radar image patch, This represents the total number of unlabeled samples;

[0040] To address issues such as band redundancy, inconsistent multimodal scales, and non-standard sample structures in hyperspectral data, step 101 involves uniformly processing the input multimodal remote sensing data to construct standardized network input samples.

[0041] The process involves reading the initial hyperspectral image, the initial synthetic aperture radar image, and the corresponding label image of the region to be classified. It should be noted that the initial hyperspectral image and the initial synthetic aperture radar image belong to the same geographic region and have been spatially accurately registered. The label image is the same ground truth map of land cover categories that is completely aligned with the initial hyperspectral image and the initial synthetic aperture radar image in terms of spatial location. It is used to label the actual land cover category corresponding to each pixel.

[0042] In this embodiment of the invention, the initial hyperspectral image and the initial synthetic aperture radar image can be represented as follows:

[0043] (1-3)

[0044] (1-4)

[0045] in, and These represent the image height and width, respectively. The image here can be either the initial hyperspectral image or the initial synthetic aperture radar image. Indicates the number of hyperspectral bands. This indicates the number of channels in the synthetic aperture radar.

[0046] In practical applications, due to the high number of original bands and data redundancy in hyperspectral images, direct training leads to high computational cost, slow convergence speed, and feature redundancy. Therefore, PCA (Principal Component Analysis) is first performed on the hyperspectral image for dimensionality reduction, which reduces dimensionality and computational cost while preserving the main information.

[0047] Because the initial hyperspectral image has a large number of original bands and high data redundancy, direct training would result in high computational cost, slow convergence speed, and feature redundancy. To reduce redundancy and improve the efficiency of subsequent network training, principal component analysis (PCA) is first performed on the initial hyperspectral image for dimensionality reduction. This reduces the computational complexity of subsequent operations while preserving key information. When the number of original bands exceeds the preset number of principal components... At that time, keep the previous one One principal component. Preferably, The dimensionality reduction process of the initial hyperspectral image can be represented by the following formula:

[0048] (1-5)

[0049] in, This represents the hyperspectral image after dimensionality reduction via principal component analysis. , This represents the initial hyperspectral image of the original input. This indicates the preset target dimension for principal component analysis. This indicates the height (in pixels) of the input remote sensing image. Indicates the width (in pixels) of the input remote sensing image; This represents the total number of bands in the initial hyperspectral image; This represents the actual number of bands in the dimensionality-reduced hyperspectral image. .

[0050] Subsequently, channel-level standardization was performed on the dimensionality-reduced hyperspectral image and the initial synthetic aperture radar image, respectively, to obtain the standardized hyperspectral image and the standardized synthetic aperture radar image. Through the above processing, the numerical differences and dimensional effects between different channels can be eliminated, the data distribution can be unified, and both modal data can satisfy the distribution with a mean of 0 and a variance of 1. The data scale is unified to facilitate subsequent network training and improve the stability of network training.

[0051] In this embodiment of the invention, the reduced hyperspectral image and the initial synthetic aperture radar image can be normalized at the channel level using the following formula:

[0052] (1-6)

[0053] in, This represents the standardized single-channel data, i.e., the standardized hyperspectral image and the standardized synthetic aperture radar image. This represents the original single-channel input data, namely the dimensionality-reduced hyperspectral image and the initial synthetic aperture radar image. This represents the mean of the data for the corresponding channel. This represents the standard deviation of the corresponding channel data.

[0054] It should be noted that, in this embodiment of the invention, if the standardized synthetic aperture radar image is in two-dimensional single-channel form, it is first expanded into a three-dimensional tensor to unify the subsequent processing flow.

[0055] Furthermore, based on the land feature categories marked on the label map, all valid pixels are divided into three categories: labeled samples, unlabeled samples, and test samples. During the division, training samples are randomly selected from each land feature category according to a preset ratio, and it is mandatory to retain at least 5 labeled samples for each land feature category to meet the needs of small sample learning.

[0056] Let the training ratio be... In the embodiments of the present invention, it is preferred that... For the first Class of samples, if the total number of samples in this class is The number of training samples can be determined by the following formula:

[0057] (1-7)

[0058] in, Indicates the first Number of training samples for land-like features This represents the sampling ratio of the training samples, with a preferred value of 0.003. Indicates the first Total number of samples of land features This represents the floor function. This represents the maximum value function, ensuring that there are no fewer than 5 training samples in each class; that is, at least 5 training samples are retained in each class.

[0059] In this embodiment of the invention, non-background samples that are not included in the training samples serve as the source of unlabeled samples and test samples.

[0060] Furthermore, to utilize pixel neighborhood spatial information, a sliding window approach is used to extract bimodal image patches. Specifically, using the pixel positions corresponding to the labeled samples, unlabeled samples, and test samples as centers, a sliding window approach is used to simultaneously extract fixed-size local image patches from the standardized hyperspectral image and the standardized synthetic aperture radar image, respectively, resulting in hyperspectral image patches and synthetic aperture radar image patches. Further, the hyperspectral image patches and synthetic aperture radar image patches corresponding to the same central position are combined to form a bimodal image patch.

[0061] It should be noted that the image boundaries are symmetrically filled before extracting the image patches. For any center pixel position... The corresponding hyperspectral image patches and synthetic aperture radar image patches are represented as follows:

[0062] (1-8)

[0063] (1-9)

[0064] in, Indicates the position of the center pixel The corresponding hyperspectral image patch, Indicates the position of the center pixel The corresponding synthetic aperture radar image patch, This represents the index of the center pixel position of the image patch. This indicates the number of bands in the dimensionality-reduced hyperspectral image. This represents the total number of channels in the input synthetic aperture radar image. The window size represents the image patch size, with a preferred value of 13, meaning the image patch size is [value missing]. Pixels Represents the real number field, with the superscript indicating the dimension of the corresponding tensor.

[0065] Furthermore, a labeled training sample set is constructed using labeled samples and corresponding bimodal image patches, an unlabeled sample set is constructed using unlabeled samples and corresponding bimodal image patches, and a labeled test sample set is constructed using test samples and corresponding bimodal image patches, which serve as input data for subsequent networks.

[0066] In step 102, in order to achieve deep fusion of multimodal complementary information and capture local details and large-scale contextual information at the same time, this embodiment of the invention constructs a dual-stream scale interactive master classification network, including a shared dual-branch convolutional feature extraction unit, a sample set-specific private feature extraction unit, a dual-stream scale interactive coding unit, a local-global collaborative fusion unit, and a multi-output classification unit.

[0067] Specifically, hyperspectral image blocks and synthetic aperture radar image blocks are respectively input into a shared dual-branch convolutional network for feature extraction, and common features of the two modes are extracted to obtain common features of hyperspectral mode and common features of synthetic aperture radar mode.

[0068] In this embodiment of the invention, the shared feature extraction functions of the shared dual-branch convolutional network are respectively... and Then, the common characteristics of hyperspectral modes and the common characteristics of synthetic aperture radar modes can be expressed by the following formulas:

[0069] (2-1)

[0070] (2-2)

[0071] in, Indicates common features of hyperspectral modes. This represents the hyperspectral modality shared feature extraction function of a shared bi-branch convolutional network. This represents the input hyperspectral image block; Indicates the common modal characteristics of synthetic aperture radar. This represents the synthetic aperture radar modal shared feature extraction function that uses a shared dual-branch convolutional network. This represents the input synthetic aperture radar image block.

[0072] In practical applications, the hyperspectral branch uses residual convolutional blocks to extract spatial-spectral features, while the synthetic aperture radar branch uses convolutional layers and cascaded convolutional structures to extract spatial-scattering features. In this embodiment, the number of convolutional reference channels is denoted as... Preferred .

[0073] Furthermore, to accommodate the distribution differences between labeled and unlabeled samples, a first private convolutional branch is set up for the labeled training sample set and a second private convolutional branch is set up for the unlabeled sample set. These branches extract the specific private features for the corresponding sample sets, allowing public and private features to be learned in parallel and avoiding information redundancy. The extraction processes for the first and second private features for the labeled training sample set and the unlabeled sample set are respectively represented as follows:

[0074] (2-3)

[0075] (2-4)

[0076] (2-5)

[0077] (2-6)

[0078] in, This represents the private feature of the first hyperspectral mode in the labeled training sample set. This represents the private features of the first synthetic aperture radar mode from the labeled training sample set. This represents the private feature of the second hyperspectral mode in an unlabeled sample set. This represents the private features of the second synthetic aperture radar mode, representing an unlabeled sample set. This represents the first hyperspectral private feature extraction function (private branch) for the labeled training sample set. Let represent the first synthetic aperture radar private feature extraction function for the labeled training sample set. This represents the second hyperspectral private feature extraction function for unlabeled sample sets. The second synthetic aperture radar private feature extraction function represents the unlabeled sample set. (Subscript) The subscript represents the variable associated with the labeled training sample set. This represents the variables associated with the unlabeled sample set.

[0079] By extracting public and private features in parallel, the network can simultaneously learn multimodal public discriminative information and sample set-specific supplementary information, significantly improving feature representation ability and adaptability.

[0080] In step 103, in order to achieve cross-modal global information interaction and overcome the limitation of single-modal receptive field, the hyperspectral common features and synthetic aperture radar common features are mapped into sequences respectively, and modal classification labels and modal position codes are added to form the input of the dual-stream scale interactive encoder.

[0081] Specifically, the flattened and transposed sequence mappings of the common features of hyperspectral modes and the common features of synthetic aperture radar modes are expressed by the following formulas:

[0082] (3-1)

[0083] (3-2)

[0084] in, This represents the initial input sequence of the hyperspectral stream, input to a two-stream scale interactive encoder. Learnable classification tokens (CLS tokens) representing hyperspectral flow. Indicates common features of hyperspectral modes. This represents the mode-specific position encoding of the hyperspectral flow. This represents the flattening and transpose sequence mapping function, which maps a two-dimensional feature map to a one-dimensional sequence. This represents a vector concatenation operation, which concatenates the classification labels and feature sequences in a dimensional manner. This represents the initial input sequence of the synthetic aperture radar flow, input to a two-stream scale interactive encoder. Learnable classification tokens (CLS tokens) representing synthetic aperture radar flow. Indicates the common modal characteristics of synthetic aperture radar. This represents the mode-specific position code of the synthetic aperture radar flow.

[0085] Furthermore, the aforementioned dual-stream sequence is fed into a dual-stream scale-interactive encoder for cross-modal interaction to achieve dual-modal information interaction between the hyperspectral stream and the synthetic aperture radar stream. This process can be represented as follows:

[0086] (3-3)

[0087] in, Indicates the process The hyperspectral stream output sequence after layer-inter-encoded layer, Indicates the process Synthetic Aperture Radar stream output sequence after layer-inter-coded layer, This represents the two-stream scale interactive encoder function (Transformer_inter), which enables cross-modal information exchange between two-modal sequences. This represents the total number of layers in a two-stream scale interactive encoder, with a preferred value of 3, abbreviated as... In this embodiment, the number of coding layers is preferably 3.

[0088] Furthermore, the first classification label of the encoded hyperspectral stream output sequence and the synthetic aperture radar stream output sequence are taken as global interactive features and concatenated to obtain global interactive fusion features, specifically:

[0089] (3-4)

[0090] (3-5)

[0091] (3-6)

[0092] in, The global interaction features representing hyperspectral modes are taken from the first classification label of the encoded sequence. The global interactive features representing the synthetic aperture radar modes are taken from the first classification label of the encoded sequence. This represents a tensor indexing operation, which retrieves the values ​​of all samples, the 0th position of the sequence (classification label), and all feature dimensions. This indicates the dual-modal global interactive fusion feature. The feature splicing function represents the splicing of the global interactive features of the hyperspectral mode and the synthetic aperture radar mode in the channel dimension;

[0093] Convolutional features excel at capturing local details, while global interactive features excel at capturing broad context. To preserve both types of information simultaneously, hyperspectral common features... Common characteristics of synthetic aperture radar By concatenating the features by channel, we obtain bimodal local convolutional features. Further, we use global average pooling to obtain local convolutional features, which can be determined by the following formula:

[0094] (3-7)

[0095] (3-8)

[0096] in, This represents the concatenated features of local convolutions in a dual-modal context. This represents the local convolutional features after global average pooling. This represents the global average pooling function, which performs average pooling on the spatial dimension of the two-dimensional feature map.

[0097] Furthermore, the bimodal global interactive fusion features and local convolutional features are added element-wise and fused to obtain the final bimodal fusion features as shown below:

[0098] (3-9)

[0099] in, This represents the final bimodal fusion feature, used for subsequent classification output. This indicates that the feature is added element by element, achieving the synergistic fusion of local convolutional features and global interactive features. This fusion method can simultaneously retain the local neighborhood information extracted by convolution and the overall correlation information after the two-stream interaction, thereby improving the feature discrimination power and robustness.

[0100] During the classification phase, based on the final bimodal fusion features Hyperspectral global interaction features Global Interaction Features with Synthetic Aperture Radar Three classification output heads are constructed to obtain the classification output results of the fusion branch, the hyperspectral single-mode branch, and the synthetic aperture radar single-mode branch, respectively, which are expressed by the following formulas.

[0101] (3-10)

[0102] (3-11)

[0103] (3-12)

[0104] in, This represents the classification output of the fusion branch. This represents the classification output of the hyperspectral single-modal branch. This represents the classification output of the single-mode branch of synthetic aperture radar. The classification header function representing the fusion branch, abbreviated as , The classification head function representing the hyperspectral branch, abbreviated as , The classification header function for the synthetic aperture radar branch, abbreviated as It should be noted that the trial phase outputs results based on the classification of the fusion branch. As the final classification result.

[0105] In step 104, in order to improve the network's discrimination ability, distribution consistency, feature decoupling, information integrity and classification stability under small sample conditions, this embodiment of the invention jointly optimizes five types of losses to form a strongly constrained semi-supervised training mechanism.

[0106] 1) Classification Loss: Supervised training is performed on the labeled training sample set, and dynamic weighting using weighted cross-entropy is adopted to alleviate the imbalance problem of land cover categories. Specifically, for the labeled training sample set, the classification loss is determined based on the classification output of the fusion branch using the following formula:

[0107] (4-1)

[0108] in, The classification loss is calculated based on the output of the fusion branch. Indicates the first Class weights for land features are used to mitigate class imbalance. This indicates an indicator function that takes the value 1 when the condition within the parentheses is true, and 0 otherwise. Indicates the first The labeled training sample set at the th ... The probability of fusion branch prediction on the class, This represents the natural logarithm function.

[0109] In this embodiment of the invention, the category weights are dynamically updated based on the number of samples of each category in the current training sample set, as shown below:

[0110] (4-2)

[0111] in, This represents a very small constant, used to avoid cases where the denominator is 0. It is typically set to a value of [value missing]. , Indicates the first The current number of training samples for land-like features. This represents the category index variable, used to iterate through all categories.

[0112] 2) Distribution Alignment Loss: To reduce the distribution difference between labeled training sample sets and unlabeled sample sets in different feature spaces, this embodiment calculates the maximum mean difference loss on hyperspectral modal common features, synthetic aperture radar modal common features, global interaction features, and final fused features. In practical applications, for any two sets of features... and Its maximum mean difference loss is determined by the following formula:

[0113] (4-3)

[0114] in, This represents the maximum mean difference loss function, used to measure the difference in distribution between two sets of features. This represents two sets of input features, corresponding to the features of the labeled training sample set and the unlabeled sample set, respectively. They represent feature sets respectively. and The number of samples, The kernel space mapping function represents the mapping from the original feature space to the reproducing kernel Hilbert space. Represents the regenerating kernel Hilbert space, the target space of the kernel mapping. denoted as the norm in the regenerated kernel Hilbert space.

[0115] Furthermore, the total distribution alignment loss is determined by the following formula:

[0116] (4-4)

[0117] (4-5)

[0118] (4-6)

[0119] (4-7)

[0120] (4-8)

[0121] in, This represents the total distribution alignment loss, used to reduce the difference in feature distributions between labeled and unlabeled sample sets. The MMD (Maximum Mean Discrepancy) loss represents the final fused features. MMD loss representing hyperspectral common features, The MMD loss represents the common characteristics of synthetic aperture radar. represents the MMD loss of the global interactive fusion features, and 0.1 represents the balancing weight, which is used to adjust the contribution of the MMD loss of different branches;

[0122] 3) Difference Constraint Loss: To enhance the difference between the bimodal common features and the corresponding private features of the sample set, a difference constraint is imposed between the bimodal common features and the corresponding private features of the sample set. Specifically, for any modality, if the flattened and normalized common features and private features are denoted as follows: and The difference constraint loss can be expressed by the following formula:

[0123] (4-9)

[0124] in, This represents the single-modal difference constraint loss, used to enhance the difference between public and private features. Represents the normalized bimodal common features. This represents the private features specific to the normalized sample set. Represents normalized bimodal common features transpose, This represents the Frobenius norm, used to measure the overall size of a matrix.

[0125] Furthermore, the total difference constraint loss is obtained by calculating and summing the results for the hyperspectral mode and the synthetic aperture radar mode separately:

[0126] (4-10)

[0127] in, The total difference constraint loss is the sum of the difference constraint losses for the hyperspectral and synthetic aperture radar modes. This represents the difference constraint loss of hyperspectral modes. This represents the difference constraint loss of the synthetic aperture radar modes.

[0128] 4) Reconstruction Loss: To maintain the reconstruction capability of the original input hyperspectral image patch and synthetic aperture radar image patch using the bimodal common features and the corresponding private features of the sample set, reconstruction is performed on the bimodal common features and the corresponding private features of the sample set, respectively. Let the reconstructors be... and ,but:

[0129] (4-11)

[0130] (4-12)

[0131] in, This represents the reconstruction result of the input hyperspectral image patch. This represents the reconstruction result of the synthetic aperture radar input data. Represents a reconstructor function for hyperspectral modes, used to reconstruct the original input from features. The reconstructor function represents the synthetic aperture radar modes, used to reconstruct the original input from the features. Indicates common features of hyperspectral modes. Indicates the common modal characteristics of synthetic aperture radar. Indicates the private features of hyperspectral modes. This represents the proprietary features of synthetic aperture radar modes.

[0132] It should be noted that the above-mentioned hyperspectral modal private features The supplementary discriminative information unique to hyperspectral modes is used to characterize the first hyperspectral mode private features corresponding to the labeled training sample set. And the private features of the second hyperspectral modality corresponding to the unlabeled sample set. Synthetic Aperture Radar Mode Private Characteristics The supplementary discriminative information unique to synthetic aperture radar modes is used to characterize the first synthetic aperture radar mode private features corresponding to the labeled training sample set. Private features of the second synthetic aperture radar mode corresponding to the unlabeled sample set. .

[0133] The reconstruction loss is the difference between the mean squared error and the scale-invariant mean squared error, expressed by the following formula:

[0134] (4-13)

[0135] (4-14)

[0136] in, This represents the single-modal reconstruction loss function. This represents the original input data. This indicates the reconstructed output data. This represents the mean squared error function, which measures the pixel-level difference between the original data and the reconstructed data. This represents a scale-invariant mean square error function that eliminates the influence of overall data scale differences. This indicates a summation operation, summing the values ​​over all pixel / feature dimensions. Representing input features The total number of pixels / feature dimension is consistent with the calculation dimension of MSE loss.

[0137] Furthermore, the total reconstruction loss can be obtained as follows:

[0138] (4-15)

[0139] in, The total reconstruction loss is the sum of the reconstruction losses of the hyperspectral mode and the synthetic aperture radar mode.

[0140] 5) Multi-output Coordination Consistency Loss: To ensure consistency among the classification outputs of the fusion branch, the hyperspectral single-mode branch, and the synthetic aperture radar single-mode branch, a softmax operation is performed on each of the three outputs:

[0141] (4-16)

[0142] (4-17)

[0143] (4-18)

[0144] in, This represents the output category probability distribution of the hyperspectral branch. This represents the output class probability distribution of the synthetic aperture radar branch. This represents the output class probability distribution of the fusion branch. This represents the softmax function, which maps the logits of the classification output to a probability distribution between 0 and 1, with the sum of the probabilities of all classes being 1.

[0145] Furthermore, the multi-output collaborative consistency loss is defined by the following formula:

[0146] (4-19)

[0147] in, This represents the multi-output collaborative consistency loss, used to constrain the output probability distributions of the three branches to remain consistent. denoted by L2 norm, it measures the difference between two probability distributions. In this embodiment, the consistency loss is calculated simultaneously on both the labeled training set and the unlabeled training set.

[0148] In summary, based on the multi-output collaborative consistency loss determined above... Total reconstruction loss Total difference constraint loss Total distribution alignment loss and classification loss The total loss of the main classification network can be obtained using the following formula:

[0149] (4-20)

[0150] in, This represents the total loss function of the main classification network, used for end-to-end training of the network. This represents the weighting coefficient of each loss term, with an optimal value of 0.01, used to balance the contribution of different loss terms;

[0151] In a preferred embodiment of the present invention, In practical applications, the main classification network is trained using the Adam optimizer, with the optimal learning rate being [missing information]. The batch size is preferably 32, and the maximum number of training rounds is preferably 300.

[0152] In step 105, in order to improve the utilization efficiency of unlabeled samples and avoid interference from erroneous pseudo-labels during training, this embodiment of the invention proposes a high-quality pseudo-label iterative update strategy.

[0153] Specifically, assuming the first The final fusion features of each sample are , No. The final fusion features of each sample are For any two samples and The number can be determined using the following formula. The sample and the first The spectral angular distance between samples is used to measure the similarity between sample features:

[0154] (5-1)

[0155] in, Indicates the first The sample and the first The spectral angular distance between samples is used to measure the similarity of sample features. Indicates the first The final fusion features of each sample Indicates the first The final fusion features of each sample This represents the dot product operation of two vectors. The L2 norm of a vector is used to measure the magnitude of the vector.

[0156] Furthermore, based on the aforementioned determined number The sample and the first The spectral angular distance between samples is used to select the nearest neighbor of each node. In this embodiment of the invention, a sample nearest neighbor graph is constructed using nodes. .

[0157] Furthermore, a graph discriminator is constructed based on the sample nearest neighbor graph. This graph discriminator consists of a graph Transformer network and a Sigmoid output layer, and is used to identify the distribution reliability of the unlabeled sample set, i.e., whether it is close to the distribution of the labeled training sample set. For the th For each sample, its output discriminant value can be expressed by the following formula:

[0158] (5-2)

[0159] in, Indicates the first The discriminator outputs a discriminant value for each sample, ranging from (0,1). A larger value indicates a higher discriminant value. The closer a sample is to the distribution of the labeled training sample set; The graph discriminator function consists of a graph Transformer network and a Sigmoid output layer. This represents a sample nearest neighbor graph constructed based on sample fusion features, consisting of sample nodes and nearest neighbor edges; The discriminant value represents the graph discriminator; the larger the discriminant value, the closer the sample is to the distribution of the labeled training sample set. .

[0160] In this embodiment of the invention, the optimization objective of the graph discriminator is a binary cross-entropy loss, which is expressed by the following formula:

[0161] (5-3)

[0162] in, This represents the binary cross-entropy loss, used in graph discriminators to distinguish between labeled training samples and unlabeled samples. This represents the set of indices for the labeled training sample set. This represents the set of indices for the unlabeled sample set. This indicates the number of labeled training samples. This represents the number of unlabeled samples in the unlabeled sample set; in the embodiment, the preferred number of training epochs for the graph discriminator is 200, and the preferred learning rate is... .

[0163] After the graph discriminator is trained, a boundary-aware strategy is used to select high-value candidate samples from the unlabeled sample set. The distance from the discriminant value to the decision boundary is determined by the following formula:

[0164] (5-4)

[0165] in, Indicates the first The distance from the discriminant value to the decision boundary of an unlabeled sample set. This represents the decision boundary parameter, with an optimal value of 0.3, used to distinguish between high-confidence and low-confidence samples. Represents the absolute value function;

[0166] In this embodiment of the invention, the selection is... The smallest front Each sample is used as a high-value candidate sample set. ,in This represents the number of candidate query samples in each round, which is preferred in this embodiment. .

[0167] Furthermore, for high-value candidate samples, the class probability distribution is calculated using the classification output of the fusion branch of the main classification network. The confidence level and predicted class are expressed as follows:

[0168] (5-5)

[0169] (5-6)

[0170] in, Indicates the first The classification confidence of a high-value candidate sample is the maximum value among the predicted probabilities of all classes. Indicates the first The high-value candidate samples in the first Predicted probabilities on the class Indicates all categories The operation of finding the maximum value. Indicates the first The pseudo-label prediction category of a high-value candidate sample. Indicates all categories Operation to retrieve the category index corresponding to the maximum value

[0171] Further calculate the global confidence threshold:

[0172] (5-7)

[0173] in, This represents the global confidence threshold, calculated based on the confidence scores of all high-value candidate samples. This represents the set of high-value candidate samples after boundary-aware screening. This represents the total number of high-value candidate samples, preferably 300.

[0174] Furthermore, for each category The number of samples whose statistical confidence exceeds the global threshold and are predicted as belonging to this category:

[0175] (5-8)

[0176] in, Indicates the first In the class, the confidence level exceeds the global threshold and the predicted class is The number of samples.

[0177] Let the mean of the number of samples exceeding the threshold for all categories be . Then the category adaptive coefficient is:

[0178] (5-9)

[0179] in, Indicates the first The class-adaptive coefficient is used to adjust the confidence threshold for that class. This represents the number of samples exceeding the threshold across all categories. The arithmetic mean, i.e. , This represents a very small constant, used to avoid cases where the denominator is 0;

[0180] Furthermore, the category-adaptive threshold is determined by the following formula:

[0181] (5-10)

[0182] in, Indicates the first An adaptive confidence threshold for a class is used to filter high-quality pseudo-labeled samples for that class. This represents a minimum value function that ensures the category threshold does not exceed 0.95, avoiding excessively high confidence requirements.

[0183] Furthermore, pseudo-labels are assigned to the high-value candidate samples that reach the category adaptive threshold and meet the confidence condition. The pseudo-label determination rule is as follows:

[0184] (5-11)

[0185] High-value candidate samples that meet this condition are assigned pseudo-labels. The high-value candidate samples with pseudo-labels are iteratively added to the labeled training sample set. After each round of pseudo-label update, the sample index is recalculated and the class weights are updated to continue training the main classification network.

[0186] In this embodiment of the invention, pseudo-label updates begin from the 100th round and are performed every 50 rounds thereafter. After each round of pseudo-label updates, the indices of the labeled training sample set and the unlabeled sample set are recalculated, and the class weights are recalculated. The main classification network continues to be trained until all training rounds are completed.

[0187] After training, the optimal main classification network parameters are loaded to perform inference on the labeled test sample set, and the final predicted category is:

[0188] (5-12)

[0189] in, This represents the final predicted category of the test sample, calculated based on the output of the fusion branch. Indicates the fusion branch at the 1st Output logits values ​​on the class; backfill the prediction results to the original image space location to generate the entire classification result map and its visualization image.

[0190] To more clearly illustrate the interactive multimodal classification method provided by the embodiments of the present invention, the following detailed description of the method is based on specific embodiments.

[0191] The method provided in this invention is implemented using PyTorch and DGL, with an NVIDIA GeForce RTX 5090Dv2 hardware platform. The current implementation employs a dual-stream scale-interactive master classification network and a graph-discriminatory pseudo-label update strategy. The master classification network has a default training epoch of 300, a training batch size of 32, a sample block size of 13×13, and an initial learning rate of 0.0001. Pseudo-label updates are introduced starting from the 100th epoch and are iteratively executed at epoch intervals of 50 epochs.

[0192] The multimodal remote sensing data used in this embodiment of the invention is the Augsburg dataset. Collected in the Augsburg region of Germany, the Augsburg dataset consists of spaceborne hyperspectral images and dual-polarization synthetic aperture radar (SAR) images, acquired by the HySpex sensor and the Sentinel-1 sensor, respectively. Both types of data have a spatial resolution of 30m and an image size of 332×485 pixels. The hyperspectral images cover a wavelength range of 0.4–2.5μm, containing 180 spectral bands. The SAR data includes four features: VV polarization intensity, VH polarization intensity, and the real and imaginary parts of the off-diagonal elements of the PolSAR covariance matrix, thus providing supplementary structural and scattering feature information for ground cover identification. In the current code, the Augsburg dataset corresponds to seven ground cover categories.

[0193] In the data preprocessing stage, principal component analysis is first performed on the registered initial hyperspectral image and the initial synthetic aperture radar image to reduce dimensionality, and then both images are standardized. Next, sample locations are divided based on a shared label map, and hyperspectral image patches and synthetic aperture radar image patches are extracted centered on corresponding locations to construct labeled training sample sets, unlabeled sample sets, and labeled test sample sets. In the current implementation, the standardized hyperspectral image and the synthetic aperture radar image are input together into the sample generation module. Training samples are extracted proportionally from all labeled pixels, with the training ratio set to 0.003.

[0194] During the network training phase, the main classification network consists of a shared dual-branch convolutional feature extraction unit, a sample set-specific private feature extraction unit, a two-stream scale interactive encoder unit, a local-global collaborative fusion unit, and a multi-output classification unit. Hyperspectral image patches and synthetic aperture radar (SAR) image patches are input into the shared dual-branch convolutional network to obtain bimodal common features. Labeled training sample sets are input into the first private convolutional branch, and unlabeled sample sets are input into the second private convolutional branch to obtain corresponding sample set-specific private features. Subsequently, the two modal common features are mapped to token sequences, and after adding modality classification labels and positional encoding, they are input into the two-stream scale interactive encoder. The hyperspectral global interactive features output by the encoder are concatenated with the SAR global interactive features to obtain global interactive fusion features, which are then combined with local convolutional fusion features to form the final bimodal fusion features. Simultaneously, the network outputs the classification results of the hyperspectral branch, the SAR branch, and the fusion branch, respectively, obtaining the classification results of the three branches for subsequent joint optimization.

[0195] In the current implementation, the main classification network is trained using the Adam optimizer with a learning rate of 0.0001, a base number of 32 channels, and 300 training epochs. The graph discriminator is also trained using the Adam optimizer with a learning rate of 0.0005 and 200 training epochs. During graph discriminator training, sample graphs are constructed based on the final bimodal fusion features output by the main classification network, and the graph discriminator network is optimized using a binary discriminant loss between labeled and unlabeled samples.

[0196] In the loss function design of the main classification network, the labeled training sample set and the unlabeled sample set are jointly optimized based on the bimodal common features, the corresponding sample set's proprietary private features, the global interactive fusion features, the final bimodal fusion features, and the classification results of the three branches. Five types of loss functions—classification loss, distribution alignment loss, difference constraint loss, reconstruction loss, and multi-output collaborative consistency loss—are used to construct a strongly constrained semi-supervised training mechanism. Specifically, the classification loss applies to the output of the fusion branch; the distribution alignment loss applies simultaneously to the hyperspectral common features, the synthetic aperture radar common features, the global interactive fusion features, and the final fusion features; the difference constraint loss enhances the difference between the common features and the proprietary features; the reconstruction loss preserves the input information; and the multi-output collaborative consistency loss constrains the prediction consistency between the hyperspectral output, the synthetic aperture radar output, and the fusion output. In the current implementation, the weights of MMD, difference constraint, and reconstruction loss are all 0.01, and the weight λ of the consistency term is set to 0.01.

[0197] In the pseudo-label update stage, a sample nearest neighbor graph is constructed based on the final bimodal fusion features. A graph discriminator identifies the distribution reliability of the unlabeled sample set. In the current implementation, when the sample size is small, a nearest neighbor graph construction method with k=20 is used. Subsequently, the graph discriminator outputs the discrimination score of the unlabeled sample and performs boundary-aware candidate sample screening based on its distance from the preset decision boundary d, where d is set to 0.3. A boundary-aware strategy is used to select high-value candidate samples from the unlabeled sample set. For the selected candidate samples, a global threshold and a class-adaptive threshold are further calculated based on the class probability output by the classifier. High-value candidate samples that reach the class-adaptive threshold and meet the confidence condition are assigned pseudo-labels. In the current implementation, the number of candidate samples per round is set to 300. The high-value candidate samples with obtained pseudo-labels are iteratively added to the labeled training sample set, forming a semi-supervised pseudo-label update mechanism.

[0198] In the experimental evaluation, three metrics—OA (Overall Accuracy), AA (Average Accuracy), and Kappa coefficient—were used to comprehensively evaluate the classification performance. In the current implementation, during the testing phase, the predicted category is calculated based on the output of the fusion branch, and the OA, AA, and Kappa values ​​are obtained by statistically analyzing the correspondence between the true and predicted labels, while simultaneously generating the complete classification result image.

[0199] To verify the effectiveness of this invention, several representative hyperspectral-SAR (Synthetic Aperture Radar) joint classification methods and semi-supervised classification methods were selected as comparison methods, and experiments were conducted on the Augsburg dataset for comparison. The experimental results are summarized in Table 1.

[0200] Table 1. Comparison of classification performance between the present invention and other methods

[0201]

[0202] Table 1 shows that the embodiments of the present invention achieve superior overall classification performance on the Augsburg dataset. Specifically, the embodiments of the present invention achieve 93.60%, 75.00%, and 90.84% ​​in OA, AA, and Kappa metrics, respectively, representing improvements of 1.29, 18.07, and 2.06 percentage points compared to the suboptimal method. Meanwhile, the embodiments of the present invention maintain high consistency in recognition results across categories, indicating that the proposed dual-stream scale interaction structure, local-global collaborative fusion method, and graph-discriminatory pseudo-label update mechanism effectively improve the stability and generalization ability of the joint classification of hyperspectral and synthetic aperture radar. The classification results are shown in the figure below. Figure 2As shown, from Figure 2 Similarly, it can be observed that the embodiments of the present invention have a clearer classification effect in the complex boundary area in the middle and the scattered small patch distribution area in the lower right. Especially at the boundary of categories and in the area of ​​fragmented land features, the classification map generated by the embodiments of the present invention is closer to the actual annotation, the boundary is more complete, and the phenomenon of local misclassification is reduced.

[0203] To verify the effectiveness of the graph discrimination pseudo-label update strategy, further ablation experiments were conducted on the Augsburg dataset. Specifically, the entire "graph discriminator + boundary-aware sample selection + category-adaptive threshold pseudo-label allocation" in the embodiments of this invention was replaced with "randomly selecting the same number of unlabeled samples and directly adding them to the training set," while keeping all other conditions unchanged. The ablation experiment results are summarized in Table 2.

[0204] Table 2. Ablation experimental results of multimodal tag assignment strategy

[0205]

[0206] As shown in Table 2, compared with randomly selected pseudo-labels, the embodiments of the present invention improved the classification performance by 0.32, 1.52, and 0.54 percentage points in OA, AA, and Kappa, respectively; compared with pseudo-labels with a fixed threshold, the present invention improved the performance by 1.22, 2.27, and 1.36 percentage points, respectively. These results indicate that the embodiments of the present invention do not simply improve classification performance by increasing the number of unlabeled samples, nor do they rely solely on a uniform threshold for pseudo-label selection. Instead, they identify sample distribution relationships through a graph discriminator and combine boundary-aware screening with class-adaptive threshold allocation to more effectively introduce pseudo-label samples with stronger representativeness and higher quality, thereby improving sample quality and class balance in subsequent training processes.

[0207] Furthermore, while random selection of pseudo-labels can expand the training sample to some extent, it is prone to introducing low-quality pseudo-labels due to the lack of control over sample validity and class distribution. Although the fixed-threshold pseudo-label method is superior to random selection in terms of sample quality control, it struggles to account for the differences between easily and difficult-to-classify categories by applying the same confidence threshold to different categories. In contrast, the embodiments of this invention employ three consecutive steps—graph discriminator, boundary-aware sample screening, and class-adaptive threshold allocation—to perform more refined screening and allocation of unlabeled samples, thus more effectively improving the utilization efficiency of unlabeled samples and overall classification performance.

[0208] To further verify the impact of different loss terms on the network classification performance provided in this embodiment of the invention, a stepwise superposition ablation analysis is performed on the loss function while keeping the main network structure and pseudo-label update strategy unchanged. Specifically, the settings are as follows:

[0209] (1) Only retain classification loss ;

[0210] (2) In Add distribution alignment loss to the basis ;

[0211] (3) Based on the above, further add difference constraint loss. ;

[0212] (4) In addition to the above, reconstruction loss is added. ;

[0213] (5) Based on the above, add the multi-output cooperative consistency loss between hyperspectral output, synthetic aperture radar output and fused output. .

[0214] The experimental results are shown in Table 3.

[0215] Table 3. Ablation Experiment Results of Loss Function

[0216]

[0217] As shown in Table 3, the network provided in this embodiment of the invention can achieve high classification accuracy when using only classification loss. This indicates that the parallel extraction of bimodal common features and corresponding private features of the sample set, along with the construction of the final bimodal fusion features, gives the main network structure strong feature extraction and fusion capabilities. Adding distribution alignment loss results in minimal performance change. Further adding difference constraint loss significantly improves OA, AA, and Kappa, indicating that applying decoupling constraints to the bimodal common features and corresponding private features of the sample set helps enhance feature discriminative power and reduce redundant information interference. Adding reconstruction loss further stabilizes the model performance. Adding multi-output collaborative consistency loss further maximizes AA and Kappa, while maintaining a high OA level. This demonstrates that applying collaborative constraints to the classification results of the three branches helps improve the overall stability of the model and the balanced recognition capability of each category. Therefore, it is evident that the loss terms in this embodiment of the invention have complementary roles in network training. The strongly constrained semi-supervised training mechanism formed by the joint optimization of five loss functions can achieve better comprehensive classification results.

[0218] Based on the same inventive concept, this invention provides an interactive multimodal classification device. Since the principle by which this device solves the technical problem is similar to that of an interactive multimodal classification method, the implementation of this device can refer to the implementation of the method, and the repeated parts will not be described again.

[0219] like Figure 3As shown, the device includes a first construction unit 301, a first obtaining unit 302, a second obtaining unit 303, a second construction unit 304, and an updating unit 305.

[0220] The first construction unit is used to perform dimensionality reduction and standardization processing on the registered initial spectral image and the initial synthetic aperture radar image, divide the sample positions according to the shared label map, and extract hyperspectral image blocks and synthetic aperture radar image blocks with the corresponding positions as the center to construct a labeled training sample set, an unlabeled sample set, and a labeled test sample set.

[0221] The first obtaining unit is used to obtain dual-modal common features from the hyperspectral image patch and the synthetic aperture radar image patch through a shared dual-branch convolutional network; the labeled training sample set is obtained through a first private convolutional branch and the unlabeled sample set is obtained through a second private convolutional branch to obtain the exclusive private features of the corresponding sample set.

[0222] The second obtaining unit is used to obtain the final bimodal fusion feature and the classification results of the three branches in sequence based on the bimodal common features and global interactive fusion features;

[0223] The second construction unit is used to jointly optimize five types of loss functions based on the labeled training sample set and the unlabeled unlabeled sample set, and construct a strongly constrained semi-supervised training mechanism.

[0224] The update unit is used to construct a sample nearest neighbor graph based on the final dual-modal fusion features, identify the distribution reliability of the unlabeled sample set through a graph discriminator, select high-value candidate samples in the unlabeled sample set using a boundary-aware strategy, assign pseudo-labels to the high-value candidate samples that have reached the class adaptation threshold and meet the confidence conditions, and iteratively add the high-value candidate samples with pseudo-labels to the labeled training sample set to form a semi-supervised pseudo-label update mechanism.

[0225] It should be understood that the units included in the above-described interactive multimodal classification device are merely a logical division based on the functions implemented by the device. In practical applications, the units can be superimposed or split. Furthermore, the functions implemented by the interactive multimodal classification device provided in this embodiment correspond one-to-one with the interactive multimodal classification method provided in the above-described embodiment. The more detailed processing flow implemented by this device has been described in detail in the first embodiment of the method described above, and will not be described in detail here.

[0226] Another embodiment of the present invention provides a computer device, the computer device including: a processor and a scene database; the scene database is used to store computer program code, the computer program code including computer instructions; when the processor executes the computer instructions, the electronic device executes the various steps of the interactive multimodal classification method shown in the above method embodiment.

[0227] Another embodiment of the present invention provides a computer-readable storage medium storing computer instructions that, when executed on a computer device, cause the computer device to perform the various steps of the interactive multimodal classification method shown in the above method embodiment.

[0228] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An interactive multimodal classification method, characterized in that, include: The registered initial spectral image and the initial synthetic aperture radar image are subjected to dimensionality reduction and standardization. The sample positions are divided according to the shared label map, and hyperspectral image patches and synthetic aperture radar image patches are extracted with the corresponding positions as the center to construct a labeled training sample set, an unlabeled sample set, and a labeled test sample set. The hyperspectral image patch and the synthetic aperture radar image patch are shared by a dual-branch convolutional network to obtain dual-modal common features respectively; the labeled training sample set is processed by a first private convolutional branch and the unlabeled sample set is processed by a second private convolutional branch to obtain the exclusive private features of the corresponding sample set respectively. Based on the bimodal common features and global interactive fusion features, the final bimodal fusion features and the classification results of the three branches are obtained sequentially; The labeled training sample set and the unlabeled sample set are jointly optimized based on bimodal common features, corresponding sample set exclusive private features, global interactive fusion features, final bimodal fusion features and classification results of three branches, and five types of loss functions are jointly optimized to construct a strongly constrained semi-supervised training mechanism. Based on the final dual-modal fusion features, a sample nearest neighbor graph is constructed. The distribution reliability of the unlabeled sample set is identified by a graph discriminator. A boundary-aware strategy is used to select high-value candidate samples from the unlabeled sample set. The high-value candidate samples that reach the class adaptation threshold and meet the confidence condition are assigned pseudo-labels. The high-value candidate samples with pseudo-labels are iteratively added to the labeled training sample set to form a semi-supervised pseudo-label update mechanism.

2. The method as described in claim 1, characterized in that, The dual-modal common features include hyperspectral modal common features and synthetic aperture radar modal common features, which are determined by the following formulas: in, Indicates common features of hyperspectral modes. This represents the hyperspectral modality shared feature extraction function of a shared bi-branch convolutional network. This represents the input hyperspectral image block; Indicates the common modal characteristics of synthetic aperture radar. This represents the synthetic aperture radar modal shared feature extraction function that uses a shared dual-branch convolutional network. This represents the input synthetic aperture radar image block.

3. The method as described in claim 1, characterized in that, The private features specific to the corresponding sample set include the first modality private features of the labeled training sample set and the second modality private features of the unlabeled sample set; The first modal private feature of the labeled training sample set includes the first hyperspectral modal private feature of the labeled training sample set and the first synthetic aperture radar modal private feature of the labeled training sample set; The second modal private features of the unlabeled sample set include the second hyperspectral modal private features of the unlabeled sample set and the second synthetic aperture radar modal private features of the unlabeled sample set. They are determined by the following formulas: in, This represents the private feature of the first hyperspectral mode in the labeled training sample set. This represents the private features of the first synthetic aperture radar mode from the labeled training sample set. This represents the private feature of the second hyperspectral mode in an unlabeled sample set. This represents the private features of the second synthetic aperture radar mode, representing an unlabeled sample set. Let represent the first hyperspectral private feature extraction function for the labeled training sample set. Let represent the first synthetic aperture radar private feature extraction function for the labeled training sample set. This represents the second hyperspectral private feature extraction function for unlabeled sample sets. The second synthetic aperture radar private feature extraction function represents the unlabeled sample set of unlabeled samples.

4. The method as described in claim 1, characterized in that, The five loss functions include multi-output collaborative consistency loss, total reconstruction loss, total difference constraint loss, total distribution alignment loss, and classification loss. The total loss of the main classification network is: in, This represents the total loss function of the main classification network. This represents the weighting coefficient of each loss term. This represents the loss of multi-output collaborative consistency. Indicates the total reconstruction loss. This represents the total variance constraint loss. Represents the total distribution alignment loss and This represents classification loss.

5. The method as described in claim 1, characterized in that, in, Indicates the first The sample and the first The spectral angular distance between samples is used to measure the similarity of sample features. Indicates the first The final fusion features of each sample Indicates the first The final fusion features of each sample This represents the dot product operation of two vectors. The L2 norm of a vector is used to measure the magnitude of the vector.

6. The method as described in claim 1, characterized in that, The boundary-aware strategy calculates the distance from the sample to the decision boundary using the following formula. in, Indicates the first The distance from the discriminant value to the decision boundary of an unlabeled sample set. This represents the decision boundary parameters.

7. The method as described in claim 1, characterized in that, The category adaptive threshold is determined by the following formula: in, This represents the global confidence threshold. This represents the set of high-value candidate samples after boundary-aware filtering. This represents the total number of high-value candidate samples. Indicates the first The class-adaptive confidence threshold This represents the function that takes the minimum value.

8. An interactive multimodal classification device, characterized in that, include: The first construction unit is used to perform dimensionality reduction and standardization processing on the registered initial spectral image and the initial synthetic aperture radar image, divide the sample positions according to the shared label map, and extract hyperspectral image blocks and synthetic aperture radar image blocks with the corresponding positions as the center to construct a labeled training sample set, an unlabeled sample set, and a labeled test sample set. The first obtaining unit is used to obtain dual-modal common features from the hyperspectral image patch and the synthetic aperture radar image patch through a shared dual-branch convolutional network; the labeled training sample set is obtained through a first private convolutional branch and the unlabeled sample set is obtained through a second private convolutional branch to obtain the exclusive private features of the corresponding sample set. The second obtaining unit is used to obtain the final bimodal fusion feature and the classification results of the three branches in sequence based on the bimodal common features and global interactive fusion features; The second construction unit is used to jointly optimize five types of loss functions based on the labeled training sample set and the unlabeled unlabeled sample set, and construct a strongly constrained semi-supervised training mechanism. The update unit is used to construct a sample nearest neighbor graph based on the final dual-modal fusion features, identify the distribution reliability of the unlabeled sample set through a graph discriminator, select high-value candidate samples in the unlabeled sample set using a boundary-aware strategy, assign pseudo-labels to the high-value candidate samples that have reached the class adaptation threshold and meet the confidence conditions, and iteratively add the high-value candidate samples with pseudo-labels to the labeled training sample set to form a semi-supervised pseudo-label update mechanism.

9. A computer-readable storage medium, characterized in that, The system contains a computer program that, when executed by a processor, causes the processor to perform an interactive multimodal classification method as described in any one of claims 1-7.

10. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, causes the processor to perform an interactive multimodal classification method as described in any one of claims 1-7.

Images