Multi-modal remote sensing image classification method and device, electronic equipment and medium

Abstract

The invention provides a multi-modal remote sensing image classification method and device, electronic equipment and a medium, and belongs to the field of remote sensing image classification, and the method comprises the steps: inputting obtained multi-modal remote sensing image data into a multi-modal remote sensing image classification network which is completely trained, performing spatial spectral feature enhancement fusion on the multi-modal remote sensing image data to obtain spatial spectral fusion features, and performing global feature enhancement fusion on the multi-modal remote sensing image data to obtain global fusion features; performing parallel Fourier transform frequency domain analysis and double-feature exchange fusion on the spatial spectrum fusion features and the global fusion features to obtain fusion output features; and carrying out classification prediction on the fusion output features to obtain a multi-modal remote sensing image classification result. According to the method, key information is extracted and effectively fused through spatial spectral feature enhanced fusion and global feature enhanced fusion, the quality of feature fusion is further improved through parallel Fourier pair and double-feature exchange fusion, and the classification performance of the network is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image classification, and in particular to a multimodal remote sensing image classification method, device, electronic equipment and medium. Background Art

[0002] In the remote sensing image classification task, single-modal data may not fully reflect the characteristics of the ground objects, while multimodal data can provide more details and supplementary information. Combining images of different modalities for analysis can overcome the limitations of single-modal remote sensing image classification. In addition, single-modal data sometimes leads to missing data or poor quality due to sensor failure, weather conditions, etc. Through the fusion of multimodal data, these defects can be supplemented and corrected to provide more complete and consistent ground object information.

[0003] However, since multimodal data comes from different sensors, they have different physical properties and data formats. In the multimodal image classification task, it is necessary to effectively fuse heterogeneous data from different sources. However, in the existing multimodal remote sensing image classification methods, usually only the data of different modalities are roughly fused, and the feature information of multimodal data cannot be fully extracted and fused, resulting in poor classification performance.

[0004] Therefore, there are technical problems in the prior art that data from different modalities are only roughly integrated, feature information of multimodal data cannot be fully extracted and integrated, and classification performance is poor, which needs to be improved. Summary of the invention

[0005] In view of this, it is necessary to provide a multimodal remote sensing image classification method, device, electronic device and medium to solve the technical problems in the prior art that data of different modalities are only roughly fused, the feature information of multimodal data cannot be fully extracted and fused, and the classification performance is poor.

[0006] In order to solve the above technical problems, on the one hand, the present invention provides a multimodal remote sensing image classification method, comprising: The acquired multimodal remote sensing image data is input into a well-trained multimodal remote sensing image classification network, and the spatial spectral feature enhancement and fusion of the multimodal remote sensing image data is performed to obtain the spatial spectral fusion feature, and the global feature enhancement and fusion of the multimodal remote sensing image data is performed to obtain the global fusion feature; Perform parallel Fourier transform frequency domain analysis and dual feature exchange fusion on the spatial spectrum fusion feature and the global fusion feature to obtain the fusion output feature; The fusion output features are classified and predicted to obtain the multimodal remote sensing image classification results.

[0007] In a possible implementation, the multimodal remote sensing image data includes spectral image data and radar image data.

[0008] In a possible implementation, spatial spectral feature enhancement fusion is performed on multimodal remote sensing image data to obtain spatial spectral fusion features, including: Perform multi-layer residual convolution feature alignment on multi-modal remote sensing image data to obtain aligned modal features, wherein the aligned modal features include spectral modal features and radar modal features; The modal features are sequentially subjected to feature dimension reduction and spatial direction flattening operations to obtain the modal feature vector; Perform matrix multiplication on the modal eigenvectors along the channel direction to obtain the channel occlusion matrix, and perform matrix multiplication on the modal eigenvectors along the spatial direction to obtain the spatial occlusion matrix; The channel close matrix is ​​dimensional compressed and reshaped, and then multiplied with the spectral modal feature to obtain the spectral enhancement feature. The spatial close matrix is ​​dimensional compressed and reshaped, and then multiplied with the spectral modal feature to obtain the spatial enhancement feature. The spectral enhancement features and spatial enhancement features are concatenated to obtain the spatial-spectral fusion features.

[0009] In a possible implementation, the global feature enhancement fusion is performed on the multimodal remote sensing image data to obtain the global fusion feature, including: Performing three-dimensional convolution feature alignment and two-dimensional convolution feature alignment on the spectral image data to obtain spectral alignment features; Perform two-dimensional convolution feature alignment on radar image data to obtain radar alignment features; The spectral alignment features are globally averaged pooled and convolution sampling feature enhanced to obtain the spectral global enhanced features, and the radar alignment features are globally averaged pooled and convolution sampling feature enhanced to obtain the radar global enhanced features; The spectral global enhancement feature and the spectral alignment feature are multiplied to obtain the spectral global feature, and the radar global enhancement feature and the radar alignment feature are multiplied to obtain the radar global feature; The global fusion features are obtained by adding and fusing the spectral global features and the radar global features.

[0010] In a possible implementation, the parallel Fourier transform frequency domain analysis includes: Perform Fourier transform on the spatial spectrum fusion features to obtain spatial spectrum frequency domain features, and perform Fourier transform on the global fusion features to obtain global frequency domain features; Performing linear convolution on the spatial spectrum frequency domain feature to obtain a first spatial spectrum complementary feature, and performing linear convolution on the global frequency domain feature to obtain a first global complementary feature; The spatial spectrum frequency domain feature is subjected to feature filtering and inverse Fourier transformation, and then added and fused with the first spatial spectrum complementary feature to obtain the spatial spectrum coding feature. The global frequency domain feature is subjected to feature filtering and inverse Fourier transformation, and then added and fused with the first global complementary feature to obtain the global coding feature.

[0011] In a possible implementation, the dual-feature exchange fusion includes: Performing feature learning on the spatial spectrum coding feature to obtain a spatial spectrum learning feature, performing feature learning on the global coding feature to obtain a global learning feature, performing linear convolution on the spatial spectrum coding feature to obtain a second spatial spectrum complementary feature, and performing linear convolution on the global coding feature to obtain a second global complementary feature; The spatial spectrum learning feature and the second spatial spectrum complementary feature are added and fused to obtain the spatial spectrum learning fusion feature, and the global learning feature and the second global complementary feature are added and fused to obtain the global learning fusion feature; The spatial spectrum learning fusion features and the global learning fusion features are added and fused to obtain the fusion output features.

[0012] In a possible implementation, the trained multimodal remote sensing image classification network is obtained by training an initial multimodal remote sensing image classification network, and the training of the initial multimodal remote sensing image classification network includes: Determine the total loss function based on exponential label smoothed cross entropy; The total loss function is decomposed into a spatial-spectral feature enhancement fusion branch loss function, a global feature enhancement fusion branch loss function, and a parallel Fourier transform feature fusion branch loss function; The initial multimodal remote sensing image classification network is iteratively trained according to the total loss function, the spatial-spectral feature enhancement fusion branch loss function, the global feature enhancement fusion branch loss function and the parallel Fourier transform feature fusion branch loss function until the loss no longer decreases, thereby obtaining a fully trained multimodal remote sensing image classification network.

[0013] On the other hand, the present invention also provides a multimodal remote sensing image classification device, comprising: The feature enhancement and fusion unit is used to input the acquired multimodal remote sensing image data into a well-trained multimodal remote sensing image classification network, perform spatial spectral feature enhancement and fusion on the multimodal remote sensing image data to obtain spatial spectral fusion features, and perform global feature enhancement and fusion on the multimodal remote sensing image data to obtain global fusion features; The frequency domain analysis fusion unit is used to perform parallel Fourier transform frequency domain analysis and dual feature exchange fusion on the spatial spectrum fusion feature and the global fusion feature to obtain the fusion output feature; The prediction output unit is used to perform classification prediction on the fused output features to obtain the multimodal remote sensing image classification results.

[0014] On the other hand, the present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the multimodal remote sensing image classification method described above is implemented.

[0015] On the other hand, the present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the above-mentioned multimodal remote sensing image classification method is implemented.

[0016] The beneficial effects of the present invention are as follows: in the multimodal remote sensing image classification method provided by the present invention, firstly, the acquired multimodal remote sensing image data is input into a well-trained multimodal remote sensing image classification network, the multimodal remote sensing image data is subjected to spatial spectral feature enhancement fusion to obtain spatial spectral fusion features, and the multimodal remote sensing image data is subjected to global feature enhancement fusion to obtain global fusion features; the spatial spectral fusion features and the global fusion features are subjected to parallel Fourier transform frequency domain analysis and dual feature exchange fusion to obtain fusion output features; the fusion output features are subjected to classification prediction to obtain the multimodal remote sensing image classification results. The present invention enhances and fuses the features of multimodal remote sensing images in different dimensions through spatial spectral feature enhancement fusion and global feature enhancement fusion, can extract more abundant key information and effectively fuse the features, and then the spatial spectral fusion features and the global fusion features obtained by fusion are subjected to frequency domain analysis by parallel Fourier and then dual feature exchange fusion is performed, which further improves the quality of feature fusion and effectively improves the classification performance of the multimodal remote sensing image classification network. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 A schematic diagram of a flow chart of an embodiment of a multimodal remote sensing image classification method provided by the present invention; Figure 2 A schematic diagram of a process for enhancing and fusing spatial and spectral features according to an embodiment of the present invention; Figure 3 A schematic diagram of a process of global feature enhancement fusion according to an embodiment of the present invention; Figure 4 A schematic diagram of a process of parallel Fourier transform frequency domain analysis according to an embodiment of the present invention; Figure 5 It is a schematic diagram of the process of dual-feature exchange and fusion according to an embodiment of the present invention; Figure 6 A schematic diagram of a process for training an initial multimodal remote sensing image classification network according to an embodiment of the present invention; Figure 7 A schematic diagram of the structure of an embodiment of a multimodal remote sensing image classification device provided by the present invention; Figure 8 A schematic structural diagram of an embodiment of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0020] In the description of the embodiments of the present invention, unless otherwise specified, "multiple" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone.

[0021] The descriptions of "first" and "second" in the embodiments of the present invention are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the technical features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0022] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0023] The present invention provides a multimodal remote sensing image classification method, device, electronic device and medium, which are respectively described below.

[0024] Figure 1 A schematic diagram of a flow chart of an embodiment of the multimodal remote sensing image classification method provided by the present invention is as follows: Figure 1 As shown, the multimodal remote sensing image classification method includes: S101, inputting the acquired multimodal remote sensing image data into a well-trained multimodal remote sensing image classification network, performing spatial spectral feature enhancement and fusion on the multimodal remote sensing image data to obtain spatial spectral fusion features, and performing global feature enhancement and fusion on the multimodal remote sensing image data to obtain global fusion features; S102, performing parallel Fourier transform frequency domain analysis and dual feature exchange fusion on the spatial spectrum fusion feature and the global fusion feature to obtain a fusion output feature; S103, classify and predict the fused output features to obtain a multimodal remote sensing image classification result.

[0025] Compared with the prior art, the multimodal remote sensing image classification method provided by the embodiment of the present invention first inputs the acquired multimodal remote sensing image data into a well-trained multimodal remote sensing image classification network, performs spatial spectral feature enhancement fusion on the multimodal remote sensing image data to obtain spatial spectral fusion features, performs global feature enhancement fusion on the multimodal remote sensing image data to obtain global fusion features; performs parallel Fourier transform frequency domain analysis and dual feature exchange fusion on the spatial spectral fusion features and the global fusion features to obtain fusion output features; and performs classification prediction on the fusion output features to obtain the multimodal remote sensing image classification results. The present invention enhances and fuses the features of multimodal remote sensing images in different dimensions through spatial spectral feature enhancement fusion and global feature enhancement fusion, can extract more abundant key information and effectively fuse the features, and then performs frequency domain analysis on the spatial spectral fusion features and the global fusion features obtained by the parallel Fourier transform and performs dual feature exchange fusion, further improving the quality of feature fusion and effectively improving the classification performance of the multimodal remote sensing image classification network.

[0026] In some embodiments of the present invention, the multimodal remote sensing image data includes spectral image data and radar image data.

[0027] Specifically, in order to make up for the problems of insufficient information and insufficient accuracy of single modality data, the embodiment uses data of two modalities, spectral image data and radar image data, to classify remote sensing images. Among them, optical images can provide rich spectral information, but they cannot work when covered by clouds. Radar images can penetrate clouds to obtain surface information, but they do not have the color information of optical images. By combining the two modalities of spectral image data and radar image data, the embodiment can overcome the limitations of a single modality and avoid data loss or poor quality of single modality data due to environmental reasons.

[0028] In some embodiments of the present invention, Figure 2 FIG. 1 is a flow chart of the spatial spectrum feature enhancement fusion process of an embodiment of the present invention. Figure 2 As shown in the figure, the spatial spectral feature enhancement fusion of multimodal remote sensing image data is performed to obtain the spatial spectral fusion features, including: S201, performing multi-layer residual convolution feature alignment on multi-modal remote sensing image data to obtain aligned modal features, wherein the aligned modal features include spectral modal features and radar modal features; S202, performing feature dimension reduction and spatial direction flattening operations on the modal features in sequence to obtain a modal feature vector; S203, performing matrix multiplication on the modal eigenvector along the channel direction to obtain a channel occlusion matrix, and performing matrix multiplication on the modal eigenvector along the spatial direction to obtain a spatial occlusion matrix; S204, compressing and reshaping the channel occludin matrix and then multiplying the matrix with the spectral modal feature to obtain a spectral enhancement feature, and compressing and reshaping the spatial occludin matrix and then multiplying the matrix with the spectral modal feature to obtain a spatial enhancement feature; S205, concatenating the spectral enhancement feature and the spatial enhancement feature to obtain the spatial-spectral fusion feature.

[0029] Specifically, in order to extract richer key information and effectively fuse features of different modalities, the embodiment constructs a multimodal remote sensing image dataset from optical images and radar images, and then fuses the multimodal remote sensing image data using two different methods: spatial spectral feature enhancement fusion and global feature enhancement fusion.

[0030] In the spatial spectral feature enhancement fusion, the embodiment first performs multi-layer feature alignment on the input multi-modal remote sensing image. In this process, both the spectral image data and the radar image data are feature aligned through three layers of residual convolution blocks to obtain spectral modal features with the same number of channels. and radar modal characteristics .

[0031] Then the embodiment respectively analyzes the modal features and After dimensionality reduction, batch normalization and Sigmoid activation function processing, the modal feature vector is obtained by flattening in the spatial direction. and .

[0032] Then the modal eigenvector and Perform matrix multiplication along the channel direction to obtain the channel occlusion matrix . The modal eigenvector and Perform matrix multiplication along the spatial direction to obtain the spatial occlusive matrix .

[0033] Next, the embodiment compresses and reshapes the channel close matrix to obtain the channel attention feature, and compresses and reshapes the spatial close matrix to obtain the spatial attention feature. Multiply to get the spectral enhancement feature , combining spatial attention features with spectral modality features Multiply to get spatial enhancement features .

[0034] The final embodiment will obtain the spectral enhancement characteristics and spatial enhancement features Use Concat to obtain spatial-spectral fusion features .

[0035] In some embodiments of the present invention, Figure 3 FIG. 1 is a flow chart of the global feature enhancement fusion process according to an embodiment of the present invention. Figure 3 As shown in the figure, the global feature enhancement fusion of multimodal remote sensing image data is performed to obtain the global fusion features, including: S301, performing three-dimensional convolution feature alignment and two-dimensional convolution feature alignment on the spectral image data to obtain spectral alignment features; S302, performing two-dimensional convolution feature alignment on radar image data to obtain radar alignment features; S303, performing global average pooling and convolution sampling feature enhancement on the spectral alignment feature to obtain a spectral global enhanced feature, and performing global average pooling and convolution sampling feature enhancement on the radar alignment feature to obtain a radar global enhanced feature; S304, multiplying the spectrum global enhancement feature and the spectrum alignment feature to obtain a spectrum global feature, and multiplying the radar global enhancement feature and the radar alignment feature to obtain a radar global feature; S305: Add and fuse the spectral global feature and the radar global feature to obtain a global fusion feature.

[0036] Specifically, in the global feature enhancement fusion, the embodiment first performs feature alignment on the multimodal remote sensing image data by combining three-dimensional convolution and two-dimensional convolution. For the spectral image data, the embodiment first performs three-dimensional convolution, batch normalization and Relu activation function to obtain the spatial extraction feature. , and then extract features from the space Perform grouped two-dimensional convolution and ordinary two-dimensional convolution respectively, and add the corresponding elements of the convolved features to obtain the spectral alignment features. For radar image data, the embodiment obtains radar alignment features through two-dimensional convolution, batch normalization and ReLU activation function. .

[0037] Then, the embodiment further enhances and fuses the key information inside the modality through an average pooling layer, and the specific steps are as follows: Spectral alignment features Aligning features with radar Perform global average pooling operation to obtain features and , then and Perform convolution downsampling and convolution upsampling operations to enhance features and obtain spectral global enhancement features and radar global enhancement features .

[0038] Then the spectrum is globally enhanced and spectral alignment features Perform the corresponding position multiplication operation to obtain the global characteristics of the spectrum , the radar global enhancement feature Aligning features with radar Perform corresponding position multiplication operation to obtain radar global features .

[0039] Finally, the obtained spectral global characteristics and radar global characteristics Perform the corresponding position addition operation to obtain the global fusion feature .

[0040] In some embodiments of the present invention, Figure 4 FIG. 4 is a flow chart of parallel Fourier transform frequency domain analysis in an embodiment of the present invention. Figure 4 As shown, parallel Fourier transform frequency domain analysis includes: S401, performing Fourier transform on the spatial spectrum fusion feature to obtain spatial spectrum frequency domain feature, and performing Fourier transform on the global fusion feature to obtain global frequency domain feature; S402, performing linear convolution on the spatial spectrum frequency domain feature to obtain a first spatial spectrum complementary feature, and performing linear convolution on the global frequency domain feature to obtain a first global complementary feature; S403, performing feature filtering and inverse Fourier transformation on the spatial spectrum frequency domain feature, and then adding and fusing it with the first spatial spectrum complementary feature to obtain the spatial spectrum coding feature, and performing feature filtering and inverse Fourier transformation on the global frequency domain feature, and then adding and fusing it with the first global complementary feature to obtain the global coding feature.

[0041] Specifically, after obtaining the spatial-spectral fusion features and the global fusion features, they need to be fused. To further improve the quality of feature fusion, the embodiment performs parallel Fourier transform frequency domain analysis and dual feature exchange fusion on the spatial-spectral fusion features and the global fusion features in this process.

[0042] In the parallel Fourier transform frequency domain analysis, the embodiment first performs spatial spectrum fusion feature And global fusion features Perform layer normalization and then perform a two-dimensional Fourier transform to obtain the corresponding spatial spectrum frequency domain features and global frequency domain characteristics The formula is as follows:

[0043]

[0044]

[0045] in, represents the two-dimensional Fourier transform, and represents the location of feature points, and Indicates the length and width of the feature map.

[0046] Then the spatial spectrum fusion feature And global fusion features The first spatial-spectral complementary feature is obtained by reducing the dimension through the fully connected layer and increasing the dimension through the linear layer and the fully connected layer. and the first global complementary feature .

[0047] Then the spatial spectrum frequency domain features and global frequency domain characteristics The features are filtered by multiplying them with the learnable filter K respectively, and the filtered spatial spectrum frequency domain features and global frequency domain features are subjected to two-dimensional inverse Fourier transform to obtain the spatial spectrum Fourier coded features. and global Fourier coded features The formula is as follows:

[0048]

[0049]

[0050]

[0051]

[0052] in, represents the two-dimensional inverse Fourier transform, and Indicates the position of the feature point in the frequency domain feature map, and Indicates the length and width of the feature map.

[0053] Then the spatial spectrum Fourier encoding feature Complementary characteristics of the first spatial spectrum Additive fusion to obtain spatial spectral coding features , the global Fourier coded features Complementary feature with the first global Add and fuse to get the global encoding features .

[0054] In some embodiments of the present invention, Figure 5 FIG. 1 is a flow chart of dual-feature exchange and fusion according to an embodiment of the present invention. Figure 5 As shown, dual feature exchange fusion includes: S501, performing feature learning on the spatial spectrum coding feature to obtain a spatial spectrum learning feature, performing feature learning on the global coding feature to obtain a global learning feature, performing linear convolution on the spatial spectrum coding feature to obtain a second spatial spectrum complementary feature, and performing linear convolution on the global coding feature to obtain a second global complementary feature; S502, adding and fusing the spatial spectrum learning feature and the second spatial spectrum complementary feature to obtain a spatial spectrum learning fusion feature, and adding and fusing the global learning feature and the second global complementary feature to obtain a global learning fusion feature; S503, adding and fusing the spatial spectrum learning fusion feature and the global learning fusion feature to obtain a fusion output feature.

[0055] Specifically, in the process of dual feature exchange and fusion, the embodiment first converts the spatial spectrum encoding feature and global encoding features Feature learning is performed in sequence through the fully connected layer, GELU activation function, Dropout layer, fully connected layer and Dropout layer to obtain the spatial spectral learning features after feature learning. and global learning features .

[0056] At the same time, the embodiment also encodes the spatial spectrum characteristics and global encoding features The second spatial-spectral complementary features are obtained by sequentially reducing the dimension through the fully connected layer, increasing the dimension through the linear layer and the fully connected layer. and the second global complementary feature .

[0057] Then the embodiment will obtain the spatial spectrum learning features and the second spatial spectrum complementary feature After adding, we get the spatial spectrum learning fusion feature of information exchange ; The global learning features obtained and the second global complementary feature After adding, we get the global learning fusion feature of information exchange .

[0058] The last embodiment then combines the spatial spectrum learning fusion feature And global learning fusion features After adding and fusion, the fusion output features are obtained , complete the feature learning and refined feature fusion of different modal data.

[0059] In some embodiments of the present invention, the fully trained multimodal remote sensing image classification network is obtained by training an initial multimodal remote sensing image classification network. Figure 6 FIG. 1 is a flow chart of training an initial multimodal remote sensing image classification network according to an embodiment of the present invention. Figure 6 As shown, the initial multimodal remote sensing image classification network is trained, including: S601, determining a total loss function based on exponential label smoothed cross entropy; S602, decomposing the total loss function into a spatial-spectral feature enhancement fusion branch loss function, a global feature enhancement fusion branch loss function, and a parallel Fourier transform feature fusion branch loss function; S603, iteratively training the initial multimodal remote sensing image classification network according to the total loss function, the spatial-spectral feature enhancement fusion branch loss function, the global feature enhancement fusion branch loss function and the parallel Fourier transform feature fusion branch loss function until the loss no longer decreases, thereby obtaining a fully trained multimodal remote sensing image classification network.

[0060] Specifically, in the multimodal remote sensing image classification network training process, the embodiment inputs sample data in the multimodal remote sensing image dataset into the model, calculates the loss function value, performs back propagation, optimizes the connection weights through the selected optimizer and corresponding parameters, and trains for multiple rounds until the loss no longer decreases or reaches a preset maximum number of iterations, thereby obtaining the final multimodal image classification model.

[0061] In this embodiment, the total loss function is determined by exponential label smoothed cross entropy, and the loss function calculation formula is:

[0062] in, is the number of categories, Represents the true label value of the corresponding category, Indicates the penalty intensity of the loss function on the model, Indicates the degree of loss function label smoothness.

[0063] In addition, in order to conduct more detailed training for each module, the embodiment also decomposes the total loss function into a spatial spectral feature enhancement fusion branch loss function: , global feature enhancement fusion branch loss function And parallel Fourier transform feature fusion branch loss function , in order to train the parameters of different modules respectively. The formula is expressed as:

[0064] in, , and Respectively represent characteristics , and The weight ratio is used to control the model's emphasis on learning features.

[0065] In summary, in order to fully extract and fuse the feature information of multimodal data and improve the model classification performance, the multimodal remote sensing image classification method provided by the present invention first inputs the acquired multimodal remote sensing image data into a well-trained multimodal remote sensing image classification network, performs spatial spectral feature enhancement fusion on the multimodal remote sensing image data to obtain spatial spectral fusion features, performs global feature enhancement fusion on the multimodal remote sensing image data to obtain global fusion features; performs parallel Fourier transform frequency domain analysis and dual feature exchange fusion on the spatial spectral fusion features and the global fusion features to obtain fusion output features; performs classification prediction on the fusion output features to obtain the multimodal remote sensing image classification results. The present invention enhances and fuses the features of multimodal remote sensing images in different dimensions through spatial spectral feature enhancement fusion and global feature enhancement fusion, can extract more abundant key information and effectively fuse the features, and then performs frequency domain analysis on the spatial spectral fusion features and the global fusion features obtained by the parallel Fourier transform and performs dual feature exchange fusion, further improving the quality of feature fusion and effectively improving the classification performance of the multimodal remote sensing image classification network.

[0066] In order to better implement the multimodal remote sensing image classification method in the embodiment of the present invention, based on the multimodal remote sensing image classification method, correspondingly, Figure 7 As shown, the present invention also provides a multimodal remote sensing image classification device, and the multimodal remote sensing image classification device 700 includes: The feature enhancement and fusion unit 701 is used to input the acquired multimodal remote sensing image data into a well-trained multimodal remote sensing image classification network, perform spatial spectral feature enhancement and fusion on the multimodal remote sensing image data to obtain spatial spectral fusion features, and perform global feature enhancement and fusion on the multimodal remote sensing image data to obtain global fusion features; The frequency domain analysis and fusion unit 702 is used to perform parallel Fourier transform frequency domain analysis and dual feature exchange fusion on the spatial spectrum fusion feature and the global fusion feature to obtain a fusion output feature; The prediction output unit 703 is used to perform classification prediction on the fused output features to obtain a multi-modal remote sensing image classification result.

[0067] The multimodal remote sensing image classification device 700 provided in the above embodiment can implement the technical solution described in the above multimodal remote sensing image classification method embodiment. The specific implementation principles of the above modules or units can refer to the corresponding contents in the above multimodal remote sensing image classification method embodiment, which will not be repeated here.

[0068] like Figure 8 As shown, the present invention also provides an electronic device 800. The electronic device 800 includes a processor 801, a memory 802 and a display 803. Figure 8 Only some components of the electronic device 800 are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0069] In some embodiments, the processor 801 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes or process data stored in the memory 802, such as the multimodal remote sensing image classification method of the present invention.

[0070] In some embodiments, the processor 801 may be a single server or a server group. The server group may be centralized or distributed. In some embodiments, the processor 801 may be local or remote. In some embodiments, the processor 801 may be implemented in a cloud platform. In one embodiment, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-cloud, etc., or any combination thereof.

[0071] In some embodiments, the memory 802 may be an internal storage unit of the electronic device 800, such as a hard disk or memory of the electronic device 800. In other embodiments, the memory 802 may also be an external storage device of the electronic device 800, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the electronic device 800.

[0072] Furthermore, the memory 802 may include both an internal storage unit of the electronic device 800 and an external storage device. The memory 802 is used to store application software installed in the electronic device 800 and various data.

[0073] In some embodiments, the display 803 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 803 is used to display information of the electronic device 800 and to display a visual user interface. The components 801-803 of the electronic device 800 communicate with each other via a system bus.

[0074] In one embodiment, when the processor 801 executes the multimodal remote sensing image classification program in the memory 802, the following steps may be implemented: The acquired multimodal remote sensing image data is input into a well-trained multimodal remote sensing image classification network, and the spatial spectral feature enhancement and fusion of the multimodal remote sensing image data is performed to obtain the spatial spectral fusion feature, and the global feature enhancement and fusion of the multimodal remote sensing image data is performed to obtain the global fusion feature; Perform parallel Fourier transform frequency domain analysis and dual feature exchange fusion on the spatial spectrum fusion feature and the global fusion feature to obtain the fusion output feature; The fusion output features are classified and predicted to obtain the multimodal remote sensing image classification results.

[0075] It should be understood that: when the processor 801 executes the multimodal remote sensing image classification program in the memory 802, in addition to the above functions, other functions can also be implemented. For details, please refer to the description of the corresponding method embodiment above.

[0076] Furthermore, the embodiment of the present invention does not specifically limit the type of the electronic device 800 mentioned, and the electronic device 800 may be a portable electronic device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a wearable device, a laptop computer, etc. Exemplary embodiments of portable electronic devices include but are not limited to portable electronic devices equipped with IOS, Android, Microsoft or other operating systems. The above-mentioned portable electronic devices may also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, the electronic device 800 may not be a portable electronic device, but a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0077] Accordingly, an embodiment of the present application also provides a computer-readable storage medium, which is used to store computer-readable programs or instructions. When the program or instructions are executed by a processor, it can implement the steps or functions in the multimodal remote sensing image classification method provided by the above-mentioned method embodiments.

[0078] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0079] The multimodal remote sensing image classification method, device, electronic device and storage medium provided by the present invention are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A multimodal remote sensing image classification method, characterized in that: include: Inputting the acquired multimodal remote sensing image data into a well-trained multimodal remote sensing image classification network, performing spatial spectral feature enhancement fusion on the multimodal remote sensing image data to obtain spatial spectral fusion features, and performing global feature enhancement fusion on the multimodal remote sensing image data to obtain global fusion features; Performing parallel Fourier transform frequency domain analysis and dual feature exchange fusion on the spatial spectrum fusion feature and the global fusion feature to obtain a fusion output feature; Classification prediction is performed on the fusion output features to obtain a multimodal remote sensing image classification result.

2. The multimodal remote sensing image classification method according to claim 1, characterized in that: The multimodal remote sensing image data includes spectral image data and radar image data.

3. The multimodal remote sensing image classification method according to claim 1, characterized in that: The step of performing spatial spectral feature enhancement fusion on the multimodal remote sensing image data to obtain spatial spectral fusion features includes: Performing multi-layer residual convolution feature alignment on the multimodal remote sensing image data to obtain aligned modal features, wherein the aligned modal features include spectral modal features and radar modal features; Performing feature dimension reduction and spatial direction flattening operations on the modal features in sequence to obtain a modal feature vector; Performing matrix multiplication on the modal eigenvector along the channel direction to obtain a channel occlusion matrix, and performing matrix multiplication on the modal eigenvector along the spatial direction to obtain a spatial occlusion matrix; The channel occludin matrix is ​​dimensionally compressed and reshaped, and then multiplied with the spectral modal feature to obtain a spectral enhancement feature; the spatial occludin matrix is ​​dimensionally compressed and reshaped, and then multiplied with the spectral modal feature to obtain a spatial enhancement feature; The spectral enhancement feature and the spatial enhancement feature are concatenated to obtain a spatial-spectral fusion feature.

4. The multimodal remote sensing image classification method according to claim 2, characterized in that: The step of performing global feature enhancement and fusion on the multimodal remote sensing image data to obtain global fusion features includes: Performing three-dimensional convolution feature alignment and two-dimensional convolution feature alignment on the spectral image data to obtain spectral alignment features; Performing two-dimensional convolution feature alignment on the radar image data to obtain radar alignment features; Performing global average pooling and convolution sampling feature enhancement on the spectral alignment feature to obtain a spectral global enhancement feature, and performing global average pooling and convolution sampling feature enhancement on the radar alignment feature to obtain a radar global enhancement feature; Multiplying the spectrum global enhancement feature and the spectrum alignment feature to obtain a spectrum global feature, and multiplying the radar global enhancement feature and the radar alignment feature to obtain a radar global feature; The spectral global feature and the radar global feature are added and fused to obtain a global fusion feature.

5. The multimodal remote sensing image classification method according to claim 1, characterized in that: The parallel Fourier transform frequency domain analysis comprises: Performing Fourier transform on the spatial spectrum fusion feature to obtain spatial spectrum frequency domain feature, and performing Fourier transform on the global fusion feature to obtain global frequency domain feature; Performing linear convolution on the spatial spectrum frequency domain feature to obtain a first spatial spectrum complementary feature, and performing linear convolution on the global frequency domain feature to obtain a first global complementary feature; The spatial spectrum frequency domain feature is subjected to feature filtering and inverse Fourier transformation, and then added and fused with the first spatial spectrum complementary feature to obtain a spatial spectrum coding feature. The global frequency domain feature is subjected to feature filtering and inverse Fourier transformation, and then added and fused with the first global complementary feature to obtain a global coding feature.

6. The multimodal remote sensing image classification method according to claim 5, characterized in that: The dual feature exchange and fusion includes: Performing feature learning on the spatial spectrum coding feature to obtain a spatial spectrum learning feature, performing feature learning on the global coding feature to obtain a global learning feature, performing linear convolution on the spatial spectrum coding feature to obtain a second spatial spectrum complementary feature, and performing linear convolution on the global coding feature to obtain a second global complementary feature; Adding and fusing the spatial spectrum learning feature and the second spatial spectrum complementary feature to obtain a spatial spectrum learning fusion feature, and adding and fusing the global learning feature and the second global complementary feature to obtain a global learning fusion feature; The spatial spectrum learning fusion feature and the global learning fusion feature are added and fused to obtain a fusion output feature.

7. The multimodal remote sensing image classification method according to claim 1, characterized in that: The fully trained multimodal remote sensing image classification network is obtained by training an initial multimodal remote sensing image classification network, wherein the training initial multimodal remote sensing image classification network comprises: Determine the total loss function based on exponential label smoothed cross entropy; Decomposing the total loss function into a spatial-spectral feature enhancement fusion branch loss function, a global feature enhancement fusion branch loss function, and a parallel Fourier transform feature fusion branch loss function; The initial multimodal remote sensing image classification network is iteratively trained according to the total loss function, the spatial-spectral feature enhancement fusion branch loss function, the global feature enhancement fusion branch loss function and the parallel Fourier transform feature fusion branch loss function until the loss no longer decreases, thereby obtaining a fully trained multimodal remote sensing image classification network.

8. A multimodal remote sensing image classification device, characterized in that: include: The feature enhancement and fusion unit is used to input the acquired multimodal remote sensing image data into a well-trained multimodal remote sensing image classification network, perform spatial spectral feature enhancement and fusion on the multimodal remote sensing image data to obtain spatial spectral fusion features, and perform global feature enhancement and fusion on the multimodal remote sensing image data to obtain global fusion features; A frequency domain analysis fusion unit, used for performing parallel Fourier transform frequency domain analysis and dual feature exchange fusion on the spatial spectrum fusion feature and the global fusion feature to obtain a fusion output feature; The prediction output unit is used to perform classification prediction on the fusion output features to obtain a multimodal remote sensing image classification result.

9. An electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the multimodal remote sensing image classification method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multimodal remote sensing image classification method according to any one of claims 1 to 7 is implemented.

Images