Collaborative object classification method for optical and SAR remote sensing images based on cross-modal attention

Abstract

The present invention discloses a collaborative object classification method for optical and SAR remote sensing images based on cross-modal attention, which can dynamically model the feature association between optical images and SAR images, adaptively adjust the modal feature weights, highlight the role of key modalities in the expression of object features, and weaken the influence of non-critical information; and through the contextual attention mechanism, capture the global and local feature associations of optical and SAR images, and effectively improve the accuracy and robustness of object classification. The cross-modal attention method proposed in the present invention optimizes the model through a joint loss function, so that it can efficiently utilize multimodal data at both the feature layer and the decision layer, providing more accurate object classification results. This method not only improves the interpretation ability of remote sensing images, but also provides accurate data support for urban planning, environmental protection, disaster monitoring and other fields. It has important value and broad application prospects in practical applications.

Description

Technical Field

[0001] The present invention relates to the technical field of optical and SAR remote sensing image processing, and in particular to a method for collaborative ground object classification of optical and SAR remote sensing images based on cross-modal attention. Background Art

[0002] With the rapid development of remote sensing technology, optical and synthetic aperture radar (SAR) remote sensing have become important pillars of Earth observation. Optical remote sensing images, with their rich color and texture information, play a key role in land cover classification, urban planning, and environmental monitoring. SAR remote sensing images, with their all-day, all-weather observation capabilities and high sensitivity to surface structure, offer unique advantages in disaster monitoring, military reconnaissance, and ocean observation.

[0003] Using remote sensing imagery for object classification provides intuitive and clear surface information for urban planning, environmental protection, disaster monitoring, and other fields, becoming an important scientific basis for related decision-making. With the intensification of global climate change and the acceleration of urbanization, various industries have placed higher demands on the monitoring and management of the surface environment, posing new challenges to the accuracy and efficiency of remote sensing image object classification. However, a single remote sensing data source struggles to comprehensively and accurately capture object information in complex and changing surface environments. Therefore, effectively integrating the advantages of optical and SAR remote sensing to improve the accuracy and robustness of object classification has become a hot topic in remote sensing research. Summary of the Invention

[0004] Aiming at the limitations of optical images being easily affected by weather changes and SAR images lacking texture information, the present invention provides a collaborative ground object classification method for optical and SAR remote sensing images based on cross-modal attention. Through cross-modal fusion, deep fusion and collaborative classification of the two remote sensing data are achieved.

[0005] According to one aspect of the present invention, a method for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention is provided, comprising:

[0006] Acquire optical and SAR remote sensing images;

[0007] The acquired optical and SAR remote sensing images are input into a trained collaborative object classification model to output object categories; wherein the training of the collaborative object classification model includes:

[0008] Construct training datasets of optical and SAR remote sensing images, and extract feature map pairs of optical and SAR remote sensing images in different dimensions;

[0009] Based on the extracted feature map, the texture information in the optical image and the structural information in the SAR image are cross-modally fused in different dimensions to obtain a fused feature map.

[0010] Upsample the fused feature map and perform a jump connection with the fused feature map, and finally decode the ground feature category to which each pixel belongs;

[0011] Design a loss function for model training and output the trained collaborative feature classification model.

[0012] As a further technical solution, feature map pairs of optical and SAR remote sensing images in different dimensions are extracted, including:

[0013] The optical and SAR remote sensing images are input into the feature extraction network, and five layers of feature map pairs are output. The feature extraction network includes a five-layer structure composed of residual convolution. The first layer Stage0 processes the input optical and SAR remote sensing images through convolution, batch normalization, ReLU activation function and maximum pooling operations to obtain 64-channel feature map pairs; the second layer Stage1, the third layer Stage2, the fourth layer Stage3 and the fifth layer Stage4 have 3, 4, 6 and 3 bottleneck modules respectively, and the output of the previous stage is used as the input of the next stage, and so on, to generate five-level feature map pairs.

[0014] As a further technical solution, the texture information in optical images and the structural information in SAR images are cross-modally fused in different dimensions, including:

[0015] The extracted optical and SAR feature maps are sent to the convolutional layer for projection to obtain different projection features;

[0016] The projection features of the two modalities are fed into the attention module, and the attention matrix is ​​obtained through attention calculation. The softmax function is used to convert the attention matrix into a probability distribution to weight the corresponding optical and SAR feature vectors.

[0017] Perform vector multiplication of the obtained optical feature probability distribution, SAR feature probability distribution and projection feature to update the weights of the optical and SAR feature vectors;

[0018] The attention-weighted optical and SAR features are added to the original optical and SAR features to obtain a fused feature map.

[0019] As a further technical solution, the fused feature map is upsampled and jump-connected to the fused feature map, and finally decoded to obtain the ground feature category to which each pixel belongs, including:

[0020] The fused feature map is upsampled using bilinear interpolation, and the upsampling result and the extracted fused feature map are jump-connected using a skip connection. The feature maps obtained by upsampling and skip connection are subjected to two convolution operations and ReLU activation function operations respectively to generate pixel-level classification results.

[0021] As a further technical solution, a loss function is designed for model training, including:

[0022] Stochastic gradient descent is used for training, and the model parameters are optimized by minimizing the cross entropy loss between the predicted probability map and the true value map.

[0023] According to one aspect of the present invention, a system for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention is provided, comprising:

[0024] Acquisition module, used to acquire optical and SAR remote sensing images;

[0025] The classification module is used to input the acquired optical and SAR remote sensing images into the trained collaborative object classification model and output the object category; wherein the training of the collaborative object classification model includes:

[0026] Construct training datasets of optical and SAR remote sensing images, and extract feature map pairs of optical and SAR remote sensing images in different dimensions;

[0027] Based on the extracted feature map, the texture information in the optical image and the structural information in the SAR image are cross-modally fused in different dimensions to obtain a fused feature map.

[0028] Upsample the fused feature map and perform a jump connection with the fused feature map, and finally decode the ground feature category to which each pixel belongs;

[0029] Design a loss function for model training and output the trained collaborative feature classification model.

[0030] According to one aspect of the present invention, a device for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention is provided, comprising a memory and a processor, wherein the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the steps of the method for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention.

[0031] According to one aspect of the present invention, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to perform the steps of the method for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] The present invention proposes a collaborative land feature classification method for optical and SAR remote sensing images based on cross-modal attention, which can effectively solve the shortcomings of optical image land feature characteristics being affected by weather changes and the shortcomings of insufficient texture information in SAR images. The designed BIIM (Bimodal Information Interaction Module) fusion module can dynamically model the correlation between optical images and SAR images, adaptively adjust the feature weights of the two modalities, highlight the key modalities in the feature expression of different land feature types, weaken the influence of non-critical information, perform deep fusion at the feature layer, capture the global and local feature associations of optical and SAR images through contextual attention, further optimize the classification results at the decision layer, and guide the model to focus on key areas through a joint loss function, providing an efficient and accurate solution for achieving high-precision land feature classification applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief introduction will be given below to the drawings used in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 A schematic diagram of a process for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention provided by an embodiment of the present invention;

[0036] Figure 2 A framework diagram of the optical and SAR imagery ground object classification method based on cross-modal attention provided by an embodiment of the present invention;

[0037] Figure 3 A structural diagram of Bottleneck 1 provided in an embodiment of the present invention;

[0038] Figure 4 A structural diagram of Bottleneck 2 provided in an embodiment of the present invention;

[0039] Figure 5 Schematic diagram of the cross-modal BIIM fusion module provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0040] It should be noted that:

[0041] Existing land object classification technologies are primarily divided into traditional methods and deep learning-based approaches. Traditional methods struggle to fully exploit the multi-scale, high-order spatial and spectral features in raw data, resulting in insufficient ability to distinguish complex land object categories (such as buildings and roads). Deep learning-based approaches, on the other hand, are capable of learning high-dimensional, nonlinear, and complex feature representations and have demonstrated significant advantages in various remote sensing tasks in recent years.

[0042] As a key component of artificial intelligence, deep learning has made remarkable progress in recent years in fields such as image recognition and natural language processing. Convolutional neural networks (CNNs), in particular, possess the ability to automatically extract features and efficiently process complex ground feature information in remote sensing imagery. Deep learning technology can automatically extract richer, more abstract feature information from optical and SAR remote sensing imagery, significantly improving the accuracy of ground feature classification. Furthermore, the fusion of multi-source remote sensing data can overcome the limitations of a single data source and further enhance classification accuracy. Deep learning models adaptively learn ground feature features, reducing reliance on artificial prior knowledge and effectively improving the intelligent level of remote sensing image interpretation. They are particularly suitable for processing large-scale, high-resolution remote sensing data.

[0043] Research on collaborative object classification using optical and SAR remote sensing imagery provides solid support for the application of remote sensing technology in urban planning, environmental monitoring, disaster warning, and other fields. This technology not only provides accurate and detailed information about objects but also lays a crucial data foundation for scientific decision-making in related industries, promoting the further development and practical application of remote sensing technology.

[0044] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention are arbitrarily combined with each other to form a new technical solution. This combination is not restricted by the sequence of steps and / or structural composition mode, but must be based on the ability of ordinary technicians in this field to implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that this combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0045] The embodiment of the present invention provides a method for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention, such as Figure 1As shown in FIG, first, optical and SAR remote sensing images are acquired; then, the acquired optical and SAR remote sensing images are input into the trained collaborative object classification model to output the object category.

[0046] Among them, the training of the collaborative land object classification model includes: constructing a training data set of optical and SAR remote sensing images, and extracting feature map pairs in different dimensions of the optical and SAR remote sensing images; based on the extracted feature maps, cross-modally fusing the texture information in the optical image and the structural information in the SAR image in different dimensions to obtain a fused feature map; upsampling the fused feature map and performing a jump connection with the fused feature map, and finally decoding to obtain the land object category to which each pixel point belongs; designing a loss function for model training, and outputting the trained collaborative land object classification model.

[0047] like Figure 2 As shown, the acquired optical and SAR remote sensing images are input into the trained collaborative land object classification model. The data processing process of the collaborative land object classification model includes: extracting feature map pairs (f i,Optic ,f i,SAR ), the extracted feature map pairs are passed through the cross-modal BIIM module to obtain fused feature maps F0, F1, F2, F3, and F4. The cross-fused feature maps are upsampled using the decoding network and skip-connected with the fused feature maps F1, F2, F3, and F4. Finally, the ground feature category to which each pixel belongs is decoded. Specifically, the following steps are included:

[0048] Step 1: Select the target dataset and convert the optical image I Optic SAR image I SAR And the label true value map GT is used as training data, input into the feature extraction network, and obtains five layers of feature map pairs. Optic ,I SAR ) is used as the input of the designed network. The encoding part of the network, i.e. the feature extraction network, is divided into 5 layers. The extracted 5-layer feature map pair (f 0,Optic ,f 0,SAR ), (f 1,Optic ,f 1,SAR ), (f 2,Optic ,f 2,SAR ), (f 3,Optic ,f 3,SAR ), (f 4,Optic ,f 4,SAR ), used as the input of the cross-modal BIIM module to fuse different features of optical and SAR remote sensing;

[0049] The feature extraction network is divided into five layers. Stage 0 processes the input optical and SAR remote sensing images through convolution, BatchNormalization, ReLU activation function, and maximum pooling operations to obtain 64-channel feature map pairs. The subsequent Stage 1, Stage 2, Stage 3, and Stage 4 have 3, 4, 6, and 3 Bottleneck modules respectively. The output of the previous stage is used as the input of the next stage, and so on, to generate five levels of feature maps (f 0,Optic ,f 0,SAR ), (f 1,Optic ,f 1,SAR ), (f 2,Optic ,f 2,SAR ), (f 3,Optic ,f 3,SAR ), (f 4,Optic ,f 4,SAR ).

[0050] Specifically, the feature extraction network consists of 5 layers consisting of residual convolution, in which Stage 0 is used to extract the input optical image (I Optic ), SAR image (I SAR ) uses a 7×7 convolution kernel with a stride of 3 for convolution operation. At this time, the number of output channels is 64. Then it passes through the BatchNormalization layer, and finally passes through the ReLU activation function and the maximum pooling layer to obtain (f 0,Optic ,f 0,SAR ).

[0051] f 0,Optic =MaxPool(ReLU(BN(Conv 7×7,stride＝3 (I Optic ))))

[0052] f 0,SAR =MaxPool(ReLU(BN(Conv 7×7,stride＝3 (I SAR ))))

[0053] Stage1, Stage2, Stage3 and Stage4 have 3, 4, 6 and 3 Bottleneck modules respectively, where the Bottleneck modules are divided into two structures: Bottleneck1 and Bottleneck2. Figure 3As shown in the figure, the input and output dimensions of Bottleneck1 are different. The strides of the three convolutional layers on the left branch are 1, 2, and 1 respectively. The output size and number of channels have changed. The right branch matches the left branch in terms of dimensions by performing a convolution operation on the input, and then performs addition and ReLU operations to obtain the final output. The role of Bottleneck1 is to change the dimension of the network and cannot be connected in series. Figure 4 As shown in the figure, the stride of the three convolutional layers in the left branch of Bottleneck2 are all 1, and their input and output dimensions are the same, so they can be connected in series to deepen the network.

[0054] For the input x, the operation of Bottleneck1 is:

[0055] Bottleneck1=ReLU(Add(Block3(Block2(Block1(x))),Block3(x)))

[0056] Bottleneck2=ReLU(Add(Block3(Block2(Block1(x))),x))

[0057] Block1, Block2, and Block3 are:

[0058] Block1=ReLU(BN(Conv 1×1 (x)))

[0059] Block2=ReLU(BN(Conv 3×3 (x)))

[0060] Block3=BN(Conv 1×1 (x))

[0061] final:

[0062]

[0063] Similarly:

[0064]

[0065] Finally, each stage will generate an optical and SAR feature map pair (f i,Optic ,f i,SAR ).

[0066] Step 2: Encode the 5-layer feature map pairs (f 0,Optic ,f 0,SAR ), (f 1,Optic ,f 1,SAR ), (f2,Optic ,f 2,SAR ), (f 3,Optic ,f 3,SAR ), (f 4,Optic ,f 4,SAR ), input the cross-modal attention BIIM module, fuse the texture information in the optical image and the structural information in the SAR image in different dimensions, weaken the influence of non-critical information, and generate interactive fusion feature maps F0, F1, F2, F3, and F4.

[0067] Step 1 generates five pairs of optical and SAR feature maps in different dimensions. In order to better interact with the two different modalities of optical and SAR images, fully explore the correlation between the two, and enhance the network model's ability to represent different types of objects, this embodiment of the present invention designs an innovative cross-modal attention BIIM module, which aims to dynamically learn the correlation information between optical and SAR images and adjust the feature representation accordingly.

[0068] Specifically, if Figure 5 As shown in the figure, the extracted optical and SAR image features are first sent to the convolution layer for projection to obtain different projection features of the two modalities. The formula is as follows:

[0069] V_optic,Q_optic=Conv 1×1,stride＝1 (f i,optic ),i=0,1,2,3,4

[0070] V_SAR,Q_SAR=Conv 1×1,stride＝1 (f i,SAR ),i=0,1,2,3,4

[0071] Then, the projected features of the two modalities are transformed in dimension. After that, the features of the optical modality and the SAR modality are dot-producted and multiplied by the scaling factor to calculate the attention matrix of the optical and SAR modal feature matrices. This attention matrix contains the feature information of both optical and SAR modalities and complements the feature information of the two modalities to make up for the shortcomings of the single modality information. In this process, the embodiment of the present invention uses the softmax function to convert the attention matrix into an effective probability distribution for weighting the corresponding optical and SAR feature vectors. The formula is as follows:

[0072] AttOut=softmax(Att(Q_optic,Q_SAR))

[0073] Next, the attention-weighted optical and SAR features are added together and output together with the input features of the two modalities. This design aims to preserve the input features to retain more detailed information and enable the network to fully utilize the complementary information between different modalities, thereby enhancing the network's representational capabilities. The final output will include the original input optical features, the attention-weighted features, and the original input SAR features to ensure that the network can fully utilize and integrate information from both modalities. The formula is as follows:

[0074] F i =Add(Matmul(V_optic,AttOut),Matmul(V_SAR,AttOut)),i=0,1,2,3,4.

[0075] Step 3: Upsample the F0, F1, F2, F3, and F4 layers generated in Step 2 layer by layer and fuse them through the designed decoder and the F1, F2, F3, and F4 layers using skip connections. Skip connections address the loss of sensitivity to detail during upsampling and enable information extraction across multiple scales. Finally, convolution operations are used to map features to the number of categories, and the category probability of each pixel is calculated. The loss function is then used to update the model.

[0076] In steps 1 and 2, the resolution of the feature map is gradually reduced through decoding and the cross-modal attention BIIM module to extract high-level semantic information. Therefore, in this step, it is necessary to gradually restore the original resolution through upsampling so that the output feature map is consistent with the input image size. In this step, bilinear interpolation is used for upsampling. For each new pixel, the nearest 2x2 pixel value is selected according to its position on the original image, and the new pixel value is calculated based on the distance weight. The calculation formula is as follows:

[0077]

[0078] Among them I x,y Indicates the pixel value of the target position, I i,j Represents the pixel value of a 2×2 area in the original image, and the weight is determined by the distance from the target position to the pixel position of the original image.

[0079] However, upsampling has the problem of losing sensitivity to detail information. Therefore, a skip connection is used to connect the upsampling result with the features F1, F2, F3, and F4 extracted in step 2. Finally, in order to further extract deep information and enhance nonlinear expression capabilities, the feature maps obtained by upsampling and skip connection are subjected to two convolution operations and ReLU activation function operations respectively. The synergistic effect of the two enables the decoder to gradually refine the target information while restoring the resolution of the feature map layer by layer, and finally generate high-quality pixel-level classification results. The formula is as follows:

[0080] U i =ReLU(Conv(ReLU(Conv(Concat(F i ,Up(U i+1 )))))),i=0,1,2,3

[0081] in:

[0082] U4=F4

[0083] For the U0 generated above, its number of channels is 64, and its height and width are 1 / 2 of the input image respectively. Finally, upsampling, cubic convolution and ReLU activation function operations are used to predict the category result of each pixel. The formula is as follows:

[0084] FinalOut=ReLU(Conv(ReLU(Conv(ReLU(Conv(Up(U0)))))))

[0085] The number of channels in the final generated FinalOut is the same as the number of categories, and the height and width are the same as the input image.

[0086] Specifically, step 3 also includes:

[0087] Step 3.1: Upsample the fused feature U4 (i.e., F4) obtained at the bottom layer in step 2, reducing the fused feature from the original 2048 channels to 1024, and changing the height and width from H / 32 and W / 32 to H / 16 and W / 16 respectively, to obtain the upsampled result;

[0088] Step 3.2: After skip connection between the upsampling result of the bottom layer and the obtained F3, two convolution operations and two ReLU activation functions are performed on each layer to obtain U3;

[0089] Step 3.3: Repeat the previous operation until U0 is obtained. At this time, upsampling, cubic convolution and ReLU activation function operations are used to predict the category result of each pixel.

[0090] Step 3.4: We optimize the model parameters by minimizing the cross entropy loss Lce between the predicted probability map and the true value map:

[0091]

[0092] where Y i Indicates the true label, indicating the change type, with a value of 0 or 1. irepresents the model's prediction value, ranging from 0 to 1. Data augmentation, including flipping, scaling, cropping, and Gaussian blurring, is performed during training. Our model is trained using the stochastic gradient descent (SGD) algorithm with a batch size of 16, an initial learning rate (lr) of 1e-3, a momentum of 0.99, a weight decay of 5e–4, and 200 iterations.

[0093] The trained network is used for validation and fusion quality assessment.

[0094] The method of the present invention will now be further described with reference to a specific embodiment.

[0095] The experimental environment used in this paper is: Python 3.9, NVIDIA RTX 3090 GPU, 24GB of video memory, and the model is implemented in PyTorch. The self-made WHU-OPT-SAR dataset is used as training data. The optimizer, training batch size, and number of iterations are the same as those in the previous article.

[0096] The comparative tests are UNet-ResNet-Optic, UNet-ResNet-SAR, and UNet-ResNet-ADD. The schemes of these three networks are:

[0097] UNet-ResNet-Optic: This solution embeds the ResNet50 migration network model into the Unet encoder part to build an overall network for training and evaluation on single-source data of optical remote sensing images.

[0098] UNet-ResNet-SAR: Similar to UNet-ResNet-Optic, this solution embeds the ResNet50 migration network model into the UNet encoder, but is trained and evaluated on single-source data based on SAR remote sensing imagery.

[0099] UNet-ResNet-ADD: This solution embeds ResNet50 as a transfer learning model into the encoder portion of UNet, creating a dual-branch input network structure. Each feature extraction branch is a separate ResNet50 network, processing optical and SAR images respectively. The optical and SAR features extracted from each branch at the corresponding level are then fed into the ADD fusion module for linear addition.

[0100] As shown in Table 1, three evaluation indicators are used to evaluate the final object classification results on the validation set and test set, namely: accuracy, category average pixel accuracy (MPA) and mean intersection over union (MIoU).

[0101] Table 1 Evaluation table of ground feature classification results

[0102]

[0103]

[0104] It can be seen from the experimental results that various indicators of the method of the present invention are higher than those of other methods.

[0105] The results on the test set showed significant advantages in all evaluation indicators, especially in comparison with other methods. Compared with UNet-ResNet-Optic and UNet-ResNet-SAR with a single data source, UNet-ResNet-ADD and the method of the present invention have improved in the Accuracy indicator. However, UNet-ResNet-ADD showed a downward trend in the MIoU and MPA indicators. This is because the direct addition operation has a negative impact on the optical and SAR remote sensing image features extracted by the network. Although the overall accuracy is improved, the ability to judge certain land object categories is weakened. In contrast, the method of the present invention adopts the BIIM fusion module to effectively fuse the optical and SAR image features. Its various indicators showed an upward trend, which also indirectly verified the effectiveness of the BIIM fusion module. These results further confirm that the use of multi-source data features and effective fusion can achieve better performance than single data source land object classification.

[0106] The above results show that the cross-modal attention BIIM module designed by this method can efficiently fuse the features of two heterogeneous data, dynamically learn the correlation information between optical and SAR images, and adjust the feature representation accordingly, thereby achieving better land object classification results under the full fusion of bimodal data features.

[0107] The implementation basis of each embodiment of the present invention is achieved through programmed processing by a device with processor functionality. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present invention are encapsulated into various modules. Based on this reality, on the basis of the above-mentioned embodiments, an embodiment of the present invention provides a system for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention. The system is used to execute the method for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention in the above-mentioned method embodiment.

[0108] The system includes: an acquisition module for acquiring optical and SAR remote sensing images; a classification module for inputting the acquired optical and SAR remote sensing images into a trained collaborative land object classification model and outputting land object categories; wherein, the training of the collaborative land object classification model includes: constructing a training data set of optical and SAR remote sensing images, and extracting feature map pairs of the optical and SAR remote sensing images in different dimensions; based on the extracted feature maps, cross-modally fusing the texture information in the optical image with the structural information in the SAR image in different dimensions to obtain a fused feature map; upsampling the fused feature map and performing a jump connection with the fused feature map, and finally decoding to obtain the land object category to which each pixel belongs; designing a loss function for model training, and outputting the trained collaborative land object classification model.

[0109] The embodiment of the present invention provides a collaborative land feature classification system for optical and SAR remote sensing images based on cross-modal attention. It addresses the limitations of optical images being susceptible to weather changes and SAR images having insufficient texture information by adopting the above-mentioned modules. The designed BIIM module can dynamically model the correlation between optical images and SAR images, adaptively adjust the feature weights of the two modalities, highlight key modalities in the feature expression of different land feature types, weaken the influence of non-critical information, perform deep fusion at the feature layer, capture the global and local feature correlations of optical and SAR images through contextual attention, further optimize the classification results at the decision layer, and guide the model to focus on key areas through a joint loss function, thereby providing an efficient and accurate solution for achieving high-precision land feature classification applications.

[0110] It should be noted that the system embodiments provided by the present invention are not only used to implement the methods in the above-mentioned method embodiments, but also used to implement the methods in other method embodiments provided by the present invention. The only difference lies in the setting of corresponding functional modules, and the principles thereof are basically the same as the principles of the above-mentioned system embodiments provided by the present invention. As long as those skilled in the art refer to the specific technical solutions in other method embodiments on the basis of the above-mentioned system embodiments, obtain corresponding technical means and technical solutions composed of these technical means by combining technical features, and on the premise of ensuring the practicality of the technical solutions, improve the modules in the above-mentioned system embodiments to obtain corresponding system class embodiments for implementing the methods in other method class embodiments.

[0111] Based on the same inventive concept as the above-mentioned embodiment, an embodiment of the present invention also provides a device for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention, including a memory and a processor, wherein the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the steps of the method for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention.

[0112] In an embodiment of the present invention, the memory may be a non-volatile memory, such as a hard disk drive (HDD) or a solid-state drive (SSD), or a volatile memory (volatile memory), such as a random-access memory (RAM). The memory is any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory in an embodiment of the present invention may also be a circuit or any other device that can implement a storage function, for storing program instructions and / or data.

[0113] In the embodiments of the present invention, the processor may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field-programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present invention may be directly implemented and executed by a hardware processor, or by a combination of hardware and software modules within the processor.

[0114] Based on the same inventive concept as the above-mentioned embodiment, an embodiment of the present invention also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions enable the computer to perform the steps of the method for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention.

[0115] In summary, the present invention aims at the limitations of optical images being easily affected by weather changes and SAR images lacking texture information, and designs a cross-modal attention BIIM module to achieve deep fusion and collaborative classification of two types of remote sensing data. This method can dynamically model the feature associations between optical images and SAR images, adaptively adjust the modal feature weights, highlight the role of key modalities in the expression of land feature, and weaken the influence of non-critical information. Through the contextual attention mechanism, the global and local feature associations of optical and SAR images are captured, effectively improving the accuracy and robustness of land object classification. The cross-modal attention method proposed in the present invention optimizes the model through a joint loss function, so that it can efficiently utilize multimodal data at both the feature layer and the decision layer, providing more accurate land object classification results. This method not only improves the interpretation ability of remote sensing images, but also provides accurate data support for urban planning, environmental protection, disaster monitoring and other fields. It has important value and broad application prospects in practical applications.

[0116] The terms "including" and "having" and any variations thereof in the description and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions, for example, a process, method, system, product or apparatus that includes a series of steps or units is not necessarily limited to the steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A collaborative object classification method for optical and SAR remote sensing images based on cross-modal attention, characterized by: include: Acquire optical and SAR remote sensing images; The acquired optical and SAR remote sensing images are input into a trained collaborative object classification model to output object categories; wherein the training of the collaborative object classification model includes: Construct training datasets of optical and SAR remote sensing images, and extract feature map pairs of optical and SAR remote sensing images in different dimensions; Based on the extracted feature maps, the texture information in the optical image and the structural information in the SAR image are cross-modally fused in different dimensions to obtain a fused feature map, including: sending the extracted optical and SAR feature map pairs to the convolution layer for projection to obtain different projection features; sending the obtained projection features of the two modalities to the attention module, and obtaining the attention matrix through attention calculation, using the softmax function to convert the attention matrix into a probability distribution to weight the corresponding optical and SAR feature vectors; performing vector multiplication on the obtained optical feature probability distribution, SAR feature probability distribution and projection feature to update the weights of the optical and SAR feature vectors; adding the attention-weighted optical and SAR features to the original optical and SAR features to obtain a fused feature map; Upsample the fused feature map and perform a jump connection with the fused feature map, and finally decode the ground feature category to which each pixel belongs; Design a loss function for model training and output the trained collaborative feature classification model.

2. The method for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention according to claim 1 is characterized in that: Extract feature map pairs in different dimensions of optical and SAR remote sensing images, including: The optical and SAR remote sensing images are input into the feature extraction network, and five layers of feature map pairs are output. The feature extraction network includes a five-layer structure composed of residual convolution. The first layer Stage0 processes the input optical and SAR remote sensing images through convolution, batch normalization, ReLU activation function and maximum pooling operations to obtain 64-channel feature map pairs; the second layer Stage1, the third layer Stage2, the fourth layer Stage3 and the fifth layer Stage4 have 3, 4, 6 and 3 bottleneck modules respectively, and the output of the previous stage is used as the input of the next stage, and so on, to generate five-level feature map pairs.

3. The method for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention according to claim 1 is characterized in that: The fused feature map is upsampled and skip-connected with the fused feature map, and finally decoded to obtain the ground feature category to which each pixel belongs, including: The fused feature map is upsampled using bilinear interpolation, and the upsampling result and the extracted fused feature map are jump-connected using a skip connection. The feature maps obtained by upsampling and skip connection are subjected to two convolution operations and ReLU activation function operations respectively to generate pixel-level classification results.

4. The method for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention according to claim 1, characterized in that: Design loss functions for model training, including: Stochastic gradient descent is used for training, and the model parameters are optimized by minimizing the cross entropy loss between the predicted probability map and the true value map.

5. A collaborative object classification system for optical and SAR remote sensing images based on cross-modal attention, characterized by: include: Acquisition module, used to acquire optical and SAR remote sensing images; The classification module is used to input the acquired optical and SAR remote sensing images into the trained collaborative object classification model and output the object category; wherein the training of the collaborative object classification model includes: Construct training datasets of optical and SAR remote sensing images, and extract feature map pairs of optical and SAR remote sensing images in different dimensions; Based on the extracted feature maps, the texture information in the optical image and the structural information in the SAR image are cross-modally fused in different dimensions to obtain a fused feature map, including: sending the extracted optical and SAR feature map pairs to the convolution layer for projection to obtain different projection features; sending the obtained projection features of the two modalities to the attention module, and obtaining the attention matrix through attention calculation, using the softmax function to convert the attention matrix into a probability distribution to weight the corresponding optical and SAR feature vectors; performing vector multiplication on the obtained optical feature probability distribution, SAR feature probability distribution and projection feature to update the weights of the optical and SAR feature vectors; adding the attention-weighted optical and SAR features to the original optical and SAR features to obtain a fused feature map; Upsample the fused feature map and perform a jump connection with the fused feature map, and finally decode the ground feature category to which each pixel belongs; Design a loss function for model training and output the trained collaborative feature classification model.

6. A device for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention, characterized in that: The invention comprises a memory and a processor, wherein the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the steps of the method for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer instructions, which enable the computer to execute the steps of the method for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention as described in any one of claims 1 to 4.

Images