Optical and SAR remote sensing image collaborative ground feature classification method based on cross-modal attention

Abstract

The invention discloses an optical and SAR remote sensing image collaborative ground feature classification method based on cross-modal attention. According to the method, feature association between an optical image and an SAR image can be dynamically modeled, modal feature weights can be adaptively adjusted, the effect of a key modal in ground feature expression is highlighted, and the influence of non-key information is weakened; and through a context attention mechanism, global and local feature association of optical and SAR images is captured, and the precision and robustness of ground feature classification are effectively improved. According to the cross-modal attention method provided by the invention, the model is optimized through the joint loss function, so that multi-modal data can be efficiently utilized in a feature layer and a decision layer, and a ground feature classification result with higher precision is provided. The method not only improves the interpretation capability of the remote sensing image, but also provides accurate data support for the fields of urban planning, environmental protection, disaster monitoring and the like, and has important value and wide application prospect in practical application.

Description

Technical Field

[0001] The present invention relates to the technical field of optical and SAR remote sensing image processing, and in particular to a method for collaborative ground object classification of optical and SAR remote sensing images based on cross-modal attention. Background Art

[0002] With the rapid development of remote sensing technology, optical remote sensing and synthetic aperture radar (SAR) remote sensing have become important pillars in the field of earth observation. Optical remote sensing images play a key role in the fields of surface cover classification, urban planning and environmental monitoring with their rich color and texture information. SAR remote sensing images, with their all-day, all-weather observation capabilities and high sensitivity to surface structure information, have shown unique advantages in disaster monitoring, military reconnaissance and ocean observation.

[0003] Using remote sensing images to classify objects can provide intuitive and clear surface information for urban planning, environmental protection, disaster monitoring and other fields, and become an important scientific basis for related decision-making. With the intensification of global climate change and the acceleration of urbanization, various industries have put forward higher requirements for the monitoring and management of the surface environment, which has also posed new challenges to the accuracy and efficiency of remote sensing image object classification. However, a single remote sensing data source is difficult to obtain comprehensive and accurate information on objects in a complex and changing surface environment. Therefore, how to effectively integrate the advantages of optical remote sensing and SAR remote sensing to improve the accuracy and robustness of object classification has become one of the hot topics in the field of remote sensing research. Summary of the invention

[0004] Aiming at the limitations of optical images being easily affected by weather changes and SAR images lacking texture information, the present invention provides a collaborative object classification method for optical and SAR remote sensing images based on cross-modal attention. Through cross-modal fusion, deep fusion and collaborative classification of the two remote sensing data are achieved.

[0005] According to one aspect of the present invention, a method for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention is provided, comprising:

[0006] Acquire optical and SAR remote sensing images;

[0007] The acquired optical and SAR remote sensing images are input into the trained collaborative object classification model to output the object category; wherein the training of the collaborative object classification model includes:

[0008] Construct training datasets of optical and SAR remote sensing images, and extract feature map pairs of optical and SAR remote sensing images in different dimensions;

[0009] Based on the extracted feature map, the texture information in the optical image and the structural information in the SAR image are cross-modally fused in different dimensions to obtain a fused feature map.

[0010] The fused feature map is upsampled and skip-connected with the fused feature map, and finally decoded to obtain the ground object category to which each pixel belongs;

[0011] Design a loss function for model training and output the trained collaborative feature classification model.

[0012] As a further technical solution, feature map pairs of optical and SAR remote sensing images in different dimensions are extracted, including:

[0013] The optical and SAR remote sensing images are input into a feature extraction network, and five layers of feature map pairs are output. The feature extraction network includes a five-layer structure composed of residual convolution. The first layer Stage0 processes the input optical and SAR remote sensing images through convolution, batch normalization, ReLU activation function and maximum pooling operations to obtain 64-channel feature map pairs; the second layer Stage1, the third layer Stage2, the fourth layer Stage3 and the fifth layer Stage4 have 3, 4, 6 and 3 bottleneck modules respectively, and the output of the previous Stage is used as the input of the next Stage, and so on, to generate five-level feature map pairs.

[0014] As a further technical solution, the texture information in the optical image and the structural information in the SAR image are cross-modally fused in different dimensions, including:

[0015] The extracted optical and SAR feature map pairs are sent to the convolutional layer for projection to obtain different projection features;

[0016] The projection features of the two modalities are sent to the attention module, and the attention matrix is ​​obtained through attention calculation. The softmax function is used to convert the attention matrix into a probability distribution to weight the corresponding optical and SAR feature vectors.

[0017] The obtained optical feature probability distribution, SAR feature probability distribution and projection feature are vector-multiplied to update the weights of the optical and SAR feature vectors;

[0018] The attention-weighted optical and SAR features are added to the original optical and SAR features to obtain a fused feature map.

[0019] As a further technical solution, the fused feature map is upsampled and jump-connected with the fused feature map, and finally decoded to obtain the ground object category to which each pixel belongs, including:

[0020] The fused feature map is upsampled using bilinear interpolation, and the upsampling result and the extracted fused feature map are jump-connected using a jump connection. The feature maps obtained by upsampling and skip connection are respectively subjected to two convolution operations and ReLU activation function operations to generate pixel-level classification results.

[0021] As a further technical solution, a loss function is designed for model training, including:

[0022] Stochastic gradient descent is used for training, and model parameters are optimized by minimizing the cross entropy loss between the predicted probability map and the true value map.

[0023] According to one aspect of the present invention, there is provided a system for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention, comprising:

[0024] Acquisition module, used to acquire optical and SAR remote sensing images;

[0025] The classification module is used to input the acquired optical and SAR remote sensing images into the trained collaborative object classification model and output the object category; wherein the training of the collaborative object classification model includes:

[0026] Construct training datasets of optical and SAR remote sensing images, and extract feature map pairs of optical and SAR remote sensing images in different dimensions;

[0027] Based on the extracted feature map, the texture information in the optical image and the structural information in the SAR image are cross-modally fused in different dimensions to obtain a fused feature map.

[0028] The fused feature map is upsampled and skip-connected with the fused feature map, and finally decoded to obtain the ground object category to which each pixel belongs;

[0029] Design a loss function for model training and output the trained collaborative feature classification model.

[0030] According to one aspect of the present invention specification, there is provided a device for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention, comprising a memory and a processor, wherein the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the steps of the method for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention.

[0031] According to one aspect of the present specification, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions enable the computer to perform the steps of the method for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] The present invention proposes a collaborative object classification method for optical and SAR remote sensing images based on cross-modal attention, which can effectively solve the shortcomings of optical image object features being affected by weather changes and the shortcomings of insufficient texture information in SAR images. The designed BIIM (Bimodal Information Interaction Module) fusion module can dynamically model the correlation between optical images and SAR images, adaptively adjust the feature weights of the two modalities, highlight the key modes in the feature expression of different object types, weaken the influence of non-critical information, perform deep fusion at the feature layer, capture the global and local feature associations of optical and SAR images through contextual attention, further optimize the classification results at the decision layer, and guide the model to focus on key areas through a joint loss function, thereby providing an efficient and accurate solution for realizing high-precision object classification applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief introduction is given below to the drawings used in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 A schematic diagram of a flow chart of a method for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention provided by an embodiment of the present invention;

[0036] Figure 2 A framework diagram of the optical and SAR image object classification method based on cross-modal attention provided by an embodiment of the present invention;

[0037] Figure 3 A structural diagram of Bottleneck 1 provided in an embodiment of the present invention;

[0038] Figure 4 A structural diagram of Bottleneck 2 provided in an embodiment of the present invention;

[0039] Figure 5 A schematic diagram of a cross-modal BIIM fusion module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] It should be noted that:

[0041] Existing ground object classification technologies are mainly divided into traditional methods and deep learning-based methods. Traditional methods are difficult to fully exploit the multi-scale, high-order spatial and spectral features in the original data, resulting in insufficient ability to distinguish complex ground object categories (such as buildings, roads, etc.). Deep learning-based methods can learn high-dimensional, nonlinear and complex feature representations, and have shown significant advantages in various remote sensing tasks in recent years.

[0042] As an important part of artificial intelligence, deep learning has made remarkable progress in fields such as image recognition and natural language processing in recent years. In particular, convolutional neural networks (CNNs) have the ability to automatically extract features and can efficiently process complex ground feature information in remote sensing images. Through deep learning technology, richer and more abstract feature information can be automatically extracted from optical and SAR remote sensing images, thereby significantly improving the accuracy of ground object classification. In addition, the fusion of multi-source remote sensing data can make up for the limitations of a single data source and further enhance the accuracy of classification. Deep learning models adaptively learn ground feature features, reduce reliance on artificial prior knowledge, effectively improve the intelligence level of remote sensing image interpretation, and are particularly suitable for processing large-scale, high-resolution remote sensing data.

[0043] The research on the collaborative object classification technology of optical and SAR remote sensing images provides solid support for the application of remote sensing technology in urban planning, environmental monitoring, disaster warning and other fields. This technology can not only provide accurate and detailed information on objects, but also lay an important data foundation for scientific decision-making in related industries, and promote the further development and practical application of remote sensing technology.

[0044] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention are arbitrarily combined with each other to form a new technical solution. This combination is not restricted by the sequence of steps and / or the structural composition mode, but must be based on the ability of ordinary technicians in this field to achieve. When the combination of technical solutions is contradictory or cannot be achieved, it should be considered that this combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0045] The embodiment of the present invention provides a method for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention, such as Figure 1As shown, first, optical and SAR remote sensing images are acquired; then, the acquired optical and SAR remote sensing images are input into the trained collaborative object classification model to output the object category.

[0046] Among them, the training of the collaborative land object classification model includes: constructing a training data set of optical and SAR remote sensing images, and extracting feature map pairs in different dimensions of the optical and SAR remote sensing images; based on the extracted feature maps, cross-modally fusing the texture information in the optical image with the structural information in the SAR image in different dimensions to obtain a fused feature map; upsampling the fused feature map and performing a jump connection with the fused feature map, and finally decoding to obtain the land object category to which each pixel belongs; designing a loss function for model training, and outputting the trained collaborative land object classification model.

[0047] like Figure 2 As shown, the acquired optical and SAR remote sensing images are input into the trained collaborative object classification model. The data processing process of the collaborative object classification model includes: extracting feature map pairs (f i,Optic ,f i,SAR ), the extracted feature map pairs are passed through the cross-modal BIIM module to obtain the fused feature map F 0 , F 1 , F 2 , F 3 , F 4 , use the decoding network to upsample the fused feature map and fused feature map F 1 , F 2 , F 3 , F 4 Perform skip connections and finally decode the ground object category to which each pixel belongs. The specific steps include:

[0048] Step 1: Select the target dataset and convert the optical image I Optic SAR image I SAR and the label truth map GT as training data, input into the feature extraction network, and obtain five layers of feature map pairs. Optic ,I SAR ) is used as the input of the designed network. The encoding part of the network, i.e., the feature extraction network, is divided into 5 layers. The extracted 5-layer feature map pair (f 0,Optic ,f 0,SAR ), (f 1,Optic ,f 1,SAR ), (f 2,Optic ,f 2,SAR ), (f 3,Optic ,f 3,SAR ), (f 4,Optic ,f 4,SAR), as the input of the cross-modal BIIM module for fusing different features of optical and SAR remote sensing;

[0049] The above feature extraction network is divided into 5 layers, among which Stage 0 processes the input optical and SAR remote sensing images through convolution, BatchNormalization, ReLU activation function, and maximum pooling operations to obtain 64-channel feature map pairs. The subsequent Stage 1, Stage 2, Stage 3, and Stage 4 have 3, 4, 6, and 3 Bottleneck modules respectively, in which the output of the previous stage is used as the input of the next stage, and so on, to generate five levels of feature maps (f 0,Optic ,f 0,SAR ), (f 1,Optic ,f 1,SAR ), (f 2,Optic ,f 2,SAR ), (f 3,Optic ,f 3,SAR ), (f 4,Optic ,f 4,SAR ).

[0050] Specifically, the feature extraction network consists of five layers consisting of residual convolution, in which Stage 0 performs a Optic ), SAR image (I SAR ) uses a 7×7 convolution kernel with a stride of 3 for convolution operation. At this time, the number of output channels is 64. Then it passes through the BatchNormalization layer, and finally passes through the ReLU activation function and the maximum pooling layer to obtain (f 0,Optic ,f 0,SAR ).

[0051] f 0,Optic =MaxPool(ReLU(BN(Conv 7×7,stride＝3 (I Optic ))))

[0052] f 0,SAR =MaxPool(ReLU(BN(Conv 7×7,stride＝3 (I SAR ))))

[0053] Stage1, Stage2, Stage3 and Stage4 have 3, 4, 6 and 3 Bottleneck modules respectively, where Bottleneck modules are divided into two structures: Bottleneck1 and Bottleneck2. Figure 3As shown in the figure, the input dimension and output dimension of Bottleneck1 are different. The strides of the three convolutional layers on the left branch are 1, 2, and 1 respectively. The output size and number of channels have changed. The right branch matches the left branch dimension by performing convolution operations on the input, and then performs addition and ReLU operations to obtain the final output. The function of Bottleneck1 is to change the dimension of the network and cannot be connected in series. Figure 4 As shown in the figure, the stride of the three convolutional layers in the left branch of Bottleneck2 is 1, and their input and output dimensions are the same, so they can be connected in series to deepen the network.

[0054] For the input x, the operation of Bottleneck1 is:

[0055] Bottleneck1=ReLU(Add(Block3(Block2(Block1(x))),Block3(x)))

[0056] Bottleneck2=ReLU(Add(Block3(Block2(Block1(x))),x))

[0057] Block1, Block2, and Block3 are:

[0058] Block1 = ReLU(BN(Conv 1×1 (x)))

[0059] Block2 = ReLU (BN (Conv 3×3 (x)))

[0060] Block3=BN(Conv 1×1 (x)

[0061] final:

[0062]

[0063] Similarly:

[0064]

[0065] Finally, each stage will generate an optical and SAR feature map pair (f i,Optic ,f i,SAR ).

[0066] Step 2: Encode the 5-layer feature map pairs (f 0,Optic ,f 0,SAR ), (f 1,Optic ,f 1,SAR ), (f2,Optic ,f 2,SAR ), (f 3,Optic ,f 3,SAR ), (f 4,Optic ,f 4,SAR ), input the cross-modal attention BIIM module, fuse the texture information in the optical image and the structural information in the SAR image in different dimensions, weaken the influence of non-critical information, and generate an interactive fusion feature map F 0 , F 1 , F 2 , F 3 , F 4 .

[0067] Step 1 generates five pairs of optical and SAR feature maps in different dimensions. In order to better interact information between the two different modalities of optical and SAR images, fully explore the correlation between the two, and enhance the network model's ability to represent different types of objects, an innovative cross-modal attention BIIM module is designed in this embodiment of the present invention, which aims to dynamically learn the correlation information between optical and SAR images and adjust the feature representation accordingly.

[0068] Specifically, if Figure 5 As shown in the figure, the extracted optical and SAR image features are first sent to the convolution layer for projection to obtain different projection features of the two modalities. The formula is as follows:

[0069] V_optic,Q_optic=Conv 1×1,stride＝1 (f i,optic ),i=0,1,2,3,4

[0070] V_SAR,Q_SAR=Conv 1×1,stride＝1 (f i,SAR ),i=0,1,2,3,4

[0071] Then, the projection features of the two modes are transformed in dimension, and then the features of the optical mode and the features of the SAR mode are dot-producted and multiplied by the scaling factor to calculate the attention matrix of the optical and SAR modal feature matrices. The attention matrix has the feature information of both the optical and SAR modes, and complements the feature information of the two modes to make up for the deficiency of the single modality information. In this process, the embodiment of the present invention uses the softmax function to convert the attention matrix into an effective probability distribution for weighting the corresponding optical and SAR feature vectors. The formula is as follows:

[0072] AttOut=softmax(Att(Q_optic,Q_SAR))

[0073] Next, the attention-weighted optical and SAR features are added and output together with the input features of the two modalities. The purpose of this design is to retain the input features to retain more detailed information and enable the network to fully utilize the complementary information between different modalities, thereby enhancing the network's representation ability. The final output will include the original input optical features, attention-weighted features, and original input SAR features to ensure that the network can fully utilize and integrate the information of the two modalities. The formula is as follows:

[0074] F i =Add(Matmul(V_optic,AttOut),Matmul(V_SAR,AttOut)),i=0,1,2,3,4.

[0075] Step 3: Place the different sizes of F generated in step 2 0 , F 1 , F 2 , F 3 , F 4 Upsampling layer by layer and passing through the designed decoder and skip connection F 1 , F 2 , F 3 , F 4 The skip connection solves the problem of losing sensitivity to detail information during upsampling and extracts information across multiple scales. Finally, the convolution operation is used to map the features to the number of categories, and the category probability of each pixel is calculated, and the loss function is calculated to update the model.

[0076] In step 1 and step 2, the resolution of the feature map is gradually reduced through decoding and the cross-modal attention BIIM module to extract high-level semantic information. Therefore, in this step, the original resolution needs to be gradually restored by upsampling so that the output feature map is consistent with the input image size. In this step, bilinear interpolation is used for upsampling. For each new pixel, the nearest 2x2 pixel value is selected according to its position on the original image, and the new pixel value is calculated according to the distance weight. The calculation formula is as follows:

[0077]

[0078] Among them I x,y Represents the pixel value of the target position, I i,j Represents the pixel value of the 2×2 area in the original image, and the weight is determined by the distance from the target position to the pixel position of the original image.

[0079] However, upsampling has the problem of losing sensitivity to detail information, so a skip connection is used to connect the upsampling result and the feature F extracted in step 2. 1 , F 2 , F 3, F 4 Finally, in order to further extract deep information and enhance nonlinear expression capabilities, the feature maps obtained by upsampling and skip connections are subjected to two convolution operations and ReLU activation function operations. The synergy of the two enables the decoder to gradually refine the target information while restoring the resolution of the feature map layer by layer, and finally generate high-quality pixel-level classification results. The formula is as follows:

[0080] U i =ReLU(Conv(ReLU(Conv(Concat(F i ,Up(U i+1 )))))),i=0,1,2,3

[0081] in:

[0082] U 4 =F 4

[0083] For the above generated U 0 , whose number of channels is 64, and the height and width are 1 / 2 of the input image respectively. Finally, upsampling, cubic convolution and ReLU activation function operations are used to predict the category result of each pixel. The formula is as follows:

[0084] FinalOut=ReLU(Conv(ReLU(Conv(ReLU(Conv(Up(U 0 )))))))

[0085] The number of channels in the final generated FinalOut is the number of categories, and the height and width are the same as the input image.

[0086] Specifically, step 3 also includes:

[0087] Step 3.1: The fusion feature U obtained at the bottom layer in step 2 4 (i.e. F 4 ) is upsampled, the fusion feature is reduced from the original 2048 channels to 1024, and the height and width are changed from H / 32 and W / 32 to H / 16 and W / 16 respectively, to obtain the upsampling result;

[0088] Step 3.2: Combine the result of upsampling the bottom layer with the obtained F 3 After the jump connection, two convolution operations and two ReLU activation functions are performed to obtain U 3 ;

[0089] Step 3.3: Repeat the previous operation until you get U 0 , at this time, upsampling, cubic convolution and ReLU activation function operations are used to predict the category result of each pixel.

[0090] Step 3.4: We optimize the model parameters by minimizing the cross entropy loss Lce between the predicted probability map and the true value map:

[0091]

[0092] where Y i Indicates the true label, indicating the change type, and the value is 0 or 1. i Represents the predicted value of the model, ranging from 0 to 1. Data augmentation is performed during the training phase, including flipping, scaling, cropping, and Gaussian blurring. Our model is trained using the stochastic gradient descent (SGD) algorithm, with a batch size of 16, an initial learning rate (lr) of 1e-3, a momentum of 0.99, a weight decay of 5e–4, and 200 iterations.

[0093] The trained network is used for validation and fusion quality assessment.

[0094] The method of the present invention is now further described in conjunction with a specific implementation manner.

[0095] The environment used in the experiment of this invention is: Python 3.9, NVIDIA RTX 3090 GPU, video memory is 24G, and the model is implemented in PyTorch. The homemade WHU-OPT-SAR dataset is used as training data, and the optimizer, training batch and number of iterations are the same as those in the previous article.

[0096] The comparative tests are UNet-ResNet-Optic, UNet-ResNet-SAR, and UNet-ResNet-ADD. The solutions of these three networks are:

[0097] UNet-ResNet-Optic: This solution embeds the ResNet50 migration network model into the Unet encoder part to build an overall network for training and evaluation on single-source data of optical remote sensing images.

[0098] UNet-ResNet-SAR: Similar to UNet-ResNet-Optic, this solution embeds the ResNet50 migration network model into the Unet encoder part, but is trained and evaluated on single-source data based on SAR remote sensing images.

[0099] UNet-ResNet-ADD: This solution embeds ResNet50 as a transfer learning model into the encoder part of Unet and builds a dual-branch input network structure. Each feature extraction branch is an independent ResNet50 network, which is used to process optical images and SAR images respectively. Then, the optical and SAR features of the corresponding level extracted from each branch are sent to the ADD fusion module for linear addition.

[0100] As shown in Table 1, three evaluation indicators are used to evaluate the final object classification results on the validation set and the test set, namely: accuracy, category average pixel accuracy (MPA) and mean intersection over union (MIoU).

[0101] Table 1 Evaluation table of ground feature classification results

[0102]

[0103]

[0104] It can be seen from the experimental results that various indicators of the method of the present invention are higher than those of other methods.

[0105] The results on the test set showed significant advantages in all evaluation indicators, especially in comparison with other methods. Compared with UNet-ResNet-Optic and UNet-ResNet-SAR of single data source, UNet-ResNet-ADD and the method of the present invention have improved in the Accuracy indicator. However, UNet-ResNet-ADD showed a downward trend in MIoU and MPA indicators. This is because the direct addition operation has a negative impact on the optical and SAR remote sensing image features extracted by the network. Although the overall accuracy is improved, the ability to judge certain types of objects is weakened. In contrast, the method of the present invention adopts the BIIM fusion module, which effectively integrates the optical and SAR image features, and its various indicators show an upward trend, which also indirectly verifies the effectiveness of the BIIM fusion module. These results further confirm that the use of multi-source data features and effective fusion can achieve better performance than single data source object classification.

[0106] The above results show that the cross-modal attention BIIM module designed by this method can efficiently fuse the features of two heterogeneous data, dynamically learn the correlation information between optical and SAR images, and adjust the feature representation accordingly, thereby achieving better land object classification results under the full fusion of bimodal data features.

[0107] The implementation basis of each embodiment of the present invention is realized by programmed processing through a device with a processor function. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present invention are encapsulated into various modules. Based on this reality, on the basis of the above embodiments, an embodiment of the present invention provides an optical and SAR remote sensing image collaborative object classification system based on cross-modal attention, which is used to execute the optical and SAR remote sensing image collaborative object classification method based on cross-modal attention in the above method embodiment.

[0108] The system includes: an acquisition module for acquiring optical and SAR remote sensing images; a classification module for inputting the acquired optical and SAR remote sensing images into a trained collaborative land object classification model and outputting land object categories; wherein the training of the collaborative land object classification model includes: constructing a training data set of optical and SAR remote sensing images, and extracting feature map pairs of optical and SAR remote sensing images in different dimensions; based on the extracted feature maps, cross-modally fusing texture information in the optical image with structural information in the SAR image in different dimensions to obtain a fused feature map; upsampling the fused feature map and performing a jump connection with the fused feature map, and finally decoding to obtain the land object category to which each pixel belongs; designing a loss function for model training, and outputting the trained collaborative land object classification model.

[0109] The optical and SAR remote sensing image collaborative object classification system based on cross-modal attention provided by the embodiment of the present invention is aimed at the limitations of optical images being easily affected by weather changes and SAR images having insufficient texture information. The above-mentioned several modules are adopted. The designed BIIM module can dynamically model the correlation between optical images and SAR images, adaptively adjust the feature weights of the two modalities, highlight the key modalities in the feature expression of different object types, weaken the influence of non-critical information, perform deep fusion at the feature layer, capture the global and local feature associations of optical and SAR images through contextual attention, further optimize the classification results at the decision layer, and guide the model to focus on key areas through a joint loss function, thereby providing an efficient and accurate solution for realizing high-precision object classification applications.

[0110] It should be noted that the system embodiments provided by the present invention are not only used to implement the methods in the above-mentioned method embodiments, but also used to implement the methods in other method embodiments provided by the present invention. The only difference lies in the setting of corresponding functional modules, and the principles thereof are basically the same as the principles of the above-mentioned system embodiments provided by the present invention. As long as technical personnel in this field refer to the specific technical solutions in other method embodiments on the basis of the above-mentioned system embodiments, obtain corresponding technical means and technical solutions composed of these technical means by combining technical features, and on the premise of ensuring the practicality of the technical solutions, they will improve the modules in the above-mentioned system embodiments to obtain corresponding system class embodiments for implementing the methods in other method class embodiments.

[0111] Based on the same inventive concept as the above-mentioned embodiment, an embodiment of the present invention also provides an optical and SAR remote sensing image collaborative object classification device based on cross-modal attention, including a memory and a processor, wherein the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the steps of the optical and SAR remote sensing image collaborative object classification method based on cross-modal attention.

[0112] In an embodiment of the present invention, the memory may be a non-volatile memory, such as a hard disk drive (HDD) or a solid-state drive (SSD), etc., or a volatile memory (volatile memory), such as a random-access memory (RAM). The memory is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory in an embodiment of the present invention may also be a circuit or any other device that can implement a storage function, for storing program instructions and / or data.

[0113] In the embodiments of the present invention, the processor may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present invention may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0114] Based on the same inventive concept as the above-mentioned embodiment, an embodiment of the present invention also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions enable the computer to perform the steps of the method for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention.

[0115] In summary, the present invention designs a cross-modal attention BIIM module to address the limitations of optical images being susceptible to weather changes and SAR images lacking texture information, thereby achieving deep fusion and collaborative classification of two types of remote sensing data. This method can dynamically model the feature associations between optical images and SAR images, adaptively adjust the modal feature weights, highlight the role of key modalities in the expression of ground object features, and weaken the influence of non-critical information. Through the contextual attention mechanism, the global and local feature associations of optical and SAR images are captured, effectively improving the accuracy and robustness of ground object classification. The cross-modal attention method proposed in the present invention optimizes the model through a joint loss function, so that multi-modal data can be efficiently utilized at both the feature layer and the decision layer, providing higher-precision ground object classification results. This method not only improves the interpretation capability of remote sensing images, but also provides accurate data support for urban planning, environmental protection, disaster monitoring and other fields, and has important value and broad application prospects in practical applications.

[0116] The terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions, for example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to the steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A collaborative object classification method for optical and SAR remote sensing images based on cross-modal attention, characterized in that: include: Acquire optical and SAR remote sensing images; The acquired optical and SAR remote sensing images are input into the trained collaborative object classification model to output the object category; wherein the training of the collaborative object classification model includes: Construct training datasets of optical and SAR remote sensing images, and extract feature map pairs of optical and SAR remote sensing images in different dimensions; Based on the extracted feature map, the texture information in the optical image and the structural information in the SAR image are cross-modally fused in different dimensions to obtain a fused feature map. The fused feature map is upsampled and skip-connected with the fused feature map, and finally decoded to obtain the ground object category to which each pixel belongs; Design a loss function for model training and output the trained collaborative feature classification model.

2. The method for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention according to claim 1, characterized in that: Extract feature map pairs in different dimensions of optical and SAR remote sensing images, including: The optical and SAR remote sensing images are input into a feature extraction network, and five layers of feature map pairs are output. The feature extraction network includes a five-layer structure composed of residual convolution. The first layer Stage0 processes the input optical and SAR remote sensing images through convolution, batch normalization, ReLU activation function and maximum pooling operations to obtain 64-channel feature map pairs; the second layer Stage1, the third layer Stage2, the fourth layer Stage3 and the fifth layer Stage4 have 3, 4, 6 and 3 bottleneck modules respectively, and the output of the previous Stage is used as the input of the next Stage, and so on, to generate five-level feature map pairs.

3. The method for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention according to claim 1, characterized in that: The texture information in the optical image and the structural information in the SAR image are cross-modally fused in different dimensions, including: The extracted optical and SAR feature map pairs are sent to the convolutional layer for projection to obtain different projection features; The projection features of the two modalities are sent to the attention module, and the attention matrix is ​​obtained through attention calculation. The softmax function is used to convert the attention matrix into a probability distribution to weight the corresponding optical and SAR feature vectors. The obtained optical feature probability distribution, SAR feature probability distribution and projection feature are vector-multiplied to update the weights of the optical and SAR feature vectors; The attention-weighted optical and SAR features are added to the original optical and SAR features to obtain a fused feature map.

4. The method for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention according to claim 1, characterized in that: The fused feature map is upsampled and skip-connected with the fused feature map, and finally decoded to obtain the ground object category to which each pixel belongs, including: The fused feature map is upsampled using bilinear interpolation, and the upsampling result and the extracted fused feature map are jump-connected using a jump connection. The feature maps obtained by upsampling and skip connection are respectively subjected to two convolution operations and ReLU activation function operations to generate pixel-level classification results.

5. The method for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention according to claim 1, characterized in that: Design loss functions for model training, including: Stochastic gradient descent is used for training, and model parameters are optimized by minimizing the cross entropy loss between the predicted probability map and the true value map.

6. A collaborative object classification system for optical and SAR remote sensing images based on cross-modal attention, characterized in that: include: Acquisition module, used to acquire optical and SAR remote sensing images; The classification module is used to input the acquired optical and SAR remote sensing images into the trained collaborative object classification model and output the object category; wherein the training of the collaborative object classification model includes: Construct training datasets of optical and SAR remote sensing images, and extract feature map pairs of optical and SAR remote sensing images in different dimensions; Based on the extracted feature map, the texture information in the optical image and the structural information in the SAR image are cross-modally fused in different dimensions to obtain a fused feature map. The fused feature map is upsampled and skip-connected with the fused feature map, and finally decoded to obtain the ground object category to which each pixel belongs; Design a loss function for model training and output the trained collaborative feature classification model.

7. A device for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention, characterized in that: It includes a memory and a processor, the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the steps of the method for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, which enable the computer to execute the steps of the method for collaborative object classification of optical and SAR remote sensing images based on cross-modal attention as described in any one of claims 1 to 5.

Images