Multi-source remote sensing data fusion classification method, device, equipment and storage medium

Through the use of dynamic convolution feature extraction and attention fusion modules, the problem of low classification accuracy in multi-source remote sensing data fusion is solved, and powerful representation ability and efficient classification effect are achieved.

CN118691909BActive Publication Date: 2024-11-26PENG CHENG LAB
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411170804.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-26
Publication Date
2024-11-26
Estimated Expiration
2044-08-26

AI Technical Summary

Technical Problem

The prior art is difficult to effectively integrate multi-source remote sensing data, resulting in low classification accuracy of remote sensing data, traditional convolutional operation CNN models are difficult to model information with relatively long spatial distances, and the Transformer model is inefficient in information mining when processing complex data.

Method used

Using dynamic convolution feature extraction method, the interactive features of different data sources are obtained through the data dynamic interaction module, and the attention fusion module is used to fusion, generating fusion semantic features with strong representation ability, and finally classifying and performing results based on these features. Fusion to improve the classification accuracy of multi-source remote sensing data.

Benefits of technology

Through dynamic convolution feature extraction and feature fusion, the advantages of each data source are leveraged, and the classification accuracy and feature representation capabilities of multi-source remote sensing data are significantly improved, and the inefficiency and characteristics of information mining in traditional methods when processing complex remote sensing data is solved. Problems of insufficient characterization.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118691909B_ABST
    Figure CN118691909B_ABST
Patent Text Reader

Abstract

The embodiments of the present application provide a multi-source remote sensing data fusion classification method, device, equipment and storage medium, and relate to the field of image data processing technology. The method performs dynamic convolution feature extraction on the first image block of the first remote sensing image and the second image block of the second remote sensing image to obtain a first interactive feature and a second interactive feature. The first interactive feature and the second interactive feature are feature fused to obtain a first semantic feature, a second semantic feature and a fused semantic feature, and the first semantic feature, the second semantic feature and the fused semantic feature are input into a classifier for classification prediction, and a fused classification result is obtained based on the three classification results obtained. The interactive features of remote sensing data from different data sources are obtained by a dynamic convolution feature extraction method. In the fusion stage, the advantages of each data source are brought into play based on the interactive features to obtain a fused semantic feature with powerful representation capabilities. The classification results are fused to further improve the classification accuracy of multi-source remote sensing data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of image data processing, and in particular to a multi-source remote sensing data fusion classification method, device, equipment and storage medium. Background Art

[0002] With the rapid development of earth observation technology, different types of sensors have been developed to obtain multi-source information of ground objects. For example, multispectral and hyperspectral cameras can obtain the spectral properties of ground objects, LiDAR (light detection and Ranging) sensors can directly obtain three-dimensional spatial information of ground objects, and Synthetic Aperture Radar (SAR) sensors can obtain amplitude and phase information. However, remote sensing data obtained by relying solely on a single sensor is incomplete.

[0003] In related technologies, deep learning is combined to fuse multi-source remote sensing data, such as based on the Convolutional Neural Networks (CNN) model. However, due to the high complexity of the spatial structure of remote sensing data itself and the strong heterogeneity between multi-source data, the CNN model using traditional convolution operations is difficult to effectively model information with a long spatial distance, which leads to insufficient feature representation capabilities of the fused features, resulting in low classification accuracy of remote sensing data based on fused features. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a multi-source remote sensing data fusion classification method, device, equipment and storage medium, to improve the fusion feature representation capability of multi-source remote sensing data and to improve the accuracy of remote sensing data classification.

[0005] To achieve the above purpose, a first aspect of an embodiment of the present application proposes a multi-source remote sensing data fusion classification method, comprising:

[0006] Acquire a first image block of a first remote sensing image and a corresponding second image block of a second remote sensing image, wherein the first remote sensing image and the second remote sensing image have different data sources;

[0007] Inputting the first image block and the second image block into a data dynamic interaction module to perform dynamic convolution feature extraction to obtain a first interaction feature and a second interaction feature;

[0008] Performing feature fusion on the first interaction feature and the second interaction feature to obtain a first semantic feature corresponding to the first interaction feature, a second semantic feature corresponding to the second interaction feature, and a fused semantic feature;

[0009] The first semantic feature, the second semantic feature and the fused semantic feature are respectively input into corresponding classifiers for classification prediction to obtain a first classification result, a second classification result and a third classification result, and the first classification result, the second classification result and the third classification result are fused to obtain a fused classification result of the first remote sensing image and the second remote sensing image.

[0010] In some embodiments, at least one of the data dynamic interaction modules is cascaded, and the first image block and the second image block are input into the data dynamic interaction module for dynamic convolution feature extraction to obtain the first interaction feature and the second interaction feature, including:

[0011] Performing dynamic convolution feature extraction on the input data using the data dynamic interaction module to obtain output data, wherein the output data of the previous data dynamic interaction module is used as the input data of the next data dynamic interaction module, and the input data of the first data dynamic interaction module is the first image block and the second image block;

[0012] The output data of the last one of the data dynamic interaction modules is obtained as the first interaction feature and the second interaction feature.

[0013] In some embodiments, the input data includes first input data and second input data, and the output data includes first output data and second output data; and the step of performing dynamic convolution feature extraction on the input data using the data dynamic interaction module to obtain the output data includes:

[0014] Using a first convolutional layer to perform feature extraction on the first input data to obtain a first dimensionality reduction feature, and using a second convolutional layer to perform feature extraction on the second input data to obtain a second dimensionality reduction feature;

[0015] Using a cascaded first pooling layer and a third convolutional layer to extract features from the first input data to obtain a first filter feature, and using a cascaded second pooling layer and a fourth convolutional layer to extract features from the second input data to obtain a second filter feature;

[0016] Performing depth-by-depth convolution on the first dimension reduction feature and the second filtering feature to obtain a first depth convolution feature, and inputting the first depth convolution feature into a fifth convolution layer to obtain the first output data;

[0017] The second dimensionality reduction feature and the first filtering feature are depth-wise convolved to obtain a second depth convolution feature, and the second depth convolution feature is input into the sixth convolution layer to obtain the second output data. The first output data of the last data dynamic interaction module is the first interaction feature, and the second output data is the second interaction feature.

[0018] In some embodiments, the fusing the first interaction feature and the second interaction feature to obtain a first semantic feature corresponding to the first interaction feature, a second semantic feature corresponding to the second interaction feature, and a fused semantic feature includes:

[0019] Using a first fusion module to extract semantic features from the first interaction features to obtain the first semantic features;

[0020] Using an attention fusion module to perform feature fusion on the first interaction feature and the second interaction feature to obtain an attention feature;

[0021] Using the second fusion module to extract semantic features from the attention features to obtain the fused semantic features;

[0022] The third fusion module is used to perform semantic feature extraction on the second interaction feature to obtain the second semantic feature.

[0023] In some embodiments, the using an attention fusion module to perform feature fusion on the first interaction feature and the second interaction feature to obtain an attention feature includes:

[0024] Accumulating the first interaction feature and the second interaction feature to obtain an accumulated feature, and inputting the accumulated feature into the attention fusion module for feature fusion to obtain a fused feature;

[0025] Obtaining a product of the first interaction feature and the fusion feature to obtain a first intermediate feature, and obtaining a product of the second interaction feature and an associated feature of the fusion feature to obtain a second intermediate feature, wherein a sum of the associated feature and the fusion feature is one;

[0026] The attention feature is obtained according to the first intermediate feature and the second intermediate feature.

[0027] In some embodiments, the step of inputting the accumulated features into the attention fusion module for feature fusion to obtain fused features includes:

[0028] Performing a pooling operation, a first point-by-point convolution operation, a first activation operation, a second point-by-point convolution operation, and a second activation operation on the accumulated features in sequence to obtain a first fused feature;

[0029] Performing a third point-by-point convolution operation, a third activation operation, a fourth point-by-point convolution operation, and a fourth activation operation on the accumulated features in sequence to obtain a second fused feature;

[0030] Accumulating the first fusion feature and the second fusion feature to obtain an accumulated fusion feature;

[0031] After performing a fifth activation operation on the accumulated fusion feature, a fused activation feature is obtained, and the product between the fused activation feature and the accumulated feature is calculated to obtain the fused feature.

[0032] In some embodiments, the first fusion module is obtained by cascading a first pooling layer, a first vector stretching layer, and a first fully connected layer, and the first fusion module is used to extract semantic features from the first interaction features to obtain the first semantic features, including:

[0033] Using the first pooling layer to perform a pooling operation on the first interaction feature to obtain a first pooling feature;

[0034] Using the first vector stretching layer to perform a vector stretching operation on the first pooled feature to obtain a first vector stretching feature;

[0035] The first fully connected layer is used to perform a fully connected operation on the first vector stretching feature to obtain the first semantic feature.

[0036] In some embodiments, acquiring a first image block of a first remote sensing image and a corresponding second image block of a second remote sensing image includes:

[0037] Selecting a first initial image block of a preset size in the first remote sensing image based on the selected marked pixels, and selecting a second initial image block of the preset size in the second remote sensing image based on the marked pixels;

[0038] The first initial image block is scaled using a first image convolution layer to obtain the first image block, and the second initial image block is scaled using a second image convolution layer to obtain the second image block.

[0039] In some embodiments, fusing the first classification result, the second classification result, and the third classification result to obtain a fused classification result of the first remote sensing image and the second remote sensing image includes:

[0040] Obtaining a weighted result of the first classification result, the second classification result, and the third classification result as a pixel classification result of the first image block and the second image block;

[0041] The pixel classification results corresponding to all image blocks of the first remote sensing image and the second remote sensing image are obtained, and the fusion classification result is obtained according to all the pixel classification results.

[0042] To achieve the above-mentioned purpose, the second aspect of the embodiment of the present application proposes a multi-source remote sensing data fusion classification device, comprising:

[0043] A data acquisition module: used for acquiring a first image block of a first remote sensing image and a corresponding second image block of a second remote sensing image, wherein the data sources of the first remote sensing image and the second remote sensing image are different;

[0044] An interactive feature extraction module: used for inputting the first image block and the second image block into a data dynamic interaction module for dynamic convolution feature extraction to obtain a first interactive feature and a second interactive feature;

[0045] A semantic feature fusion module: used for fusing the first interaction feature and the second interaction feature to obtain a first semantic feature corresponding to the first interaction feature, a second semantic feature corresponding to the second interaction feature, and a fused semantic feature;

[0046] Classification output module: used to input the first semantic feature, the second semantic feature and the fused semantic feature into corresponding classifiers for classification prediction, obtain a first classification result, a second classification result and a third classification result, fuse the first classification result, the second classification result and the third classification result, and obtain a fused classification result of the first remote sensing image and the second remote sensing image.

[0047] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0048] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0049] The multi-source remote sensing data fusion classification method, device, equipment and storage medium proposed in the embodiment of the present application obtains the first image block of the first remote sensing image and the second image block of the corresponding second remote sensing image, wherein the data sources of the first remote sensing image and the second remote sensing image are different. Then the first image block and the second image block are input into the data dynamic interaction module for dynamic convolution feature extraction to obtain the first interaction feature and the second interaction feature. Then the first interaction feature and the second interaction feature are feature fused to obtain the first semantic feature corresponding to the first interaction feature, the second semantic feature corresponding to the second interaction feature and the fusion semantic feature, and finally the first semantic feature, the second semantic feature and the fusion semantic feature are respectively input into the corresponding classifier for classification prediction to obtain the first classification result, the second classification result and the third classification result, and the fusion classification result of the first remote sensing image and the second remote sensing image is obtained according to the first classification result, the second classification result and the third classification result. In the embodiment of the present application, the dynamic convolution feature extraction method can be used to obtain the interaction features between remote sensing data from different data sources. In the fusion stage, based on these interaction features, the advantages of each data source can be brought into play, thereby obtaining a fusion semantic feature with powerful representation ability. Finally, classification is performed based on these fused semantic features, and the classification results are fused to further improve the classification accuracy of multi-source remote sensing data. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a flow chart of the multi-source remote sensing data fusion classification method provided in an embodiment of the present application.

[0051] Figure 2 It is a data processing diagram of the first image block and the second image block provided in an embodiment of the present application.

[0052] Figure 3 It is a connection diagram of the data dynamic interaction module provided in an embodiment of the present application.

[0053] Figure 4 This is a flowchart of using a data dynamic interaction module to perform dynamic convolution feature extraction on input data to obtain output data, as provided in an embodiment of the present application.

[0054] Figure 5 It is a flow chart of obtaining the first semantic feature, the second semantic feature and the fused semantic feature provided by an embodiment of the present application.

[0055] Figure 6 It is a schematic diagram of the network structure for feature fusion in the embodiment of the present application.

[0056] Figure 7 It is a schematic diagram of the network structure of the attention fusion module outputting attention features provided in an embodiment of the present application.

[0057] Figure 8It is a structural diagram of the attention fusion module MS-CAM provided in an embodiment of the present application.

[0058] Fig. 9 It is a network diagram for performing classification and fusion according to the first semantic feature, the second semantic feature and the fused semantic feature provided in an embodiment of the present application.

[0059] Fig.10 A schematic diagram of a multi-source remote sensing data fusion classification model provided in an embodiment of the present application.

[0060] Fig.11 It is a schematic diagram of the fusion effect of the multi-source remote sensing data fusion classification method provided in an embodiment of the present application.

[0061] Fig.12 This is a structural block diagram of a multi-source remote sensing data fusion classification device provided by another embodiment of the present application.

[0062] Fig.13 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0064] It should be noted that although the functional modules are divided in the device schematic and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0066] First, some nouns involved in this application are analyzed:

[0067] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0068] With the rapid development of earth observation technology, different types of sensors have been developed to obtain multi-source information of ground objects. For example, multispectral and hyperspectral cameras can obtain the spectral properties of ground objects, LiDAR (light detection and Ranging) sensors can directly obtain three-dimensional spatial information of ground objects, and Synthetic Aperture Radar (SAR) sensors can obtain amplitude and phase information. For example, hyperspectral images have rich spectral information and can identify different material properties, but it is difficult to distinguish between ground objects with similar spectral information but different elevation information (for example, relying solely on spectral information cannot effectively distinguish between grass and trees, parking lots and building roofs, roads and viaducts, etc.); on the other hand, LiDAR data can use height information to directly classify ground objects at different elevations, but it cannot distinguish between ground objects with the same height but different spectra (for example, using only LiDAR point clouds cannot effectively distinguish between asphalt and concrete, iron tiles and glazed tiles, trees and pseudo-tree signal towers, etc.).

[0069] It can be seen that the remote sensing data obtained by relying solely on a single sensor is incomplete and difficult to meet the needs of reliable remote sensing object classification. Therefore, it is necessary to integrate remote sensing information from different data sources to make full use of different types of data in order to achieve fine classification of remote sensing images.

[0070] In related technologies, the fusion of multi-source remote sensing data is usually combined with deep learning technology, or based on traditional methods such as morphological operations and manifold learning. Among them, morphological operations and manifold learning are classic traditional multi-source data fusion methods. Compared with remote sensing data classification based on deep learning, the feature extraction capabilities of these methods are relatively limited. Among deep learning methods, common applications are based on CNN (convolutional neural network) models and Transformer models. However, due to the high complexity of the spatial structure of remote sensing data itself and the strong heterogeneity between multi-source data, the CNN model using traditional convolution operations is difficult to effectively model information with a long spatial distance, resulting in insufficient feature representation capabilities obtained by fusion. In addition, when the Transformer model processes data with high complexity, the information mining efficiency in the fusion process is low. These problems thus affect the classification accuracy and efficiency of remote sensing data based on fusion features.

[0071] Based on this, the embodiments of the present application provide a multi-source remote sensing data fusion classification method, device, equipment and storage medium, which can obtain the interactive features between remote sensing data from different data sources through a dynamic convolution feature extraction method. In the fusion stage, based on these interactive features, the advantages of each data source can be brought into play, thereby obtaining fused semantic features with powerful representation capabilities. Finally, classification is performed based on these fused semantic features, and the classification results are fused, which can further improve the classification accuracy of multi-source remote sensing data.

[0072] The embodiments of the present application provide a multi-source remote sensing data fusion classification method, device, equipment and storage medium, which are specifically illustrated by the following embodiments. First, the multi-source remote sensing data fusion classification method in the embodiments of the present application is described.

[0073] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0074] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0075] The multi-source remote sensing data fusion classification method provided in the embodiment of the present application relates to the field of image data processing technology. The multi-source remote sensing data fusion classification method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be a computer program running in a terminal or a server side. For example, a computer program can be a native program or software module in an operating system; it can be a native (Native) application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a client that supports multi-source remote sensing data fusion classification, or it can be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form. Among them, the terminal communicates with the server through a network. The multi-source remote sensing data fusion classification method can be executed by a terminal or a server, or by a terminal and a server in collaboration.

[0076] In some embodiments, the terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer, or a smart watch. In addition, the terminal may also be an intelligent vehicle-mounted device. The intelligent vehicle-mounted device applies the multi-source remote sensing data fusion classification method of this embodiment to provide related services and enhance the driving experience. The server may be an independent server, or it may be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms; it may also be a service node in a blockchain system, and each service node in the blockchain system forms a peer-to-peer (P2P, Peer To Peer, P2P) network. The P2P protocol is an application layer protocol running on the Transmission Control Protocol (TCP) protocol. The terminal and the server may be connected via Bluetooth, Universal Serial Bus (USB) or network communication connection methods, which are not limited in this embodiment.

[0077] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0078] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0079] The following describes a multi-source remote sensing data fusion classification method in an embodiment of the present application.

[0080] Figure 1 is an optional flow chart of the multi-source remote sensing data fusion classification method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps 110 to 140. It can also be understood that this embodiment Figure 1 The order of step 110 to step 140 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0081] Step 110: Acquire a first image block of a first remote sensing image and a corresponding second image block of a second remote sensing image.

[0082] In one embodiment, the data sources of the first remote sensing image and the second remote sensing image are different. The process of obtaining the first image block of the first remote sensing image and the corresponding second image block of the second remote sensing image specifically includes: selecting a first initial image block of a preset size in the first remote sensing image based on the selected marked pixels, and selecting a second initial image block of a preset size in the second remote sensing image based on the marked pixels. The first initial image block is scaled using the first image convolution layer to obtain the first image block, and the second initial image block is scaled using the second image convolution layer to obtain the second image block.

[0083] In one embodiment, the first remote sensing image is recorded as remote sensing image A, and the second remote sensing image is recorded as remote sensing image B. At this time, corresponding marked pixels are obtained from the same positions of the first remote sensing image and the second remote sensing image. The marked pixels can be understood as a sequence of pixels, which is used to define the position of the pixels on the remote sensing image. Among them, the marked pixels of remote sensing image A can be recorded as , expressed as , the labeled pixels of the remote sensing image B are recorded as , expressed as , where K represents the number of labeled pixels, Represents the number of spectral bands of remote sensing image A, Indicates the number of spectral bands of remote sensing image B, represents the i-th labeled pixel corresponding to the remote sensing image A, Represents the corresponding i-th marked pixel on the remote sensing image B.

[0084] In one embodiment, after selecting the marked pixel, the marked pixel and As the center, image blocks of preset sizes are intercepted from remote sensing image A and remote sensing image B respectively. The preset sizes are determined according to the actual requirements of classification accuracy. Then, image pair samples are constructed. ,in, is the first initial image block corresponding to the remote sensing image A, is the second initial image block corresponding to the remote sensing image B, The default size.

[0085] Next, considering that the data dimensions corresponding to remote sensing images from different data sources may be different, it is necessary to align the dimensions of the first initial image block and the second initial image block. Specifically, two different convolutional layers are used to align the dimensions of the first initial image block. and the second initial image block A convolution operation is performed to achieve scale adjustment so that the data dimensions of the two are equal, so that the same network structure can be used to extract features from different data sources simultaneously.

[0086] In one embodiment, referring to Figure 2 , Figure 2 It is a data processing diagram of the first image block and the second image block provided in an embodiment of the present application. Figure 2 The network structure becomes the data preprocessing module, where the data dimension of the remote sensing image A is , the data dimension of remote sensing image B is , the first initial image block The data dimension is , the second initial image block The data dimension is , the first image convolution layer and the second image convolution layer The convolution kernel sizes are and At this time, the first image convolution layer is used to resize the first initial image block, and the first image block obtained is The data dimension is , and using the second image convolution layer to resize the second initial image block to obtain a second image block The data dimension is , Indicates the dimension after adjustment. The above dimension-related values ​​can be set according to actual conditions.

[0087] The above size adjustment process is expressed as:

[0088] ;

[0089] ;

[0090] After preprocessing, the next step is to extract dynamic interactive features from multi-source data.

[0091] Step 120: Input the first image block and the second image block into a data dynamic interaction module to perform dynamic convolution feature extraction to obtain a first interaction feature and a second interaction feature.

[0092] In one embodiment, dynamic interaction feature extraction of multi-source data is performed after cascading at least one data dynamic interaction module. Specifically, the process of inputting the first image block and the second image block into the data dynamic interaction module for dynamic convolution feature extraction to obtain the first interaction feature and the second interaction feature includes: using the data dynamic interaction module to perform dynamic convolution feature extraction on the input data to obtain output data. Due to the cascade structure, the output data of the previous data dynamic interaction module serves as the input data of the next data dynamic interaction module. The input data of the first data dynamic interaction module is the first image block and the second image block. Next, the output data of the last data dynamic interaction module is obtained as the first interaction feature and the second interaction feature.

[0093] In one embodiment, referring to Figure 3 , Figure 3 It is a connection diagram of the data dynamic interaction module provided in an embodiment of the present application. Figure 3 In the figure, three identical data dynamic interaction modules are cascaded to each other for illustration, which does not limit the number of data dynamic interaction modules. Figure 3 ,The structure of each data dynamic interaction module is the same and contains two ,branches of the same structure, and each branch is used to process the ,remote sensing data of one data source.,It can be understood that although the two branches have the same ,structure, they have different network parameters.

[0094] Specifically, each branch includes a feature generator and a filter generator, wherein the feature generator is composed of a convolution layer, and the filter generator is composed of a cascaded pooling layer and a convolution layer. The feature generator is used to generate deep features of the corresponding data, and the filter generator is used to generate convolution filters for the corresponding data. Different from the convolution operation of the traditional CNN model, the embodiment of the present application adopts a dynamic convolution operation, that is, the filter parameters in the filter generator are completely obtained from the input data of the data dynamic interaction module, so it can achieve adaptation to the input data, avoiding the defect that the filter parameters of the convolution network in the related technology cannot be changed after the training is completed, thereby improving the ability of feature characterization.

[0095] In one embodiment, in combination Figure 3 From the structure, it can be seen that for each data dynamic interaction module, its input data includes first input data and second input data, which correspond to two branches respectively, and the output data includes first output data and second output data, among which, except the first one, the other two data dynamic interaction modules are simplified schematically.

[0096] In one embodiment, the first data dynamic interaction module is taken as an example for explanation. The first input data of the first data dynamic interaction module is the first image block. , the second input data is the second image block . Reference Figure 4 , Figure 4 The present invention provides a flowchart of using a data dynamic interaction module to extract dynamic convolution features from input data to obtain output data, which specifically includes the following steps:

[0097] Step 410: Use the first convolutional layer to perform feature extraction on the first input data to obtain a first dimensionality reduction feature, and use the second convolutional layer to perform feature extraction on the second input data to obtain a second dimensionality reduction feature.

[0098] In one embodiment, in combination Figure 3 , the feature generator in the first branch of the data dynamic interaction module is the first convolutional layer, and its convolution kernel is , the feature generator in the second branch is the second convolutional layer, and its convolution kernel is also Therefore, the first dimension reduction feature and the second dimension reduction feature It is expressed as:

[0099] ;

[0100] ;

[0101] in, express The convolution operation, represents the first dimension reduction feature, represents the second dimension reduction feature, Represents the dimension after the convolution operation, and the dimension Less than indicates dimension .

[0102] Step 420: Use the cascaded first pooling layer and the third convolution layer to extract features from the first input data to obtain a first filter feature, and use the cascaded second pooling layer and the fourth convolution layer to extract features from the second input data to obtain a second filter feature.

[0103] In one embodiment, in combination Figure 3 In the first branch of the data dynamic interaction module, the filter generator is composed of the first pooling layer and the third convolution layer in cascade, and in the second branch, the filter generator is composed of the second pooling layer and the fourth convolution layer in cascade, and the convolution kernel size of the third and fourth convolution layers is also Therefore, the first filter feature and the second filter feature are expressed as:

[0104] ;

[0105] ;

[0106] in, express The convolution operation, represents the pooling operation, represents the first filtering feature, represents the second filtering feature, Represents the dimension of the data after pooling, the first filter feature and the second filter characteristic It can be considered as a filter with texture information.

[0107] Step 430: Perform depth-wise convolution on the first dimensionality reduction feature and the second filtering feature to obtain a first depth-wise convolution feature, and input the first depth-wise convolution feature into the fifth convolution layer to obtain first output data.

[0108] Step 440: Perform depth-wise convolution on the second dimensionality reduction feature and the first filtering feature to obtain a second depth-wise convolution feature, and input the second depth-wise convolution feature into the sixth convolution layer to obtain second output data.

[0109] Since the core of the data dynamic interaction module in the embodiment of the present application is to use the texture filter generated by a certain data to guide the feature extraction of another type of data (i.e., dynamic convolution operation), considering that there are rich complementary related information between remote sensing data from different sources in the same scene, the cascaded data dynamic interaction module is used to effectively mine this complementary related information between data and extract deep features with strong representation capabilities. Therefore, it is also necessary to perform complementary operations on the first filter features and the second filter features obtained by the two branches.

[0110] Specifically, the first dimension reduction feature and the second filter feature are convolved depth by depth to obtain the first depth convolution feature, and then the The fifth convolutional layer convolves the first depth convolutional feature to fuse the information of different channels and obtain the first output data. In the same operation process, the second dimension reduction feature and the first filter feature are convolved depth by depth to obtain the second depth convolutional feature, and then the second depth convolutional feature is used. The sixth convolutional layer convolves the second deep convolutional feature to fuse the information of different channels and obtain the second output data. This process is expressed as:

[0111] ;

[0112] ;

[0113] ;

[0114] ;

[0115] ;

[0116] in, represents the depth-wise convolution operation, represents the first depth convolution feature, represents the second depth convolution feature, represents the first output data, represents the second output data, Represents the Dynamic Interaction Module (DIM).

[0117] Among them, the depth-wise convolution operation is different from the traditional convolution operation. The depth-wise convolution operation performs a separate convolution operation on each input channel in order to reduce the amount of calculation and the size of the model while maintaining a high accuracy. In the process of depth-wise convolution, a single-channel convolution kernel is assigned to each input channel. Each convolution kernel only performs convolution operations on its corresponding input channel to generate an output feature map. If the input has D channels, then the depth-wise convolution will produce D output feature maps, each feature map comes from the convolution result of an input channel.

[0118] Combination Figure 3 , the embodiment of the present application can fully extract the deep features of the first data block and the second data block through the cascaded multiple data dynamic interaction modules, and the first output data of the last data dynamic interaction module is the first interaction feature, and the second output data is the second interaction feature, which is expressed as:

[0119]

[0120] in, represents the first interaction feature, represents the second interaction feature, Represents three cascaded data dynamic interaction modules.

[0121] Next, a fusion process of the first interaction feature and the second interaction feature is performed.

[0122] Step 130: Fusing the first interaction feature and the second interaction feature to obtain a first semantic feature corresponding to the first interaction feature, a second semantic feature corresponding to the second interaction feature, and a fused semantic feature.

[0123] In one embodiment, in order to fully explore the complementary information between multi-source data, the embodiment of the present application adopts an attention-based attention fusion module (AFFM) to perform feature-level fusion on an interactive feature and a second interactive feature. Figure 5 , Figure 5This is a flow chart of obtaining a first semantic feature, a second semantic feature, and a fused semantic feature provided by an embodiment of the present application, which specifically includes the following steps:

[0124] Step 510: Use the first fusion module to extract semantic features from the first interaction feature to obtain a first semantic feature.

[0125] Step 520: Use the attention fusion module to fuse the first interaction feature and the second interaction feature to obtain an attention feature.

[0126] Step 530: Use the second fusion module to extract semantic features from the attention features to obtain fused semantic features.

[0127] Step 540: Use the third fusion module to extract semantic features from the second interaction features to obtain second semantic features.

[0128] In one embodiment, referring to Figure 6 , Figure 6 It is a schematic diagram of the network structure for feature fusion in the embodiment of the present application. Figure 6 The first fusion module, the second fusion module, the third fusion module and the attention fusion module are shown in FIG, wherein the first fusion module, the second fusion module and the third fusion module are all obtained by cascading the pooling layer, the vector stretching layer and the fully connected layer. Specifically, the first fusion module is obtained by cascading the first pooling layer, the first vector stretching layer and the first fully connected layer, the second fusion module is obtained by cascading the second pooling layer, the second vector stretching layer and the second fully connected layer, and the third fusion module is obtained by cascading the third pooling layer, the third vector stretching layer and the third fully connected layer.

[0129] Taking the first fusion module as an example, the embodiment of the present application uses the first fusion module to extract semantic features from the first interaction features, and the process of obtaining the first semantic features specifically includes: using the first pooling layer to perform a pooling operation on the first interaction features to obtain the first pooling feature, using the first vector stretching layer to perform a vector stretching operation on the first pooling feature to obtain the first vector stretching feature, and using the first fully connected layer to perform a fully connected operation on the first vector stretching feature to obtain the first semantic feature.

[0130] Among them, the first pooling layer performs a global pooling operation (GlobalAvgPool) on the first interactive feature, averages the spatial dimensions of the first interactive feature, that is, the width and height, and outputs the average of all element values ​​of the first interactive feature to obtain the first pooling feature, with the purpose of reducing the number of model parameters and reducing the risk of overfitting while retaining important information. After global average pooling, the first interactive feature is converted into a set of numerical values, and then the first vector stretching layer is used to perform a vector stretching operation (Flatten) on the first pooling feature, and these numerical values ​​of the first pooling feature are "flattened" into a one-dimensional array or vector as the input of the fully connected layer. The first fully connected layer is then used to perform a fully connected operation (FullyConnection) on the first vector stretching feature, and it is linearly transformed through the weight matrix and the bias term to capture higher-level features or patterns in the input first vector stretching feature to obtain the first semantic feature.

[0131] Combination Figure 6 For the attention fusion module, its input is the first interaction feature and the second interaction feature, and its output is the attention feature obtained after feature fusion. However, unlike the first fusion module, whose input is the first interaction feature and the third fusion module, whose input is the second interaction feature, the input of the second fusion module is the attention feature output by the attention fusion module.

[0132] The following describes the process of the attention fusion module outputting attention features. Figure 7 , Figure 7 It is a schematic diagram of the network structure of the attention fusion module outputting attention features provided in an embodiment of the present application.

[0133] Reference Figure 7 , the process of using the attention fusion module to fuse the first interactive feature and the second interactive feature to obtain the attention feature can be described as: accumulating the first interactive feature and the second interactive feature to obtain the accumulated feature, and inputting the accumulated feature into the attention fusion module MS-CAM for feature fusion to obtain the fused feature. Next, the product of the first interactive feature and the fused feature is obtained to obtain the first intermediate feature, and the product of the second interactive feature and the associated feature of the fused feature is obtained to obtain the second intermediate feature. Then the attention feature is obtained based on the first intermediate feature and the second intermediate feature. The entire operation of this process is recorded as AFFM (Attentional Feature Fusion Module), which is expressed as:

[0134]

[0135] in, represents the cumulative feature, represents the fusion feature, represents the first intermediate feature, represents the associated feature, where the sum of the associated feature and the fusion feature is one. represents the second intermediate feature, According to the first interaction feature and the second interaction feature Computed attention features.

[0136] Next, the process of inputting the accumulated features into the attention fusion module MS-CAM for feature fusion to obtain the fused features is described.

[0137] In one embodiment, referring to Figure 8 , Figure 8 Schematic diagram of the structure of the attention fusion module MS-CAM provided in the embodiment of the present application. Figure 8 From the network structure, we can see that the attention fusion module extracts global and local features respectively, and finally uses element-by-element multiplication operations to fuse the final attention features. Specifically, it is divided into three branches. In the first branch, the accumulated features are subjected to global average pooling operations in turn, and then the first point-by-point convolution operation is performed, and then the first activation operation is performed. At this time, the first activation operation continuously uses batch normalization (Batch Normalization, BN) and nonlinear activation function Rectified Linear Unit (Rectified Linear Unit, ReLU) as activation functions, followed by the second point-by-point convolution operation and the second activation operation. Here, the second activation function operation uses batch normalization BN as the activation function, and finally the first fusion feature is obtained.

[0138] Next, in the second branch, the accumulated features are sequentially subjected to the third point-by-point convolution operation, the third activation operation, the fourth point-by-point convolution operation, and the fourth activation operation to obtain the second fused feature. Among them, the third activation operation also continuously uses batch normalization BN and nonlinear activation function corrected linear unit ReLU as activation functions, and the fourth activation operation uses batch normalization BN as activation function. Then, the first fused feature and the second fused feature obtained in the first two branches are accumulated to obtain the accumulated fused feature, and then the accumulated fused feature is subjected to the fifth activation operation to obtain the fused activated feature, wherein the fifth activation operation uses the Sigmoid activation function.

[0139] Finally, the product of the accumulated features of the third branch and the fused activation features obtained above is calculated to obtain the final fused features.

[0140] Combination Figure 6 , Figure 7For example, the first fusion module obtains the first semantic feature according to the first interaction feature of the input, the second fusion module obtains the fusion semantic feature according to the attention feature of the input, and the third fusion module obtains the second semantic feature according to the second interaction feature. It can be expressed as:

[0141] ;

[0142] ;

[0143] ;

[0144] in, represents the first semantic feature, represents the second semantic feature, Represents fusion semantic features.

[0145] It can be understood that the first semantic feature and the second semantic feature are both single-source data semantic features, and the fused semantic feature is a semantic feature generated after the data from the two data sources are fused.

[0146] Step 140: respectively input the first semantic feature, the second semantic feature and the fused semantic feature into corresponding classifiers for classification prediction to obtain a first classification result, a second classification result and a third classification result, and fuse the first classification result, the second classification result and the third classification result to obtain a fused classification result of the first remote sensing image and the second remote sensing image.

[0147] In one embodiment, for the i-th marked pixel, refer to Fig. 9 , Fig. 9 It is a network diagram for performing classification and fusion according to the first semantic feature, the second semantic feature and the fused semantic feature provided in an embodiment of the present application. Fig. 9 In , a classifier is set for the first semantic feature, the second semantic feature and the fused semantic feature respectively. The structures of these three classifiers can be the same or different. For example, the fully connected layer with softmax as the activation function can be used as the classifier. After the first semantic feature, the second semantic feature and the fused semantic feature are respectively input into the corresponding classifiers for classification prediction, the first classification result corresponding to the first semantic feature, the third classification result corresponding to the fused semantic feature and the second classification result corresponding to the second semantic feature can be obtained. Then, the decision-level fusion strategy is used to further optimize these three classification results, and the weighted results of the first classification result, the second classification result and the third classification result are obtained as the pixel classification result between the first image block of the first remote sensing image and the second image block of the second remote sensing image. It is expressed as:

[0148]

[0149] in, represents the decision weight for the first classification result, represents the decision weight for the second classification result, represents the decision weight for the third classification result, Represents the pixel classification result after weighted operation.

[0150] In one embodiment, since there are multiple labeled pixels in the first remote sensing image and the second remote sensing image, it is necessary to obtain the final fused classification result based on the pixel classification results of all labeled pixels, specifically: obtain the pixel classification results corresponding to all image blocks of the first remote sensing image and the second remote sensing image, and obtain the fused classification result based on all the pixel classification results. In this process, the pixel classification results can be summarized in a clustering manner to obtain the fused classification result.

[0151] It can be understood that the various network structures of the multi-source remote sensing data fusion classification method of the embodiment of the present application can be aggregated to obtain an overall multi-source remote sensing data fusion classification model. The multi-source remote sensing data fusion classification model is trained in advance using a training data set to obtain reasonable parameters of the model before the model can be applied.

[0152] Reference Fig.10 , Fig.10 A schematic diagram of a multi-source remote sensing data fusion classification model provided in an embodiment of the present application. Fig.10 It includes data preprocessing module, multi-source data dynamic interaction module and feature fusion module. The structure of data preprocessing module includes Figure 2 The network structure shown is mainly used to perform the process of dimensional alignment of the first initial image block and the second initial image block to obtain the first image block and the second image block. The structure of the multi-source data dynamic interaction module includes the above Figure 3 The network structure is mainly used to execute the process of obtaining the first interactive feature and the second interactive feature according to the first image block and the second image block as described above. The structure of the feature fusion module includes the above Figure 6 and Fig. 9 The network structure is mainly used to obtain the first semantic feature, the second semantic feature and the fused semantic feature according to the first interaction feature and the second interaction feature, and then obtain the pixel classification result according to the first semantic feature, the second semantic feature and the fused semantic feature. It can be understood that the multi-source data dynamic interaction model in the embodiment of the present application can also be replaced by a deeper network model.

[0153] For the training process of the multi-source remote sensing data fusion classification model, we can obtain sample pairs corresponding to multiple sets of labeled pixels as training data sets based on sample data, and then divide the training data sets into training sample pair sets and test sample pair sets according to a preset ratio, and obtain the corresponding label data. The label data set is expressed as ,in represents the total number of categories to be classified, and represents the number of labeled pixels. and Represent the training sample pair set and the test sample pair set respectively, and Represent the true label sets of the training sample pair set and the test sample pair set, respectively, where .

[0154] Then, the training sample set is used to train the multi-source remote sensing data fusion classification model. 3) The cross entropy loss function is used to measure the loss between the predicted label and the true label. The final loss function is defined as:

[0155]

[0156] in, represents the first semantic feature of the prediction, represents the second semantic feature of the prediction, represents the predicted fused semantic features, represents the label value, and L represents the value of the loss function.

[0157] Afterwards, the multi-source remote sensing data fusion classification model constructed by supervised training using the stochastic gradient descent algorithm is continuously updated and iterated to obtain appropriate network parameters. Finally, the set is tested with test samples to obtain the trained multi-source remote sensing data fusion classification model.

[0158] In one embodiment, in order to verify the effectiveness of the multi-source remote sensing data fusion classification method provided in the embodiment of the present application, a multi-source data fusion classification experiment was conducted on the Trento dataset. The dataset records a rural area of ​​a certain country. The size of the dataset is 166×600 pixels, the spatial resolution is 1 meter, and it contains remote sensing data from two sources, one of which is a hyperspectral image and the other is a lidar data. Hyperspectral images (HSI) are collected by the Aerial Hyperspectral Imaging System (AISA) Eagle sensor, which has 63 spectral bands with a wavelength range of 0.42 to 0.99 microns. LiDAR (light detection and ranging, LiDAR) data is acquired by the Optech Aerial Laser Topography (ALTM) 3100EA sensor. There are six different types of ground categories in the corresponding ground scene.

[0159] The training samples and test samples used in the experiment are based on this data and the published standard data set. The preset sizes of the first image block and the second image block are set to , the batch size of training is set to 64, the number of iterations is 100, and the initial learning rate is set to 0.001. In order to further compare the classification effect, in addition to the multi-source remote sensing data fusion classification method of the embodiment of the present application, other multi-source data classification methods based on deep learning are further compared, including: EndNet model, TBCNN model, FusetNet model and CCR-Net model.

[0160] Reference Fig.11 , Fig.11 : is a schematic diagram of the fusion effect of the multi-source remote sensing data fusion classification method provided in the embodiment of the present application. Fig.11 The pseudo-color image of the hyperspectral HSI, the depth image of the LiDAR, the real object map of the data set and the classification maps obtained by different classification methods are respectively displayed. By comparison, it can be found that the multi-source remote sensing data fusion classification method proposed in the embodiment of the present application can classify different objects well, and the obtained classification map has a high similarity with the real object map. In addition, by comparing the overall classification accuracy (Overall Accuracy, OA), the multi-source remote sensing data fusion classification method of the embodiment of the present application obtained the highest classification result. The above experimental results prove the effectiveness and superiority of the multi-source remote sensing data fusion classification method of the embodiment of the present application for multi-source remote sensing data fusion classification.

[0161] The technical solution provided in the embodiment of the present application is obtained by obtaining the first image block of the first remote sensing image and the second image block of the corresponding second remote sensing image, wherein the data sources of the first remote sensing image and the second remote sensing image are different. Then the first image block and the second image block are input into the data dynamic interaction module for dynamic convolution feature extraction to obtain the first interaction feature and the second interaction feature. Then the first interaction feature and the second interaction feature are feature fused to obtain the first semantic feature corresponding to the first interaction feature, the second semantic feature corresponding to the second interaction feature and the fused semantic feature, and finally the first semantic feature, the second semantic feature and the fused semantic feature are respectively input into the corresponding classifier for classification prediction to obtain the first classification result, the second classification result and the third classification result, and the fusion classification result of the first remote sensing image and the second remote sensing image is obtained according to the first classification result, the second classification result and the third classification result. In the embodiment of the present application, the dynamic convolution feature extraction method can be used to obtain the interaction features between remote sensing data from different data sources. In the fusion stage, the interaction between data is focused on, and the advantages of each data source are brought into play based on the interaction features to achieve complementary advantages, thereby obtaining fused semantic features with powerful representation capabilities. Finally, classification is performed based on these fused semantic features, and the classification results are fused. The classification accuracy of multi-source remote sensing data is further improved through feature-level fusion and decision-level fusion.

[0162] The present application also provides a multi-source remote sensing data fusion classification device, which can implement the multi-source remote sensing data fusion classification method. Fig.12 , the device comprises:

[0163] Data acquisition module 1210: used to acquire a first image block of a first remote sensing image and a corresponding second image block of a second remote sensing image, where the data sources of the first remote sensing image and the second remote sensing image are different.

[0164] The interactive feature extraction module 1220 is used to input the first image block and the second image block into the data dynamic interaction module for dynamic convolution feature extraction to obtain the first interactive feature and the second interactive feature.

[0165] Semantic feature fusion module 1230: used to fuse the first interaction feature and the second interaction feature to obtain a first semantic feature corresponding to the first interaction feature, a second semantic feature corresponding to the second interaction feature, and a fused semantic feature.

[0166] Classification output module 1240: used to input the first semantic feature, the second semantic feature and the fused semantic feature into the corresponding classifiers for classification prediction, obtain the first classification result, the second classification result and the third classification result, fuse the first classification result, the second classification result and the third classification result, and obtain the fused classification result of the first remote sensing image and the second remote sensing image.

[0167] The specific implementation of the multi-source remote sensing data fusion classification device of this embodiment is basically the same as the specific implementation of the multi-source remote sensing data fusion classification method described above, and will not be repeated here.

[0168] The present application also provides an electronic device, including:

[0169] at least one memory; at least one processor; at least one program;

[0170] The program is stored in the memory, and the processor executes the at least one program to implement the multi-source remote sensing data fusion classification method implemented in this application. The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0171] See also Fig.13 , Fig.13 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0172] The processor 1301 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0173] The memory 1302 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1302 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1302, and the processor 1301 calls and executes the multi-source remote sensing data fusion classification method of the embodiment of this application;

[0174] Input / output interface 1303, used to implement information input and output;

[0175] Communication interface 1304, used to realize communication interaction between the device and other devices, which can be realized through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.); and

[0176] A bus 1305 that transmits information between various components of the device (e.g., the processor 1301, the memory 1302, the input / output interface 1303, and the communication interface 1304);

[0177] The processor 1301 , the memory 1302 , the input / output interface 1303 and the communication interface 1304 are connected to each other in communication within the device via a bus 1305 .

[0178] An embodiment of the present application also provides a storage medium, which is a storage medium that stores a computer program. When the computer program is executed by a processor, the multi-source remote sensing data fusion classification method is implemented.

[0179] As a non-transitory storage medium, the memory can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0180] The multi-source remote sensing data fusion classification method, device, equipment and storage medium proposed in the embodiment of the present application obtains the first image block of the first remote sensing image and the corresponding second image block of the second remote sensing image, wherein the data sources of the first remote sensing image and the second remote sensing image are different. Then the first image block and the second image block are input into the data dynamic interaction module for dynamic convolution feature extraction to obtain the first interaction feature and the second interaction feature. Then the first interaction feature and the second interaction feature are feature fused to obtain the first semantic feature corresponding to the first interaction feature, the second semantic feature corresponding to the second interaction feature and the fusion semantic feature, and finally the first semantic feature, the second semantic feature and the fusion semantic feature are respectively input into the corresponding classifier for classification prediction to obtain the first classification result, the second classification result and the third classification result, and the fusion classification result of the first remote sensing image and the second remote sensing image is obtained according to the first classification result, the second classification result and the third classification result. In the embodiment of the present application, the dynamic convolution feature extraction method can obtain the interaction features between remote sensing data from different data sources. In the fusion stage, the interaction between data is focused on, and the advantages of each data source are brought into play based on the interaction features to achieve complementary advantages, thereby obtaining fusion semantic features with powerful representation capabilities. Finally, classification is performed based on these fused semantic features, and the classification results are fused. The classification accuracy of multi-source remote sensing data is further improved through feature-level fusion and decision-level fusion.

[0181] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0182] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0183] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0184] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0185] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0186] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0187] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0188] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0189] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0190] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store programs.

[0191] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A multi-source remote sensing data fusion classification method, It is characterized in that include: Acquire a first image block of a first remote sensing image and a corresponding second image block of a second remote sensing image, wherein the first remote sensing image and the second remote sensing image have different data sources; Performing dynamic convolution feature extraction on input data using at least one cascaded data dynamic interaction module to obtain output data, wherein the output data of the previous data dynamic interaction module is used as the input data of the next data dynamic interaction module, the input data of the first data dynamic interaction module is the first image block and the second image block, and the output data of the last data dynamic interaction module is obtained as the first interaction feature and the second interaction feature; Using a first fusion module to extract semantic features from the first interaction feature to obtain a first semantic feature; using an attention fusion module to fuse the first interaction feature and the second interaction feature to obtain an attention feature; using a second fusion module to extract semantic features from the attention feature to obtain a fused semantic feature; using a third fusion module to extract semantic features from the second interaction feature to obtain a second semantic feature; The first semantic feature, the second semantic feature and the fused semantic feature are respectively input into corresponding classifiers for classification prediction to obtain a first classification result, a second classification result and a third classification result, and a weighted result of the first classification result, the second classification result and the third classification result is obtained as a pixel classification result of the first image block and the second image block; the pixel classification results corresponding to all image blocks of the first remote sensing image and the second remote sensing image are obtained, and a fused classification result is obtained according to all the pixel classification results.

2. The multi-source remote sensing data fusion classification method according to claim 1, It is characterized in that The input data includes first input data and second input data, and the output data includes first output data and second output data; and the step of extracting dynamic convolution features from the input data using the data dynamic interaction module to obtain output data includes: Using a first convolutional layer to perform feature extraction on the first input data to obtain a first dimensionality reduction feature, and using a second convolutional layer to perform feature extraction on the second input data to obtain a second dimensionality reduction feature; Using a cascaded first pooling layer and a third convolutional layer to extract features from the first input data to obtain a first filter feature, and using a cascaded second pooling layer and a fourth convolutional layer to extract features from the second input data to obtain a second filter feature; Performing depth-by-depth convolution on the first dimension reduction feature and the second filtering feature to obtain a first depth convolution feature, and inputting the first depth convolution feature into a fifth convolution layer to obtain the first output data; The second dimensionality reduction feature and the first filtering feature are depth-wise convolved to obtain a second depth convolution feature, and the second depth convolution feature is input into the sixth convolution layer to obtain the second output data. The first output data of the last data dynamic interaction module is the first interaction feature, and the second output data is the second interaction feature.

3. The multi-source remote sensing data fusion classification method according to claim 1, It is characterized in that The using the attention fusion module to perform feature fusion on the first interaction feature and the second interaction feature to obtain an attention feature includes: Accumulating the first interaction feature and the second interaction feature to obtain an accumulated feature, and inputting the accumulated feature into the attention fusion module for feature fusion to obtain a fused feature; Obtaining a product of the first interactive feature and the fused feature to obtain a first intermediate feature, and obtaining a product of the second interactive feature and an associated feature of the fused feature to obtain a second intermediate feature, wherein a sum of the associated feature and the fused feature is one; The attention feature is obtained according to the first intermediate feature and the second intermediate feature.

4. The multi-source remote sensing data fusion classification method according to claim 3, It is characterized in that The step of inputting the accumulated features into the attention fusion module for feature fusion to obtain fused features includes: Performing a pooling operation, a first point-by-point convolution operation, a first activation operation, a second point-by-point convolution operation, and a second activation operation on the accumulated features in sequence to obtain a first fused feature; Performing a third point-by-point convolution operation, a third activation operation, a fourth point-by-point convolution operation, and a fourth activation operation on the accumulated features in sequence to obtain a second fused feature; Accumulating the first fusion feature and the second fusion feature to obtain an accumulated fusion feature; After performing a fifth activation operation on the accumulated fusion feature, a fused activation feature is obtained, and the product between the fused activation feature and the accumulated feature is calculated to obtain the fused feature.

5. The multi-source remote sensing data fusion classification method according to claim 1, It is characterized in that The first fusion module is obtained by cascading a first pooling layer, a first vector stretching layer, and a first fully connected layer, and the first fusion module is used to extract semantic features from the first interaction features to obtain the first semantic features, including: Using the first pooling layer to perform a pooling operation on the first interaction feature to obtain a first pooling feature; Using the first vector stretching layer to perform a vector stretching operation on the first pooled feature to obtain a first vector stretching feature; The first fully connected layer is used to perform a fully connected operation on the first vector stretching feature to obtain the first semantic feature.

6. The multi-source remote sensing data fusion classification method according to claim 1, It is characterized in that The step of acquiring a first image block of a first remote sensing image and a corresponding second image block of a second remote sensing image comprises: Selecting a first initial image block of a preset size in the first remote sensing image based on the selected marked pixels, and selecting a second initial image block of the preset size in the second remote sensing image based on the marked pixels; The first initial image block is scaled using a first image convolution layer to obtain the first image block, and the second initial image block is scaled using a second image convolution layer to obtain the second image block.

7. A multi-source remote sensing data fusion classification device, It is characterized in that include: A data acquisition module: used for acquiring a first image block of a first remote sensing image and a corresponding second image block of a second remote sensing image, wherein the data sources of the first remote sensing image and the second remote sensing image are different; Interaction feature extraction module: used to perform dynamic convolution feature extraction on input data using at least one cascaded data dynamic interaction module to obtain output data, wherein the output data of the previous data dynamic interaction module is used as the input data of the next data dynamic interaction module, the input data of the first data dynamic interaction module is the first image block and the second image block, and the output data of the last data dynamic interaction module is obtained as the first interaction feature and the second interaction feature; Semantic feature fusion module: used to use the first fusion module to perform semantic feature extraction on the first interaction feature to obtain a first semantic feature; use the attention fusion module to perform feature fusion on the first interaction feature and the second interaction feature to obtain an attention feature; use the second fusion module to perform semantic feature extraction on the attention feature to obtain a fused semantic feature; use the third fusion module to perform semantic feature extraction on the second interaction feature to obtain a second semantic feature; Classification output module: used to input the first semantic feature, the second semantic feature and the fused semantic feature into corresponding classifiers for classification prediction, respectively, to obtain a first classification result, a second classification result and a third classification result, to obtain a weighted result of the first classification result, the second classification result and the third classification result as a pixel classification result of the first image block and the second image block; to obtain the pixel classification results corresponding to all image blocks of the first remote sensing image and the second remote sensing image, and to obtain a fused classification result based on all the pixel classification results.

8. An electronic device, It is characterized in that The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the multi-source remote sensing data fusion classification method according to any one of claims 1 to 6 when executing the computer program.

9. A storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the multi-source remote sensing data fusion classification method according to any one of claims 1 to 6 is implemented.