Optical and SAR remote sensing image ground object classification method based on multi-scale perception
Through the multi-scale sensing optical and SAR remote sensing image geographic classification method, the multi-source remote sensing image fusion network model is used for modulation and upsampling, which solves the problem of failing to fully utilize image complementary information in the prior art, and improves the problem of Effect and robustness of landform classification.
Patent Information
- Application Number
- CN202410797984.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-20
- Publication Date
- 2024-11-26
AI Technical Summary
When the prior art uses optical remote sensing images and SAR remote sensing images to classify land objects, the complementary information and interaction relationships of multi-source remote sensing images are not fully explored, resulting in poor classification effects.
Using the optical and SAR remote sensing image classification method based on multi-scale perception, the pre-trained multi-source remote sensing image fusion network model is input to the pre-trained multi-source remote sensing image fusion network model, modular feature extraction and modulation are performed, combined with multi-scale perception and upsample to get better classification results.
The full fusion of optical and SAR remote sensing image features is achieved, the effect and robustness of land objects classification is improved, the impact of land objects size differences under the unique viewing angle of remote sensing images is overcome, and the generalization ability of classification effect is improved.
Smart Images

Figure CN119027800A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a method and device for classifying ground objects in optical and SAR remote sensing images based on multi-scale perception. Background Art
[0002] Remote sensing technology is an important means of earth observation. The remote sensing images obtained by it have a unique bird's-eye view, rich information about ground objects, and a large difference in the scale of ground object categories. Optical remote sensing images and SAR (synthetic aperture radar) remote sensing images are two widely used remote sensing data. Optical remote sensing images can intuitively reflect the color information of ground objects and the real surface conditions, but are easily affected by weather and lighting conditions. SAR remote sensing images are sensitive to specific ground objects such as surface height changes, textures, and water bodies. They have the characteristics of all-day, all-weather, and ignoring adverse weather and atmospheric disturbances such as fog, snow, clouds, and rain. However, due to its imaging mechanism, there is coherent speckle noise and poor interpretation. In order to achieve better ground object classification results, some studies have proposed the combined use of optical remote sensing images and SAR remote sensing images for ground object classification.
[0003] At present, the main implementation method for combining optical remote sensing images and SAR remote sensing images for object classification is the pixel-level implementation method. Specifically, pixel values are obtained from various modal images through a neural network-based method, and directly input into a multi-layer perceptron for pixel-level classification. The obtained pixel classification results are then summarized to obtain the final classification result. However, this method is only a multimodal fusion at the pixel level, and does not fully explore the complementary information and interactive relationship of multi-source remote sensing images, and the final classification effect is poor. Summary of the invention
[0004] In order to solve some or all of the technical problems existing in the above-mentioned prior art, the present invention provides a method and device for classifying objects in optical and SAR remote sensing images based on multi-scale perception.
[0005] The technical solution of the present invention is as follows:
[0006] In a first aspect, a method for classifying objects in optical and SAR remote sensing images based on multi-scale perception is provided, the method comprising:
[0007] Acquire optical remote sensing images and SAR remote sensing images to be detected;
[0008] Preprocessing the optical remote sensing image and the SAR remote sensing image to meet the format required by the constructed multi-source remote sensing image fusion network model;
[0009] Inputting the optical remote sensing image and the SAR remote sensing image into the pre-trained multi-source remote sensing image fusion network model to obtain corresponding classification results, wherein the multi-source remote sensing image fusion network model extracts features of the optical remote sensing image and features of the SAR remote sensing image, performs modal feature modulation on the two extracted features, and performs multi-scale perception and upsampling on the modulated fused modal features to obtain classification results;
[0010] The classification result is output.
[0011] In some optional implementations, the preprocessing includes: image cropping.
[0012] In some optional implementations, the multi-source remote sensing image fusion network model includes:
[0013] A feature extraction module, comprising a first feature extraction unit and a second feature extraction unit, wherein the input of the first feature extraction unit and the input of the second feature extraction unit are inputs of the multi-source remote sensing image fusion network model, the first feature extraction unit is used to extract the features of the input optical remote sensing image, and the second feature extraction unit is used to extract the features of the input SAR remote sensing image;
[0014] a modal feature modulation module, connected to the first feature extraction unit and the second feature extraction unit, respectively, for processing the feature map output by the first feature extraction unit and the feature map output by the second feature extraction unit according to a preset modulation operation, and obtaining and outputting a fused modal feature map;
[0015] A multi-scale perception module, connected to the modal feature modulation module, for performing multi-scale feature extraction on the fused modal feature map, and obtaining and outputting a rich-scale feature map that captures receptive fields of multiple different scales;
[0016] The upsampling module is connected to the multi-scale perception module and is used to upsample the rich-scale feature map to the same size as the input remote sensing image to obtain and output a classification result.
[0017] In some optional implementations, the first feature extraction unit and the second feature extraction unit adopt a deep convolutional neural network.
[0018] In some optional implementations, the preset modulation operation includes:
[0019] Stacking and compressing the feature map output by the first feature extraction unit and the feature map output by the second feature extraction unit;
[0020] Perform global pooling on the first fused feature map obtained after compression;
[0021] The second fused feature map obtained by global pooling is split into two tensors of the same shape to obtain the second fused feature map of the optical remote sensing image and the second fused feature map of the SAR remote sensing image;
[0022] Normalizing the second fused feature map of the optical remote sensing image and the second fused feature map of the SAR remote sensing image respectively;
[0023] The two modal feature maps obtained after the normalization operation are concatenated.
[0024] In some optional implementations, the first fused feature map is represented as:
[0025]
[0026] The second fusion feature map is expressed as:
[0027]
[0028] The two modal characteristic graphs are expressed as:
[0029]
[0030] in, Represents the first fusion feature map, Cov opt Represents the feature map corresponding to the optical remote sensing image output by the first feature extraction unit, Cov SAR represents the feature map corresponding to the SAR remote sensing image output by the second feature extraction unit, || represents the channel stacking operation, Squeezed represents the channel compression operation, represents the second fusion feature map, Avg represents global pooling, softmax represents normalization operation, W opt Represents the modal feature map corresponding to the optical remote sensing image, Represents the second fusion feature map of the optical remote sensing image, W SAR Represents the modal characteristic map corresponding to the SAR remote sensing image, Represents the second fused feature map of SAR remote sensing image.
[0031] In some optional implementations, the multi-source remote sensing image fusion network model is trained in the following manner:
[0032] Acquire a training sample set, wherein the training sample includes paired optical remote sensing images and SAR remote sensing images, object detection frame annotations corresponding to the remote sensing images, and annotation information of each object category;
[0033] The optical remote sensing images and SAR remote sensing images in the training samples in the training sample set are taken as input, and the ground object detection frame annotations and each ground object category annotation information in the training samples are taken as output to train the multi-source remote sensing image fusion network model.
[0034] In a second aspect, a device for classifying objects in optical and SAR remote sensing images based on multi-scale perception is provided, the device comprising:
[0035] An acquisition unit is configured to acquire an optical remote sensing image and a SAR remote sensing image to be detected;
[0036] A preprocessing unit is configured to preprocess the optical remote sensing image and the SAR remote sensing image to meet the format required by the constructed multi-source remote sensing image fusion network model;
[0037] The detection unit is configured to input the optical remote sensing image and the SAR remote sensing image into the pre-trained multi-source remote sensing image fusion network model to obtain a corresponding classification result, wherein the multi-source remote sensing image fusion network model extracts the features of the optical remote sensing image and the features of the SAR remote sensing image, performs modal feature modulation on the two extracted features, performs multi-scale perception and upsampling on the modulated fusion modal features, and obtains the classification result;
[0038] The output unit is configured to output the classification result.
[0039] The main advantages of the technical solution of the present invention are as follows:
[0040] The optical and SAR remote sensing image ground object classification method and device based on multi-scale perception of the present invention can fully explore the respective characteristics and complementary information of optical remote sensing images and SAR remote sensing images by utilizing adaptive modal feature modulation to process the characteristics of optical remote sensing images and SAR remote sensing images, realize the full fusion of the characteristics of multi-source heterogeneous data, obtain better feature expression, and improve the classification effect; by performing multi-scale receptive field mapping, multi-scale ground object targets can be perceived, avoiding the influence of the problem that the ground objects present large size differences under the bird's-eye view unique to remote sensing images, further improving the classification effect, and having better robustness and generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] The drawings described herein are used to provide a further understanding of the embodiments of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:
[0042] Figure 1 This is a flow chart of a method for classifying objects in optical and SAR remote sensing images based on multi-scale perception according to an embodiment of the present invention;
[0043] Figure 2 A schematic diagram of the structure of a multi-source remote sensing image fusion network model provided by an embodiment of the present invention;
[0044] Figure 3 The present invention is a block diagram of a device for classifying objects in optical and SAR remote sensing images based on multi-scale perception according to an embodiment of the present invention. DETAILED DESCRIPTION
[0045] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0046] The technical solution provided by the embodiments of the present invention is described in detail below with reference to the accompanying drawings.
[0047] See also Figure 1 In a first aspect, an embodiment of the present invention provides a method for classifying objects in optical and SAR remote sensing images based on multi-scale perception, the method comprising the following steps S1-S4:
[0048] Step S1, obtaining an optical remote sensing image and a SAR remote sensing image to be detected.
[0049] Specifically, optical remote sensing images and SAR remote sensing images to be used for object classification detection are acquired according to actual needs.
[0050] Step S2, preprocessing the optical remote sensing image and the SAR remote sensing image to meet the format required by the constructed multi-source remote sensing image fusion network model.
[0051] In one embodiment of the present invention, when using a multi-source remote sensing image fusion network model to perform ground object classification detection on optical remote sensing images and SAR remote sensing images, the optical remote sensing images and the SAR remote sensing images are required to be paired with each other, that is, the optical remote sensing images and the SAR remote sensing images are required to have the same imaging area and size.
[0052] In order to meet the use requirements of the multi-source remote sensing image fusion network model, in one embodiment of the present invention, the acquired optical remote sensing images and SAR remote sensing images are preprocessed to obtain paired optical remote sensing images and SAR remote sensing images, and the image sizes of the preprocessed optical remote sensing images and SAR remote sensing images are made to conform to the format required by the multi-source remote sensing image fusion network model.
[0053] Since the original image sizes of optical remote sensing images and SAR remote sensing images are usually large, in one embodiment of the present invention, the preprocessing includes: image cropping.
[0054] Specifically, according to the imaging areas of the optical remote sensing images and SAR remote sensing images and the input image format requirements of the multi-source remote sensing image fusion network model, the acquired optical remote sensing images and SAR remote sensing images are cropped without overlapping to obtain multiple groups of optical remote sensing image and SAR remote sensing image slices that are paired with each other and whose image sizes meet the input image format requirements.
[0055] Step S3, inputting the optical remote sensing image and the SAR remote sensing image into a pre-trained multi-source remote sensing image fusion network model to obtain corresponding classification results.
[0056] refer to Figure 2 In one embodiment of the present invention, in order to enable the multi-source remote sensing image fusion network model to fully explore and fuse the complementary information and interactive relationship of optical remote sensing images and SAR remote sensing images, obtain better cross-modal feature expression, and overcome the problem of large size differences of objects under the bird's-eye view of remote sensing images, which affects the classification and detection of objects, and improve the classification effect of objects, the constructed multi-source remote sensing image fusion network model includes:
[0057] The feature extraction module includes a first feature extraction unit and a second feature extraction unit. The input of the first feature extraction unit and the input of the second feature extraction unit are inputs of the multi-source remote sensing image fusion network model. The first feature extraction unit is used to extract the features of the input optical remote sensing image, and the second feature extraction unit is used to extract the features of the input SAR remote sensing image.
[0058] A modal feature modulation module, connected to the first feature extraction unit and the second feature extraction unit, respectively, for processing the feature map output by the first feature extraction unit and the feature map output by the second feature extraction unit according to a preset modulation operation, and obtaining and outputting a fused modal feature map;
[0059] The multi-scale perception module is connected to the modal feature modulation module and is used to extract multi-scale features from the fused modal feature map, and obtain and output a rich-scale feature map that captures receptive fields of multiple different scales;
[0060] The upsampling module is connected to the multi-scale perception module and is used to upsample the rich-scale feature map to the same size as the input remote sensing image to obtain and output the classification result.
[0061] Specifically, the first feature extraction unit and the second feature extraction unit respectively receive the optical remote sensing image and the SAR remote sensing image input into the multi-source remote sensing image fusion network model, the first feature extraction unit extracts features from the optical remote sensing image, obtains the corresponding feature map and outputs it to the modal feature modulation module, the second feature extraction unit extracts features from the SAR remote sensing image, obtains the corresponding feature map and outputs it to the modal feature modulation module; the modal feature modulation module processes the two received feature maps according to the preset modulation operation, obtains the fused modal feature map and outputs it to the multi-scale perception module; the multi-scale perception module performs multi-scale feature extraction on the received fused modal feature map, obtains a rich-scale feature map that captures receptive fields of multiple different scales and outputs it to the upsampling module; the upsampling module performs an upsampling operation on the received rich-scale feature map to upsample the rich-scale feature map to the same size as the input remote sensing image, and outputs the obtained feature map as the classification result.
[0062] In order to improve the model's ability to represent features and achieve full mining and fusion of multi-source remote sensing image information, in one embodiment of the present invention, the first feature extraction unit and the second feature extraction unit use a deep convolutional neural network with the same structure.
[0063] Specifically, the feature map corresponding to the optical remote sensing image extracted by the first feature extraction unit is expressed as:
[0064]
[0065] The feature map corresponding to the SAR remote sensing image extracted by the second feature extraction unit is expressed as:
[0066]
[0067] Among them, Cov opt Represents the feature map corresponding to the optical remote sensing image output by the first feature extraction unit, Cov SAR represents the feature map corresponding to the SAR remote sensing image output by the second feature extraction unit, L represents the number of convolutional layers of the deep convolutional neural network, represents optical remote sensing image, represents SAR remote sensing image, represents the linear projection of the optical remote sensing image, Represents the linear projection of SAR remote sensing image, represents the feature map learned by the deep convolutional neural network with a convolutional layer depth of i through optical remote sensing images, Represents the feature map learned by the deep convolutional neural network with a convolutional layer depth of j through SAR remote sensing images.
[0068] Furthermore, in one embodiment of the present invention, the preset modulation operation includes:
[0069] Stacking and compressing the feature map output by the first feature extraction unit and the feature map output by the second feature extraction unit;
[0070] Perform global pooling on the first fused feature map obtained after compression;
[0071] The second fused feature map obtained by global pooling is split into two tensors of the same shape to obtain the second fused feature map of the optical remote sensing image and the second fused feature map of the SAR remote sensing image;
[0072] Normalizing the second fused feature map of the optical remote sensing image and the second fused feature map of the SAR remote sensing image respectively;
[0073] The two modal feature maps obtained after the normalization operation are concatenated.
[0074] Specifically, the feature map output by the first feature extraction unit and the feature map output by the second feature extraction unit are stacked and compressed to be expressed as:
[0075]
[0076] The global pooling of the first fused feature map obtained after compression is expressed as:
[0077]
[0078] The second fused feature map obtained after global pooling Split into two tensors of the same shape to obtain the second fusion feature map of the optical remote sensing image The second fusion feature map of SAR remote sensing image
[0079] The normalization operation of the second fusion feature map of the optical remote sensing image and the second fusion feature map of the SAR remote sensing image is expressed as:
[0080]
[0081] in, Represents the first fusion feature map, Cov opt Represents the feature map corresponding to the optical remote sensing image output by the first feature extraction unit, Cov SAR represents the feature map corresponding to the SAR remote sensing image output by the second feature extraction unit, || represents the channel stacking operation, Squeezed represents the channel compression operation, represents the second fusion feature map, Avg represents global pooling, softmax represents normalization operation, W opt Represents the modal feature map corresponding to the optical remote sensing image, Represents the second fusion feature map of the optical remote sensing image, W SAR Represents the modal characteristic map corresponding to the SAR remote sensing image, Represents the second fusion feature map of SAR remote sensing image, through the modal feature map W opt and the modal feature map W SAR The fused modal feature map is obtained by splicing.
[0082] In one embodiment of the present invention, the above-mentioned modulation operation can fully integrate the feature maps of the two modalities, accurately control the fusion ratio of multimodal data, guide the network model to mine the implicit features and complementary information of cross-modal interactions, obtain better cross-modal feature expression, and thus improve the classification effect of remote sensing image objects.
[0083] Furthermore, in one embodiment of the present invention, the multi-scale perception module performs multi-scale feature extraction on the fused modal feature map output by the modal feature modulation module to obtain a rich-scale feature map that captures receptive fields of four different scales with pixel area sizes of 7, 9, 15, and 29.
[0084] In one embodiment of the present invention, the multi-scale perception module is implemented in parallel using a multi-branch multi-layer convolution structure with different receptive fields.
[0085] In one embodiment of the present invention, by performing rich-scale information aggregation of a multi-scale receptive field, it is possible to overcome the influence of the large size differences of objects in a bird's-eye view unique to remote sensing images, thereby improving the object classification effect.
[0086] Furthermore, in one embodiment of the present invention, the upsampling module performs 2 times upsampling each time when performing the upsampling operation until the rich-scale feature map is upsampled to the same size as the input remote sensing image.
[0087] Furthermore, in one embodiment of the present invention, the multi-source remote sensing image fusion network model is trained in the following manner:
[0088] Step S31, obtaining a training sample set, the training samples including paired optical remote sensing images and SAR remote sensing images, object detection frame annotations corresponding to the remote sensing images, and annotation information of each object category;
[0089] Step S32, taking the optical remote sensing images and SAR remote sensing images in the training samples in the training sample set as input, taking the ground object detection frame annotations and each ground object category annotation information in the training samples as output, and training the multi-source remote sensing image fusion network model.
[0090] Specifically, in one embodiment of the present invention, when obtaining a training sample set, the existing optical remote sensing images and SAR remote sensing images are searched, and image registration is performed using ENVI software; then, optical remote sensing image slices and SAR remote sensing image slices of a specific size with the same area are cropped from the registered images; finally, each feature in the remote sensing image slice is labeled with a detection frame and category information. The specific size is set according to the actual situation, and in one embodiment of the present invention, it is 256*256 pixels.
[0091] Furthermore, in one embodiment of the present invention, the optical remote sensing images and SAR remote sensing images in the training samples in the training sample set are used as input, and the ground object detection frame annotations and each ground object category annotation information in the training samples are used as output to train the multi-source remote sensing image fusion network model, which specifically includes the following steps S321-S323:
[0092] Step S321, input the optical remote sensing images and SAR remote sensing images in all training samples into the multi-source remote sensing image fusion network model in batches, and obtain the predicted ground object position and category information corresponding to the remote sensing images output by the multi-source remote sensing image fusion network model.
[0093] In one embodiment of the present invention, the optical remote sensing images and SAR remote sensing images in the training samples are input from the input end of the multi-source remote sensing image fusion network model, processed in turn by each layer in the multi-source remote sensing image fusion network model, and output from the output end of the multi-source remote sensing image fusion network model. The information output by the multi-source remote sensing image fusion network model is the predicted ground object position and category information corresponding to the remote sensing image.
[0094] In one embodiment of the present invention, the multi-source remote sensing image fusion network model is a model to be trained, and the model parameters are initialization parameters. During the training process of the model, the model parameters are continuously updated and learned.
[0095] Step S322, calculating a preset loss function according to the ground object detection frame annotation corresponding to the remote sensing image in the training sample and each ground object category annotation information and the predicted ground object position and category information corresponding to the remote sensing image.
[0096] Specifically, based on a pre-set loss function, the loss function is calculated according to the object detection box annotations and each object category annotation information corresponding to the remote sensing image in each training sample input into the multi-source remote sensing image fusion network model and the predicted object position and category information corresponding to the remote sensing image output by the multi-source remote sensing image fusion network model.
[0097] In one embodiment of the present invention, the specific type of the loss function is set according to actual needs. For example, the loss function adopts the cross entropy loss function, which is specifically expressed as:
[0098] Loss=-∑ i (y i ·log(p i ))
[0099] Among them, y i Indicates the value of the i-th category in the true label, which is 0 or 1. i Represents the probability that the model predicts the i-th category.
[0100] Step S323, determine whether the preset training stop condition is reached, if so, use the current multi-source remote sensing image fusion network model as the multi-source remote sensing image fusion network model that has completed training, if not, use the preset loss function to update the parameters of the multi-source remote sensing image fusion network model, and return to step S321.
[0101] In one embodiment of the present invention, the training stop condition is specifically set according to the actual situation, for example, the number of training iterations reaches a set iteration number or the optimization index reaches a set threshold. Among them, the loss function can be used as the optimization index.
[0102] Furthermore, in one embodiment of the present invention, a stochastic gradient descent method is used to train and update the parameters of the multi-source remote sensing image fusion network model.
[0103] Specifically, the following formula is used to update the parameters of the multi-source remote sensing image fusion network model:
[0104]
[0105] Among them, θ represents the parameter set of the multi-source remote sensing image fusion network model, Δ[·] represents the optimizer, η represents the learning rate, and Loss represents the loss function. Among them, the optimizer is set according to the actual situation, for example, Adam, and the learning rate is pre-set to control the speed of updating the parameters of the multi-source remote sensing image fusion network model.
[0106] Step S4, output the classification result.
[0107] In one embodiment of the present invention, after step S3, a classification result map corresponding to the optical remote sensing image and the SAR remote sensing image output by the multi-source remote sensing image fusion network model can be obtained, and the classification result map is the final classification result.
[0108] refer to Figure 3 In a second aspect, an embodiment of the present invention further provides an optical and SAR remote sensing image object classification device based on multi-scale perception, the device comprising:
[0109] The acquisition unit 100 is configured to acquire the optical remote sensing image and the SAR remote sensing image to be detected;
[0110] The preprocessing unit 200 is configured to preprocess the optical remote sensing image and the SAR remote sensing image to meet the format required by the constructed multi-source remote sensing image fusion network model;
[0111] The detection unit 300 is configured to input the optical remote sensing image and the SAR remote sensing image into a pre-trained multi-source remote sensing image fusion network model to obtain a corresponding classification result;
[0112] The output unit 400 is configured to output the classification result.
[0113] In one embodiment of the present invention, the above-mentioned units are devices corresponding to the steps of the above-mentioned method. The specific working principle of each unit and its beneficial effects can be found in the above-mentioned method, which will not be repeated here.
[0114] Furthermore, in one embodiment of the present invention, the device also includes: a training unit 500, which is configured to obtain a training sample set, the training samples include paired optical remote sensing images and SAR remote sensing images, object detection frame annotations corresponding to the remote sensing images, and each object category annotation information; use the optical remote sensing images and SAR remote sensing images in the training samples in the training sample set as input, and use the object detection frame annotations and each object category annotation information in the training samples as output, to train the multi-source remote sensing image fusion network model.
[0115] The optical and SAR remote sensing image ground object classification method and device based on multi-scale perception provided by an embodiment of the present invention can fully explore the respective characteristics and complementary information of optical remote sensing images and SAR remote sensing images by utilizing adaptive modal feature modulation to process the characteristics of optical remote sensing images and SAR remote sensing images, realize the full fusion of the characteristics of multi-source heterogeneous data, obtain better feature expression, and improve the classification effect; by performing multi-scale receptive field mapping, it is possible to perceive multi-scale ground object targets, avoid the influence of the problem that the ground objects present large size differences under the bird's-eye view unique to remote sensing images, further improve the classification effect, and have better robustness and generalization ability.
[0116] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for classifying objects in optical and SAR remote sensing images based on multi-scale perception, characterized in that: include: Acquire optical remote sensing images and SAR remote sensing images to be detected; Preprocessing the optical remote sensing image and the SAR remote sensing image to meet the format required by the constructed multi-source remote sensing image fusion network model; Inputting the optical remote sensing image and the SAR remote sensing image into the pre-trained multi-source remote sensing image fusion network model to obtain corresponding classification results, wherein the multi-source remote sensing image fusion network model extracts features of the optical remote sensing image and features of the SAR remote sensing image, performs modal feature modulation on the two extracted features, and performs multi-scale perception and upsampling on the modulated fused modal features to obtain classification results; The classification result is output.
2. The method for classifying objects in optical and SAR remote sensing images based on multi-scale perception according to claim 1, characterized in that: The preprocessing includes: image cropping.
3. The method for classifying objects in optical and SAR remote sensing images based on multi-scale perception according to claim 1, characterized in that: The multi-source remote sensing image fusion network model includes: A feature extraction module, comprising a first feature extraction unit and a second feature extraction unit, wherein the input of the first feature extraction unit and the input of the second feature extraction unit are inputs of the multi-source remote sensing image fusion network model, the first feature extraction unit is used to extract the features of the input optical remote sensing image, and the second feature extraction unit is used to extract the features of the input SAR remote sensing image; a modal feature modulation module, connected to the first feature extraction unit and the second feature extraction unit, respectively, for processing the feature map output by the first feature extraction unit and the feature map output by the second feature extraction unit according to a preset modulation operation, and obtaining and outputting a fused modal feature map; A multi-scale perception module, connected to the modal feature modulation module, for performing multi-scale feature extraction on the fused modal feature map, and obtaining and outputting a rich-scale feature map that captures receptive fields of multiple different scales; The upsampling module is connected to the multi-scale perception module and is used to upsample the rich-scale feature map to the same size as the input remote sensing image to obtain and output a classification result.
4. The method for classifying objects in optical and SAR remote sensing images based on multi-scale perception according to claim 3 is characterized in that: The first feature extraction unit and the second feature extraction unit adopt a deep convolutional neural network.
5. The method for classifying objects in optical and SAR remote sensing images based on multi-scale perception according to claim 3 is characterized in that: The preset modulation operation includes: Stacking and compressing the feature map output by the first feature extraction unit and the feature map output by the second feature extraction unit; Perform global pooling on the first fused feature map obtained after compression; The second fused feature map obtained by global pooling is split into two tensors of the same shape to obtain the second fused feature map of the optical remote sensing image and the second fused feature map of the SAR remote sensing image; Normalizing the second fused feature map of the optical remote sensing image and the second fused feature map of the SAR remote sensing image respectively; The two modal feature maps obtained after the normalization operation are concatenated.
6. The method for classifying objects in optical and SAR remote sensing images based on multi-scale perception according to claim 5, characterized in that: The first fusion feature map is expressed as: The second fusion feature map is expressed as: The two modal characteristic graphs are expressed as: in, Represents the first fusion feature map, Cov opt Represents the feature map corresponding to the optical remote sensing image output by the first feature extraction unit, Cov SAR It represents the feature map corresponding to the SAR remote sensing image output by the second feature extraction unit, ∥ represents the channel stacking operation, Squeezed represents the channel compression operation, represents the second fusion feature map, Avg represents global pooling, softmax represents normalization operation, W opt Represents the modal feature map corresponding to the optical remote sensing image, Represents the second fusion feature map of the optical remote sensing image, W SAR Represents the modal characteristic map corresponding to the SAR remote sensing image, Represents the second fused feature map of SAR remote sensing image.
7. The method for classifying objects in optical and SAR remote sensing images based on multi-scale perception according to any one of claims 1 to 6, characterized in that: The multi-source remote sensing image fusion network model is trained in the following way: Acquire a training sample set, wherein the training sample includes paired optical remote sensing images and SAR remote sensing images, object detection frame annotations corresponding to the remote sensing images, and annotation information of each object category; The optical remote sensing images and SAR remote sensing images in the training samples in the training sample set are taken as input, and the ground object detection frame annotations and each ground object category annotation information in the training samples are taken as output to train the multi-source remote sensing image fusion network model.
8. An optical and SAR remote sensing image object classification device based on multi-scale perception, characterized in that: include: An acquisition unit is configured to acquire an optical remote sensing image and a SAR remote sensing image to be detected; A preprocessing unit is configured to preprocess the optical remote sensing image and the SAR remote sensing image to meet the format required by the constructed multi-source remote sensing image fusion network model; The detection unit is configured to input the optical remote sensing image and the SAR remote sensing image into the pre-trained multi-source remote sensing image fusion network model to obtain a corresponding classification result, wherein the multi-source remote sensing image fusion network model extracts the features of the optical remote sensing image and the features of the SAR remote sensing image, performs modal feature modulation on the two extracted features, performs multi-scale perception and upsampling on the modulated fusion modal features, and obtains the classification result; The output unit is configured to output the classification result.
Citation Information
Patent Citations
Multi-source remote sensing image fusion method based on deep learning
CN106295714A
Multi-modal image land coverage type classification method based on network architecture search
CN116486142A
