A method, device, equipment and product for classifying objects in remote sensing images

Abstract

This application proposes a method, device, equipment, and product for remote sensing image object classification, which are applied to the field of remote sensing image processing. The method for remote sensing image object classification includes: acquiring a target remote sensing image; transforming the target remote sensing image from the spatial domain to the frequency domain to obtain a frequency domain image; performing feature extraction on the target remote sensing image to obtain a spatial domain feature map; performing feature extraction on the frequency domain image to obtain a frequency domain feature map; fusing the spatial domain feature map and the frequency domain feature map to obtain a fused feature map; and classifying the fused feature map to obtain an object classification result for the target remote sensing image. Thus, by combining the spatial domain features and frequency domain features of the remote sensing image, remote sensing image object classification based on multimodal features is achieved, thereby improving classification accuracy.

Description

Technical Field

[0001] The present application is applied to the field of remote sensing image technology, and in particular relates to a method, device, equipment and product for classifying objects in remote sensing images. Background Art

[0002] With the rapid development of remote sensing technology, remote sensing images have been widely used in land resource monitoring, environmental assessment, disaster warning and other fields, including land feature classification based on remote sensing images.

[0003] In practical applications, remote sensing images are typically acquired using multispectral or hyperspectral sensors. These sensors can capture the spectral characteristics of objects in different wavelengths, allowing remote sensing images to contain rich information about their characteristics. Remote sensing images are also characterized by high complexity, such as different objects exhibiting similar spectral characteristics and blurred boundaries between them.

[0004] Faced with complex remote sensing images, the accuracy of remote sensing image object classification needs to be improved. Summary of the Invention

[0005] In order to solve the above problems, the present application proposes a method, device, equipment and product for classifying objects in remote sensing images, which can improve the accuracy of object classification in remote sensing maps.

[0006] In a first aspect, the present application provides a method for classifying land objects in a remote sensing map, comprising: acquiring a target remote sensing image; transforming the target remote sensing image from a spatial domain to a frequency domain to obtain a frequency domain image; performing feature extraction on the target remote sensing image to obtain a spatial domain feature map; performing feature extraction on the frequency domain image to obtain a frequency domain feature map; fusing the spatial domain feature map and the frequency domain feature map to obtain a fused feature map; and classifying the fused feature map to obtain a land object classification result of the target remote sensing image.

[0007] In a possible implementation, the feature extraction of the target remote sensing image is performed to obtain a spatial domain feature map; the feature extraction of the frequency domain image is performed to obtain a frequency domain feature map; the spatial domain feature map and the frequency domain feature map are fused to obtain a fused feature map; and the fused feature map is classified to obtain a ground object classification result of the target remote sensing image, including: inputting the target remote sensing image and the frequency domain image into a classification model; performing feature extraction on the target remote sensing image through a first feature extraction network in the classification model to obtain the spatial domain feature map; performing feature extraction on the frequency domain image through a second feature extraction network in the classification model to obtain the frequency domain feature map; performing fusion of the spatial domain feature map and the frequency domain feature map through a feature fusion network in the classification model to obtain the fused feature map; and classifying the fused feature map through a classification network in the classification model to obtain a ground object classification result of the target remote sensing image.

[0008] In one possible implementation, the first feature extraction network is a residual network, which includes a convolution module and multiple series-connected residual blocks. The first feature extraction network in the classification model is used to extract features from the target remote sensing image to obtain the spatial domain feature map, including: extracting features from the target remote sensing image through the convolution module to obtain the convolution feature map of the target remote sensing image; and downsampling the convolution feature map multiple times through the multiple series-connected residual blocks to obtain the spatial domain feature map.

[0009] In one possible implementation, the second feature extraction network is a two-layer routing attention network, which includes a convolution module and an attention module. The attention mechanism of the attention module adopts a two-layer routing attention mechanism. The second feature extraction network in the classification model is used to extract features from the frequency domain image to obtain the frequency domain feature map, including: extracting features from the frequency domain image through the convolution module to obtain a convolution feature map of the frequency domain image; performing cross-region feature interaction based on the two-layer routing attention mechanism on the convolution feature map through the attention module to obtain a first attention feature map; and obtaining the frequency domain feature map based on the first attention feature map.

[0010] In one possible implementation, the convolution module includes a convolution layer and a feature splicing layer, the position coding matrix in the feature splicing layer is a learning parameter in the training process, and the frequency domain image is subjected to feature extraction through the convolution module to obtain a convolution feature map of the frequency domain image, including: dividing the frequency domain image into blocks to obtain a plurality of non-overlapping image blocks; performing feature extraction on the plurality of non-overlapping image blocks respectively through the convolution layer to obtain feature maps corresponding to the plurality of non-overlapping image blocks; sorting the feature maps corresponding to the plurality of non-overlapping image blocks respectively according to the position order of the plurality of non-overlapping image blocks in the feature splicing layer, and adding position information to the sorted feature maps according to the position coding matrix to obtain the convolution feature map.

[0011] In one possible implementation, the attention module includes an attention layer, through which the convolutional feature map is subjected to cross-regional feature interaction based on a double-layer routing attention mechanism to obtain a first attention feature map, including: window division of the convolutional feature map to obtain multiple feature windows; determining the double-layer routing between the multiple feature windows according to the feature similarity between each of the multiple feature windows, the double-layer routing indicating the associated feature windows corresponding to the multiple feature windows in the dimension of feature similarity; determining the query vectors corresponding to the multiple feature windows, the key vectors corresponding to the multiple feature windows, and the value vectors corresponding to the multiple feature windows according to the double-layer routing and the feature representations corresponding to the multiple feature windows on the convolutional feature map; in the attention layer, determining the attention weight matrix according to the query vectors corresponding to the multiple feature windows, the key vectors corresponding to the multiple feature windows, the value vectors corresponding to the multiple feature windows, and the weight parameters of the attention layer, adding the attention weight matrix to the convolutional feature map to obtain the first attention feature map.

[0012] In one possible implementation, the attention module also includes layer normalization and a feedforward neural network. After the attention module performs cross-region feature interaction based on a double-layer routing attention mechanism on the convolution feature map to obtain a first attention feature map, it also includes: performing preliminary processing on the first attention feature map through the layer normalization; and enhancing the attention feature map after preliminary processing through the feedforward neural network.

[0013] In one possible implementation, obtaining the frequency domain feature map based on the first attention feature map includes: performing dimension adjustment on the first attention feature map; performing cross-region feature interaction based on a double-layer routing attention mechanism on the dimensionally adjusted first attention feature map through the attention module to obtain a second attention feature map; and obtaining the frequency domain feature map based on the second attention feature map.

[0014] In a possible implementation, the feature fusion network in the classification model is used to fuse the spatial domain feature map and the frequency domain feature map to obtain the fused feature map, including: in the feature fusion network, a dynamic exponential sliding average is used to fuse the spatial domain feature map and the frequency domain feature map to obtain the fused feature map, and the smoothing factor in the exponential sliding average is related to the training parameters of the feature fusion network.

[0015] The second aspect of the present application provides a device for classifying objects in a remote sensing map, comprising: an acquisition unit for acquiring a target remote sensing image; a spatial domain-frequency domain conversion unit for converting the target remote sensing image from the spatial domain to the frequency domain to obtain a frequency domain image; a spatial domain feature extraction unit for performing feature extraction on the target remote sensing image to obtain a spatial domain feature map; a frequency domain feature extraction unit for performing feature extraction on the frequency domain image to obtain a frequency domain feature map; a feature fusion unit for fusing the spatial domain feature map and the frequency domain feature map to obtain a fused feature map; and a object classification unit for classifying the fused feature map to obtain an object classification result of the target remote sensing image.

[0016] The third aspect of the present application provides an electronic device comprising a memory and a processor; the memory is connected to the processor and is used to store a program; the processor is used to implement the method for classifying land features in a remote sensing map as described in the first aspect of the present application or any possible implementation method of the first aspect of the present application by running the program in the memory.

[0017] The fourth aspect of the present application provides a chip, including a processor and a data interface, wherein the processor reads and runs a program stored in a memory through the data interface to execute the method for classifying land features in a remote sensing map as described in the first aspect of the present application or any possible implementation method of the first aspect of the present application.

[0018] The fifth aspect of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method for classifying land features in a remote sensing map as described in the first aspect of the present application or any possible implementation method of the first aspect of the present application.

[0019] In a sixth aspect, the present application provides a storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for classifying features in a remote sensing map as described in the first aspect of the present application or any possible implementation of the first aspect of the present application is implemented.

[0020] According to the method, device, equipment and product for object classification of remote sensing maps proposed in this application, in the process of object classification of target remote sensing images, the target remote sensing images are converted from the spatial domain to the frequency domain to obtain a frequency domain image; feature extraction is performed on the target remote sensing images and the frequency domain images respectively to obtain a spatial domain feature map and a frequency domain feature map. The spatial domain feature map reflects the detailed features of the target remote sensing image, such as color, texture, etc. The frequency domain feature map reflects the distribution of different frequency components in the target remote sensing image, which can better distinguish the noise and effective information of the target remote sensing image; the fused feature map obtained by fusing the spatial domain feature map and the frequency domain feature map is classified to determine the object classification result of the target remote sensing image. Compared with remote sensing image object classification based on single modal features, this application can extract richer multimodal features from complex remote sensing images, combining the advantages of spatial domain feature maps and frequency domain feature maps, and effectively improving the accuracy of remote sensing image object classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0022] Figure 1 It is a schematic diagram of the implementation environment involved in the embodiment of the present application.

[0023] Figure 2 Schematic diagram of the process of the method for classifying objects in remote sensing maps provided in the embodiment of the present application Figure 1 .

[0024] Figure 3 Schematic diagram of the process of the method for classifying objects in remote sensing maps provided in the embodiment of the present application Figure 2 .

[0025] Figure 4 This is an example diagram of classifying objects in remote sensing images using a classification model.

[0026] Figure 5 Schematic diagram of the structure of a device for classifying land features in a remote sensing map provided according to an embodiment of the present application.

[0027] Figure 6A schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0029] Sensors used to collect remote sensing images (such as multispectral sensors and hyperspectral sensors) can capture the spectral characteristics of objects in different wavelengths (for example, the chlorophyll content and moisture content of crops have unique spectral characteristics in different wavelengths). However, different objects may have similar spectral characteristics (for example, different crops at the same growth stage have similar spectral characteristics; for example, the reflectance of wheat and corn in certain wavelengths may be very similar). Furthermore, the boundaries between different objects may be blurred (for example, agricultural surfaces contain a variety of object types, such as bare land, roads, and water bodies, and the boundaries between some of these object types are blurred). Low-resolution remote sensing images may contain mixed pixels (the spectral characteristics of mixed pixels are a mixture of the spectral characteristics of different object types). This increases the difficulty of object classification in remote sensing images and affects their accuracy. More sophisticated classification methods are needed for remote sensing image object classification. Remote sensing image object classification, also known as remote sensing image classification, remote sensing image-based object classification, or remote sensing image-based object recognition, refers to the extraction of object category information from remote sensing images.

[0030] In related technologies, remote sensing image object classification methods include remote sensing image object classification methods based on traditional machine learning and remote sensing image object classification methods based on deep learning.

[0031] Traditional machine learning-based remote sensing image object classification methods include supervised and unsupervised classification methods. In supervised classification methods, a sufficient number of representative remote sensing images for each typical object type are manually selected as training samples. Based on these training samples, a classifier is trained using machine learning algorithms, such as maximum likelihood classification (MLC), support vector machines (SVM), and random forests (RF), to learn object characteristics. During training, classifier parameters are adjusted to optimize classification performance. However, these methods are limited in their ability to classify objects in complex remote sensing images. Unsupervised classification methods use clustering algorithms to cluster the pixel spectra of remote sensing images. Cluster centers are iteratively optimized during the clustering process, ultimately resulting in multiple clusters. The object type corresponding to each cluster is then manually interpreted. Consequently, traditional machine learning-based remote sensing image object classification methods rely on feature engineering and domain knowledge, requiring expert experience to optimize the classification process. Consequently, they are limited in their ability to classify objects in complex remote sensing images and struggle to capture the detailed information and contextual information found in high-resolution remote sensing images.

[0032] The specific process of deep learning-based remote sensing image object classification methods includes: first, acquiring remote sensing images and labeling the object types in the images; then, preprocessing and data augmentation of the remote sensing images. Preprocessing involves cutting large remote sensing images into smaller sizes, and data augmentation involves rotating, flipping, cropping, adding noise, and color dithering the cut images; then, constructing and adapting a deep neural network; training the deep neural network using optimization strategies and loss functions; and finally, deploying the trained deep neural network in actual operational systems to achieve object classification in remote sensing images. Deep neural networks are powerful frameworks that can directly learn image representations from massive amounts of image data. Compared to traditional machine learning-based remote sensing image object classification methods, deep learning-based remote sensing image object classification methods apply deep neural networks to remote sensing images containing a large amount of unknown information, improving the accuracy of object classification in remote sensing images. However, deep learning-based remote sensing image classification methods focus on single spatial domain features, resulting in insufficient deep neural networks for object classification in remote sensing images.

[0033] Remote sensing image classification methods based on deep learning have the following main shortcomings: First, they are sensitive to changes in illumination and scale, resulting in reduced classification performance in complex scenes (such as those where the ground is obscured by clouds and fog, where the spectral characteristics of different types of objects vary slightly, and where the spectral characteristics of the same type of objects vary greatly). Classification stability is particularly poor in scenes with cloud and fog obscuration. Second, remote sensing images contain complex backgrounds and noise. Deep neural networks that focus on single spatial domain features ignore high-frequency details and have difficulty distinguishing foreground objects from background information in high-noise images. The model's classification performance degrades in the face of background interference, resulting in insufficient robustness of deep neural networks for object classification in remote sensing images across different scenarios. Third, if deep neural networks focus on single frequency domain features, they struggle to retain precise spatial location information, resulting in blurred boundaries between different types of objects. Consequently, focusing on a single feature domain reduces the classification capabilities of deep neural networks, including decreased stability, robustness, and accuracy.

[0034] In view of this, the embodiments of the present application propose a method, apparatus, device, and product for classifying objects in remote sensing maps. This method transforms remote sensing images into frequency domain images through spatial-frequency domain transformations. Feature extraction is performed on the remote sensing images and frequency domain images, respectively, to obtain spatial-domain feature maps and frequency-domain feature maps. The spatial-domain feature maps and frequency-domain feature maps are then fused, and object classification of the remote sensing images is performed based on the fused features. This method thereby enhances the complementarity of image features across the spatial and frequency domains, addresses the issue of insufficient object classification capabilities for remote sensing images based on a single feature domain, and effectively improves the stability, robustness, and accuracy of object classification for remote sensing images. In particular, it improves the accuracy of object classification for remote sensing images (or complex remote sensing images) in complex scenes.

[0035] Exemplary Implementation Environment

[0036] Please refer to Figure 1 , Figure 1 The figure is a schematic diagram of an implementation environment according to an embodiment of the present application. The implementation environment includes a processing device 110. The method for classifying features in remote sensing maps provided in an embodiment of the present application can be used on the processing device 110 to classify features in an input remote sensing map. Optionally, a classification model can be deployed on the processing device 110 to classify features in the input remote sensing map.

[0037] The processing device 110 may be a terminal device (such as a mobile phone, a computer, a smart wearable device, a smart pen, a vehicle-mounted terminal, etc.) or a server. Figure 1 Take the processing device 110 as an example, which is a server.

[0038] Exemplary Methods

[0039] See also Figure 2 In an exemplary embodiment, a method for classifying objects in a remote sensing map is provided. The method for classifying objects in a remote sensing map comprises the following steps:

[0040] S201, acquiring a target remote sensing image.

[0041] In this embodiment, the target remote sensing image refers to the remote sensing image currently being classified. The target remote sensing image can be obtained from a sensor (a multispectral sensor or a hyperspectral sensor); or it can be obtained from another device (such as a user's terminal device); or it can be obtained from a task list.

[0042] S202: transform the target remote sensing image from the spatial domain to the frequency domain to obtain a frequency domain image.

[0043] In this embodiment, the target remote sensing image is a spatial domain image. A signal transformation method can be used to transform the target remote sensing image from the spatial domain to the frequency domain to obtain a frequency domain image of the target remote sensing image.

[0044] In one example, Fourier transform can be used to transform the target remote sensing image from the spatial domain to the frequency domain, that is, to convert the spatial domain signal into the frequency domain signal to obtain a frequency domain image.

[0045] Furthermore, the fast Fourier transform (FFT) is used to transform the target remote sensing image from the spatial domain to the frequency domain to obtain a frequency domain image, so as to improve the efficiency of transforming the target remote sensing image from the spatial domain to the frequency domain and save computing resources in the transformation process.

[0046] S203: Extract features from the target remote sensing image to obtain a spatial domain feature map.

[0047] Among them, the spatial domain feature map refers to the features extracted in the pixel space of the target remote sensing image, focusing on information such as the grayscale value of the pixel point and the position of the pixel point.

[0048] In this embodiment, feature extraction may be performed on the target remote sensing image by using an image feature extraction method to obtain image features of the target remote sensing image, namely, a spatial domain feature map.

[0049] S204: Extract features from the frequency domain image to obtain a frequency domain feature map.

[0050] Among them, the frequency domain feature map refers to the features extracted from the frequency components of the frequency domain image, and focuses on information such as the amplitude, phase, and distribution of the frequency components.

[0051] In this embodiment, feature extraction may be performed on the frequency domain image by using an image feature extraction method to obtain image features of the frequency domain image, namely, a frequency domain feature map.

[0052] For the classification of objects in remote sensing images, frequency domain feature maps have the following advantages: on the one hand, in the frequency domain feature map, the frequency components corresponding to the image foreground and the frequency components corresponding to the image background are significantly different. In remote sensing images with low contrast or high noise, the frequency domain feature map can still clearly show the true image features. Therefore, with the help of frequency domain images, the foreground objects can be separated from the background interference, and the recognition ability of object areas in remote sensing images with complex backgrounds (in remote sensing images with complex backgrounds, the object image area may be obscured by background information), remote sensing images with low contrast, and remote sensing images with high noise can be improved. On the other hand, for frequency domain images with changing illumination or scale, the frequency domain features are invariant. For example, the frequency domain energy distribution of vegetation in different seasons is highly consistent. Therefore, frequency domain images can improve the stability of remote sensing image object classification. On the other hand, objects with similar spectral characteristics (such as asphalt pavement and shaded water surface) have different corresponding phase characteristics. Therefore, based on frequency domain feature maps, the classification ability of object types with small inter-class differences can be improved, and the classification accuracy in scenes with small inter-class differences can be improved.

[0053] S205: Fusing the spatial domain feature map and the frequency domain feature map to obtain a fused feature map.

[0054] In this embodiment, a multimodal feature map fusion method is adopted to fuse the spatial domain feature map and the frequency domain feature map to obtain a fused feature map.

[0055] In one example, the spatial domain feature map and the frequency domain feature map may be concatenated to obtain a fused feature map; or the spatial feature map and the frequency domain feature map may be weighted to obtain a fused feature map.

[0056] In another example, a dynamic feature fusion mechanism (such as an attention mechanism) can be used to dynamically fuse the spatial domain feature map and the frequency domain feature map to obtain a fused feature map, so as to improve the fusion effect of the multimodal feature map and thereby improve the accuracy of ground object classification in the target remote sensing image.

[0057] S206: Classify the fused feature map to obtain a ground object classification result of the target remote sensing image.

[0058] In this embodiment, the fused feature map is used as the final feature information of the target remote sensing image. By classifying the fused feature map, the object classification result of the target remote sensing image is obtained. The object classification result may include the object area in the target remote sensing image and the object type corresponding to the object area.

[0059] In the embodiment of the present application, remote sensing image ground object classification is performed based on the fusion characteristics of the spatial domain feature map and the frequency domain feature map. The spatial domain feature map reflects the visual characteristics and spatial position characteristics of the ground objects in the remote sensing image. There is a significant difference between the frequency components of the foreground of the remote sensing image and the frequency components of the background of the remote sensing image in the frequency domain feature map, which realizes the complementary enhancement of the spatial domain feature map and the frequency domain feature map, provides more comprehensive remote sensing image features for the ground object classification of remote sensing images, improves the remote sensing image ground object classification capability, and especially improves the ground object classification accuracy of remote sensing images in complex scenes.

[0060] In some embodiments, the classification of objects in the target remote sensing image can be achieved through a classification model. Corresponding embodiments will be given below in conjunction with the accompanying drawings.

[0061] See also Figure 3 In another exemplary embodiment, a method for classifying objects in a remote sensing map is provided. The method for classifying objects in a remote sensing map comprises the following steps:

[0062] S301, acquiring a target remote sensing image.

[0063] S302: transform the target remote sensing image from the spatial domain to the frequency domain to obtain a frequency domain image.

[0064] The implementation principles and technical effects of S301 to S302 may refer to the aforementioned embodiments and will not be described in detail.

[0065] In one possible implementation, S302 may include: performing Fourier transform on the target remote sensing image to obtain a spectrum map; centering and / or enhancing the spectrum map to obtain a preprocessed spectrum map; and copying the preprocessed spectrum map multiple times to obtain a frequency domain image.

[0066] In this implementation, a Fourier transform is performed on the target remote sensing image, converting it from the spatial domain to the frequency domain to produce a spectrogram. In the spectrogram, low-frequency components correspond to the slowly varying portions of the target remote sensing image and contain most of the image's useful information, such as the general shape and spectral characteristics of the features. High-frequency components correspond to rapidly varying portions of the image, such as feature edges, image details, and image noise. Low-frequency components are typically distributed at the edges of the spectrogram (e.g., the left side). To extract more useful information, the spectrogram can be centered, moving the low-frequency portion of the spectrogram to its center. The spectrogram can also be enhanced, particularly by enhancing high-frequency features associated with feature edges and image details, to obtain richer high-frequency features. After performing the centering or enhancement preprocessing operation, the preprocessed spectrogram can be replicated multiple times to increase the number of image channels, ensuring that the frequency domain image meets the input image channel requirements of the classification model.

[0067] Optionally, the formula for Fourier transform of the target remote sensing image is expressed as:

[0068] ;

[0069] Among them, M and N represent the number of rows and columns of pixels in the target remote sensing image, respectively. Represents the pixel value at the image coordinate (x, y) in the target remote sensing image, Represents the signal metric value (which may include the amplitude and phase of the signal vibration) at the image coordinate (u,v) in the spectrum graph, is an imaginary unit, =-1, is a negative exponential basis, which represents orthogonal basis functions of different frequencies and is used to decompose spatial domain signals.

[0070] Optionally, the formula for centering the spectrum graph is expressed as:

[0071] ;

[0072] in, Represents the centered spectrum graph.

[0073] Optionally, after the spectrum graph is centralized to obtain a centralized spectrum graph, the centralized spectrum graph is enhanced. The formula for enhancing the centralized spectrum graph is expressed as:

[0074] ;

[0075] in, Indicates taking the absolute value, This represents the enhanced spectrogram. By using a logarithmic transformation, the signal metric values ​​of the spectrogram are adjusted to a range more suitable for processing and observation. This emphasizes frequency components with small and meaningful signal metric values ​​and enhances the contrast between different frequency components.

[0076] Optionally, the input image of the classification model is a three-channel image, and the target remote sensing image is a three-channel image; the preprocessed spectrum graph is copied multiple times, including: copying the preprocessed spectrum graph three times to obtain a frequency domain image with three channels, so that the number of channels of the frequency domain image is consistent with the number of channels of the target remote sensing image, meeting the classification model's requirements for the number of channels of the input image.

[0077] S303: Input the target remote sensing image and the frequency domain image into the classification model.

[0078] In this embodiment, the target remote sensing image and the frequency domain image may be preprocessed respectively, and the preprocessed target remote sensing image and the preprocessed frequency domain image may be input into the classification model respectively.

[0079] In one possible implementation, the target remote sensing image and the frequency domain image are preprocessed separately, including normalizing the target remote sensing image and the frequency domain image. This produces a normalized target remote sensing image and a normalized frequency domain image. The normalized target remote sensing image and the normalized frequency domain image can then be input into a classification model.

[0080] In this implementation, the formula for normalizing the target remote sensing image can be expressed as:

[0081] ;

[0082] in, represents a target remote sensing image, and the image size of the target remote sensing image is, for example, 256×256×3. represents the normalized target remote sensing image, represents the mean value of the pixel values ​​in the target remote sensing image, Represents the standard deviation of pixel values ​​in the target remote sensing image.

[0083] The formula for normalizing the frequency domain image can refer to the formula for normalizing the target remote sensing image, and will not be described in detail.

[0084] S304: extract features from the target remote sensing image through the first feature extraction network in the classification model to obtain a spatial domain feature map.

[0085] Among them, the classification model can adopt a deep neural network, which includes a feature extraction network for remote sensing images, a feature extraction network for frequency domain images, a feature fusion network for fusing spatial domain feature maps with frequency domain feature maps, and a classification network for land object classification.

[0086] In the classification model, the feature extraction network for remote sensing images and the feature extraction network for frequency domain images can be different feature extraction networks. This allows for the design and training of corresponding feature extraction networks based on the differences between spatial and frequency domain images, thereby improving the accuracy of spatial and frequency domain feature extraction. For ease of distinction, the feature extraction network for remote sensing images is referred to as the first feature extraction network, and the feature extraction network for frequency domain images is referred to as the second feature extraction network.

[0087] In this embodiment, the first feature extraction network includes multiple network layers. After the target remote sensing image is input into the classification model, the target remote sensing image is feature extracted through the multiple network layers in the first feature extraction network to obtain a spatial domain feature map.

[0088] In one possible implementation, the first feature extraction network is a residual network. During the feature extraction process of the target remote sensing image, the residual network can retain the detailed information of the target remote sensing image, has a strong image detail detection capability, and can more accurately capture the local texture features and local shape features of the target remote sensing image in the spatial domain.

[0089] Among them, the residual network includes a convolution module and multiple series-connected residual blocks.

[0090] Based on the first feature extraction network being a residual network, and the residual network including a convolution module and multiple serially connected residual blocks, S304 may include: S3041, performing feature extraction on the target remote sensing image using the convolution module to obtain a convolution feature map of the target remote sensing image; and S3042, performing multiple downsampling on the convolution feature map using the multiple serially connected residual blocks to obtain a spatial domain feature map. Thus, preliminary feature extraction is first performed using the convolution module, and then multiple downsampling is performed using the multiple serially connected residual blocks to extract more in-depth local detail features, resulting in a rich and accurate spatial domain feature map.

[0091] In S3041, the convolution module may include a convolution layer, and features of the target remote sensing image may be extracted through the convolution layer in the convolution module to obtain a convolution feature map of the target remote sensing image.

[0092] Optionally, the convolution module may also include a batch normalization (BN) layer and an activation function. In the convolution module, the target remote sensing image is convolved with the convolution layer to obtain a feature map output by the convolution layer. The feature map output by the convolution layer is input to the BN layer, where it is normalized to obtain a feature map output by the BN layer. The feature map output by the BN layer is input to the activation function, where it is processed to obtain a convolution feature map of the target remote sensing image.

[0093] Introducing BN layers and activation functions outside the convolutional layer can bring many advantages. For example, the BN layer can improve the convergence speed of the classification model during training, prevent gradient problems (gradient explosion, gradient disappearance), and prevent overfitting. For example, the activation function can introduce nonlinear relationships, thereby enhancing the classification model's ability to learn the spatial domain features of the target remote sensing image.

[0094] For example, the convolution processing of the target remote sensing image through the convolution layer can be expressed as:

[0095] ;

[0096] in, Represents the feature map output by the convolutional layer, represents the convolution kernel parameters in the convolution layer, Represents the bias parameters in the convolutional layer, and Conv represents the convolutional layer.

[0097] For example, the normalization of the feature map through the BN layer can be expressed as:

[0098] ;

[0099] in, Represents the feature map output by the BN layer, and BN represents the BN layer.

[0100] Furthermore, the activation function uses the rectified linear unit (ReLU) activation function, and the feature map output by the BN layer is processed by the ReLU activation function.

[0101] For example, the feature map output by the BN layer is processed by the ReLU activation function and can be expressed as:

[0102] ;

[0103] The ReLU activation function can be expressed as:

[0104] ;

[0105] Where a is a constant, for example, a is 0. In the ReLU activation function, for x with a value greater than a, the value of x is output, and for x with a value less than or equal to a, a is output. Represents the feature map output by the ReLU activation function, that is, the convolution feature map output by the convolution module.

[0106] For example, in the convolution module, the convolution kernel size of the convolution layer is 7×7, the stride used by the convolution layer (that is, the stride size of the convolution kernel sliding on the target remote sensing image) is 2, and the padding value used by the convolution layer is 3. After the convolution module is used to extract features of the target remote sensing image with a scale of 256×256×3, a convolution feature map with a scale of 128×128×64 can be obtained.

[0107] In S3042, in the plurality of residual blocks connected in series, each residual block has the same structure. The convolution feature map may be input into the first residual block in the plurality of residual blocks connected in series. Starting from the first residual block, the convolution feature map may be downsampled. The convolution feature map may be downsampled multiple times through the plurality of residual blocks connected in series to obtain a spatial domain feature map.

[0108] Optionally, the residual network also includes a max pooling layer, which is located between the convolution module and the first residual block in the series of multiple residual blocks. After obtaining the convolution feature map, the convolution feature map can be input into the max pooling layer for max pooling operation to obtain the feature map output by the max pooling layer, thereby reducing the spatial size of the feature map. The feature map output by the max pooling layer is multiply downsampled through multiple series of residual blocks to obtain the spatial domain feature map.

[0109] For example, the pooling operation of the maximum pooling layer can be expressed as:

[0110] ;

[0111] in, Represents the feature map output by the maximum pooling layer, and MaxPool represents the maximum pooling layer.

[0112] For example, the size of the maximum pooling layer is 3×3, the stride used by the maximum pooling layer is 2, and the padding value used is 1. After the maximum pooling operation is performed on the convolutional feature map of 128×128×64 scale through the maximum pooling layer, the feature map of 64×64×64 scale output by the maximum pooling layer is obtained.

[0113] Optionally, performing multiple downsampling of the convolution feature map through multiple series-connected residual blocks may include: in the i-th residual block, first performing feature extraction on the feature map of the i-th residual block through the first convolution layer in the i-th residual block to obtain the feature map output by the first convolution layer, where i is greater than or equal to 1; inputting the feature map output by the first convolution layer into the first BN layer in the i-th residual block, normalizing the feature map input to the first BN layer through the first BN layer, and obtaining the feature map output by the first BN layer; inputting the feature map output by the first BN layer into the first activation function in the i-th residual block to obtain the feature map output by the first activation function; and inputting the feature map output by the first activation function into the first activation function. The feature map output by the activation function is input to the second convolutional layer in the i-th residual block. The feature map input to the second convolutional layer is extracted by the second convolutional layer to obtain the feature map output by the second convolutional layer. The feature map output by the second convolutional layer is input to the second BN layer in the i-th residual block. The feature map input to the second BN layer is normalized by the second BN layer to obtain the feature map output by the second BN layer. The feature map input to the i-th residual block and the feature map output by the second BN layer are input to the second activation function in the i-th residual block to obtain the feature map output by the second activation function. The feature map output by the second activation function is the feature map output by the i-th residual block. Thus, through multiple convolutional layers, multiple BN layers, and multiple activation functions, multiple sampling of the feature map is achieved to extract more and more accurate local detail features in the target remote sensing image.

[0114] For example, taking the last residual block as an example, its processing flow can be expressed as:

[0115] ;

[0116] ;

[0117] ;

[0118] Among them, Conv1, BN1, ReLU1, Conv2, BN2, and ReLU2 represent the first convolutional layer, the first BN layer, the first activation function, the second convolutional layer, the second BN layer, and the second activation function in the last residual block, respectively. Represents the feature map output by the first activation function, Represents the feature map output by the second BN layer, Represents the feature map output by the last residual block, which is the spatial domain feature map.

[0119] For example, the residual network includes 16 serial residual blocks, each residual block contains two 3×3 convolutional layers, which downsample the input feature map by 8 times, and finally obtain a spatial domain feature map of 3×3×512 scale.

[0120] S305 , performing feature extraction on the frequency domain image through the second feature extraction network in the classification model to obtain a frequency domain feature map.

[0121] In this embodiment, the second feature extraction network includes multiple network layers. After the frequency domain image is input into the classification model, the multiple network layers in the second feature extraction network perform feature extraction on the frequency domain image to generate a frequency domain feature map. Based on the frequency domain feature map, the classification model's ability to identify object regions in remote sensing images with complex backgrounds, low-contrast and high-noise images, and images with little inter-object differentiation can be improved.

[0122] In one possible implementation, the second feature extraction network is a two-layer routing attention network, which includes a convolution module and an attention module. The attention mechanism of the attention module adopts a two-layer routing attention (Bi-Level Routing Attention, BRA) mechanism.

[0123] The Bi-Level Routing Attention Network refers to a Vision Transformer with Bi-Level Routing Attention (BiFormer) network. During feature extraction of frequency domain images using the Bi-Level Routing Attention Network, the Bi-Level Routing Attention mechanism can be used to model long-range spatial dependencies between frequency domain features (especially between global and local frequency domain features). This solves the problem that deep neural networks have difficulty modeling long-range spatial dependencies between features. Compared with traditional attention mechanisms, the Bi-Level Routing Attention mechanism has higher computational efficiency and lower computational complexity, solving the problem of high computational complexity leading to difficulty in training deep neural networks after the introduction of the attention mechanism.

[0124] Based on the second feature extraction network being a two-layer routing attention network, which includes a convolution module and an attention module utilizing a two-layer routing attention mechanism, S305 may include: S3051, performing feature extraction on the frequency domain image via the convolution module to obtain a convolution feature map of the frequency domain image; S3052, performing cross-region feature interaction on the convolution feature map via the attention module using a two-layer routing attention mechanism to obtain a first attention feature map; and S3053, obtaining a frequency domain feature map based on the first attention feature map. Thus, during the frequency domain feature extraction process, the two-layer routing attention mechanism within the two-layer routing attention network is utilized to establish long-range spatial dependencies between frequency domain features, extracting more accurate frequency domain features (such as periodic texture features of large-scale farmland). Furthermore, the attention mechanism is used to focus on key features contained in the high-frequency components of the frequency domain image (such as road edge features contained in the high-frequency components).

[0125] In S3051, the convolution module may include a convolution layer, and the convolution layer in the convolution module may be used to perform feature extraction on the frequency domain image to obtain a convolution feature map of the frequency domain image.

[0126] Optionally, the convolution module includes a convolution layer and a feature splicing layer, and the position coding matrix in the feature splicing layer is a learning parameter in the training process. S3051 includes: dividing the frequency domain image into blocks to obtain multiple non-overlapping image blocks; performing feature extraction on the multiple non-overlapping image blocks through the convolution layer to obtain feature maps corresponding to the multiple non-overlapping image blocks; in the feature splicing layer, sorting the feature maps corresponding to the multiple non-overlapping image blocks according to the position order of the multiple non-overlapping image blocks, and adding position information to the sorted feature maps according to the position coding matrix to obtain a convolution feature map.

[0127] In this optional method, the frequency domain image can be divided into multiple non-overlapping image blocks by image segmentation; the multiple non-overlapping image blocks are respectively input into the convolution layer, and the multiple non-overlapping image blocks can be encoded into embedded vectors by linear projection in the convolution layer to obtain feature maps corresponding to the multiple non-overlapping image blocks. Thus, by segmenting the frequency domain image and extracting features from the segmented image blocks, the local features of the frequency domain image can be accurately extracted. Then, in the feature splicing layer, the feature maps corresponding to the multiple non-overlapping image blocks can be expanded in the spatial dimension according to the position order of the multiple non-overlapping image blocks, so as to achieve spatial sorting of the feature maps corresponding to the multiple non-overlapping image blocks, so that the feature maps corresponding to the multiple non-overlapping image blocks are expanded into an image sequence, and the image sequence forms a feature map; the position coding matrix contains the position coding corresponding to the multiple non-overlapping image blocks, and the position coding is added to the sorted feature map according to the position coding matrix to enhance the perception of the second feature extraction network on the position information, solving the problem that the deep neural network is difficult to retain accurate spatial position information when focusing on frequency domain features.

[0128] During the training process of the classification model, the position encoding matrix in the feature concatenation layer is adjusted to improve the accuracy of the position encoding matrix so that the second feature extraction network can perceive accurate position information.

[0129] For example, the processing of frequency domain images in the convolution module can be expressed as:

[0130] ;

[0131] ;

[0132] Among them, Conv represents the convolution layer, Expressed as a frequency domain image, in this formula, we can first Divide into multiple non-overlapping image blocks; Represents the feature maps corresponding to multiple non-overlapping image blocks; Flatten() represents the sequence formed according to the spatial dimension. represents the position encoding matrix, Represents the convolution feature map obtained after the frequency domain image is processed by the convolution layer and the feature splicing layer.

[0133] For example, the size of the frequency domain image is 256×256×3. The frequency domain image is divided into non-overlapping blocks of 4×4 size, and 64×64 non-overlapping blocks are obtained. These non-overlapping blocks are converted into embedding vectors by linear projection through a convolution layer with a convolution kernel size of 4×4, and the feature maps corresponding to these non-overlapping blocks are obtained. Then, these feature maps are expanded into sequences in the spatial dimension and the corresponding position codes are added to obtain a convolution feature map of size 64×64×64.

[0134] In S3052, in the attention module, the query vector, key vector and value vector of the local features in the convolution feature map of the frequency domain image can be determined based on the double-layer routing attention mechanism, and feature processing can be performed based on the query vector, key vector and value vector of the local features to realize cross-regional feature interaction based on the double-layer routing attention mechanism and obtain the first attention feature map.

[0135] Optionally, the attention module includes an attention layer, and S3052 includes: performing window division on the convolution feature map of the frequency domain image to obtain multiple feature windows; determining a double-layer routing between the multiple feature windows based on the feature similarity between each of the multiple feature windows, the double-layer routing indicating the associated feature windows corresponding to the multiple feature windows in the dimension of feature similarity; determining the query vectors corresponding to the multiple feature windows, the key vectors corresponding to the multiple feature windows, and the value vectors corresponding to the multiple feature windows based on the double-layer routing and the feature representations corresponding to the multiple feature windows on the convolution feature map; in the attention layer, determining the attention weight matrix based on the query vectors corresponding to the multiple feature windows, the key vectors corresponding to the multiple feature windows, the value vectors corresponding to the multiple feature windows, and the weight parameters of the attention layer, adding the attention weight matrix to the convolution feature map to obtain a first attention feature map. As can be seen, the two-layer routing attention mechanism uses local feature windows and their attention vectors to focus on and capture high-frequency details (such as road cracks and building outlines), avoiding redundant global computation. By determining the correlation between feature windows based on their feature similarity, global sparse attention is established, focusing on key areas within low-frequency components (such as large areas of farmland and water). This enables the calculation of cross-regional dependencies between local frequency-domain features and reduces computational complexity, enabling the second feature extraction network to focus on the boundaries and outlines of objects, improving the accuracy, robustness, and stability of the classification model for object classification in remote sensing images in complex scenes.

[0136] In this optional method, after dividing to obtain multiple feature windows, the similarity between the feature representations corresponding to the multiple feature windows on the convolution feature map can be calculated to obtain the feature similarity between the multiple feature windows; for the first feature window, the first K feature windows can be selected from the second feature window in descending order of the feature similarity between the first feature window and the second feature window as the associated feature windows corresponding to the first feature window, the first feature window is any feature window in the multiple feature windows, and the second feature window is the remaining feature windows in the multiple feature windows except the first feature window. In this way, a two-layer routing between the multiple feature windows is obtained. Afterwards, for each feature window, the query vector, key vector and value vector corresponding to the feature window can be calculated based on the features covered by the feature window on the convolution feature map and the features covered by the associated feature window of the feature window; in this way, the query vectors corresponding to the multiple feature windows, the key vectors corresponding to the multiple feature windows, and the value vectors corresponding to the multiple feature windows are obtained. Finally, according to the query vectors corresponding to the multiple feature windows, the key vectors corresponding to the multiple feature windows, the value vectors corresponding to the multiple feature windows and the weight parameters of the attention layer, the attention weight matrix is ​​determined, and the attention weight matrix is ​​added to the convolution feature map to obtain the first attention feature map.

[0137] Furthermore, the attention module also includes layer normalization (LN) and a feedforward neural network. After obtaining the first attention feature map, it also includes: preliminary processing of the first attention feature map through layer normalization; and enhancement of the initially processed attention feature map through a feedforward neural network (FFN, composed of fully connected layers). Thus, layer normalization alleviates the gradient explosion problem caused by the dot product operation in the attention mechanism, and the feedforward neural network optimizes the feature representation of the attention feature map.

[0138] For example, the above process can be expressed as:

[0139] ;

[0140] ;

[0141] ;

[0142] ;

[0143] ;

[0144] in, and They represent the feature representation of the i-th feature window and the j-th feature window respectively, Sim() represents the calculation of the feature similarity between feature windows, express and The feature similarity between them; TopK() means selecting the feature windows with the highest feature similarity among the first k features. Indicates the associated feature window corresponding to the i-th feature window; 、 、 They represent the query vector, key vector, and value vector corresponding to the i-th feature window, respectively. Softmax represents the normalized exponential function, and d represents the dimension of the query vector. represents the attention weight matrix of the i-th feature window; LN represents layer normalization, represents the first attention feature map after layer normalization, Represents the first attention feature map after enhancement by the feedforward neural network.

[0145] For example, the convolutional feature map of size 64×64×64 is divided into feature windows of size 8×8, and 64 feature windows are obtained. The above attention operation is performed on the 64 feature windows respectively, and finally the first attention feature map of size 64×64×64 is obtained.

[0146] In S3053, the first attention feature map can be determined as a frequency domain feature map, or the first attention feature map can be further processed to obtain a frequency domain feature map. In particular, the first attention feature map can be subjected to cross-regional feature interaction based on a double-layer routing attention mechanism to obtain a second attention feature map, and the second attention feature map can be subjected to cross-regional feature interaction based on a double-layer routing attention mechanism to obtain a third attention feature map. Through repeated attention operations, a deep cross-regional feature interaction is established to improve the extraction effect of frequency domain features.

[0147] Optionally, S3053 includes: dimensionality adjustment of the first attention feature map; performing cross-regional feature interaction based on a double-layer routing attention mechanism on the dimensionally adjusted first attention feature map through the attention module to obtain a second attention feature map; and obtaining a frequency domain feature map based on the second attention feature map. Obtaining a frequency domain feature map based on the second attention feature map may include: determining the second attention feature map as a frequency domain feature map; or dimensionality adjustment of the second attention; performing cross-regional feature interaction based on a double-layer routing attention mechanism on the dimensionally adjusted second attention feature map through the attention module to obtain a third attention feature map, and dimensionality adjustment of the third attention feature map to obtain a frequency domain feature map. In this way, through repeated dimensionality adjustment and attention operations, deep cross-regional feature interaction is performed to improve the extraction effect of frequency domain features.

[0148] In this optional method, multiple feature blocks adjacent to each other on the same channel dimension in the first attention feature map can be merged, and the number of channels of the first attention feature map can be adjusted to achieve dimensional adjustment of the first attention feature map. The dimensional adjustment of the second attention feature map and the cross-region feature interaction of the second feature map can refer to the above description of the first attention feature map and will not be repeated here.

[0149] Optionally, the size of the frequency domain feature map is consistent with that of the spatial domain feature map, so as to facilitate fusion of the frequency domain feature map and the spatial domain feature map.

[0150] For example, for the first attention feature map of size 64×64×64 , merge adjacent 2×2 feature blocks on the same channel dimension so that the first attention feature map The size of the first attention feature map is changed to 32*32*64; then, the first attention feature map The number of channels is adjusted to 256, so that the first attention feature map The size of the first attention feature map is changed to 32*32*256; then the first attention feature map is transformed into The number of channels is projected to 128 to reduce the output of redundant information and obtain the final first attention feature map The size of the first attention feature map after dimension adjustment is 32*32*128. The cross-region feature interaction based on the double-layer routing attention mechanism is performed on the first attention feature map after dimension adjustment to obtain the second attention feature map , and then the second attention feature map By performing the dimension adjustment operation, we can obtain the second attention feature map with a size of 16*16*256 ; Second attention feature map after dimension adjustment Perform cross-region feature interaction based on the double-layer routing attention mechanism to obtain the third attention feature map , and then the third attention feature map By performing the dimension adjustment operation, a frequency domain feature map with a size of 8*8*512 can be obtained. .

[0151] S306: Fusing the spatial domain feature map and the frequency domain feature map through the feature fusion network in the classification model to obtain a fused feature map.

[0152] In this embodiment, if the frequency domain features and spatial features are simply spliced ​​together, it will be impossible to establish deep correlations between cross-domain features, and there will be a lack of targeted modeling of high-order semantics in the frequency domain. To solve this problem, a feature fusion network is used to fuse the spatial domain feature map and the frequency domain feature map. During the training process, the feature fusion network can dynamically learn the deep correlations between cross-domain features and high-order semantics in the frequency domain, realize dynamic fusion of cross-domain features, improve the feature fusion effect, and thereby improve the classification accuracy of ground object classification in remote sensing images based on fused features.

[0153] In one possible implementation, S306 includes: using a dynamic exponential moving average (EMA) in the feature fusion network to fuse the spatial domain feature map and the frequency domain feature map to obtain a fused feature map, where the smoothing factor in the EMA is related to the training parameters of the feature fusion network. During training, the training parameters of the feature fusion network can be dynamically adjusted. Changes in the weight parameters cause changes in the smoothing factor in the EMA, allowing the feature fusion network to dynamically adjust the feature fusion effect during training, thereby improving the cross-domain feature fusion effect of the feature fusion network in actual applications.

[0154] An optional, dynamic exponential moving average smoothing factor is given by:

[0155] ;

[0156] ;

[0157] in, represents the dynamic fusion coefficient, that is, the smoothing factor in the exponential sliding average; t represents the number of training steps of the classification model, represents the spatial domain feature map extracted during the training process, Represents the frequency domain feature map extracted during the training process; w and b are the training parameters (i.e., learning parameters) of the feature fusion network, where w is the weight parameter, which is used to control the adjustment rate of the feature fusion network during the training process, and b is the bias parameter, which is used to control the offset of the feature fusion network adjustment during the training process; Sigmoid represents the activation function, which is used to map the variable value of the input activation function to a value range of 0 to 1. It can be seen that during the training process, Changes with the number of training steps t (for example, in the initial training process, the frequency domain feature map is biased to achieve Dynamic update of the spatial domain feature map in the later training process), in the verification and testing phase of the model is a fixed value.

[0158] S307, classifying the fused feature map through the classification network in the classification model to obtain the ground object classification result of the target remote sensing image.

[0159] In this embodiment, the fused feature map is input into the classification network, and the fused feature map can be mapped to the category space in the classification network to obtain scores corresponding to multiple land object types. The scores corresponding to the multiple land object types are converted into probability values ​​corresponding to the multiple land object types. The land object classification results can be determined based on the probability values ​​corresponding to the multiple land object types. For example, the land object type with the largest probability value is determined to be the land object type corresponding to the remote sensing image.

[0160] Optionally, the classification network includes a global pooling layer, a fully connected layer, and a normalized exponential function. The global pooling layer can first be used to compress the spatial dimension of the fused feature map and retain the channel information of the fused feature map to obtain the feature map output by the global pooling layer; the fully connected layer can then be used to map the feature map output by the global pooling layer to the category space to obtain scores corresponding to multiple land feature types; and the normalized exponential function can then be used to convert the scores corresponding to the multiple land feature types into probabilities corresponding to the multiple land feature types. The formula for this process can be expressed as:

[0161] ;

[0162] ;

[0163] ;

[0164] ;

[0165] Among them, GlobalAvgPool represents the global pooling layer, represents the fused feature map, Represents the feature map output by the global pooling layer; W and b represent the training parameters of the fully connected layer, and Z represents the score output by the fully connected layer; Softmax represents the normalized exponential function, and P represents the probability distribution. In the probability distribution, the probabilities corresponding to multiple land feature types are respectively represented. Indicates the probability corresponding to the first feature type, Indicates the probability corresponding to the Cth feature type.

[0166] In the embodiments of the present application, a classification model is used to implement multimodal feature-based classification of remote sensing image objects. Specifically, the classification model improves the extraction of spatial domain features through a residual network, improves the extraction of frequency domain features through a two-layer routing attention network, and improves the fusion of spatial and frequency domain features through a dynamic exponential sliding average. This improves the accuracy, robustness, and stability of the classification model for remote sensing image object classification in many ways.

[0167] Figure 4 This is an example of classifying objects in remote sensing images using a classification model. Figure 4 As shown in the figure, the remote sensing image is input into the residual network, and the feature extraction of the remote sensing image is performed through the residual network to obtain the spatial domain feature map; the remote sensing image is subjected to FFT transformation to obtain the frequency domain image, and the frequency domain image is input into the double-layer routing attention network, and the feature extraction of the frequency domain image is performed through the double-layer routing attention network to obtain the frequency domain feature map; the spatial domain feature map and the frequency domain feature map are subjected to feature fusion based on exponential sliding average to obtain a fused feature map; finally, the fused feature map is input into the classification layer for classification to obtain the ground object type corresponding to the remote sensing image.

[0168] Below, an example of a training process for a classification model is provided.

[0169] In some embodiments, the training process of the classification model may include: obtaining a training data set, the training data set including training samples and sample labels of the training samples, the training samples are remote sensing images, and the sample labels of the training samples are real object classification results of the remote sensing images; transforming the training samples from the spatial domain to the frequency domain to obtain frequency domain images corresponding to the training samples; inputting the training samples and the frequency domain images corresponding to the training samples into the classification model; performing feature extraction on the training samples through the first feature extraction network in the classification model to obtain a spatial domain feature map; performing feature extraction on the frequency domain images corresponding to the training samples through the second feature extraction network in the classification model to obtain a frequency domain feature map; performing fusion of the spatial domain feature map and the frequency domain feature map through the feature fusion network in the classification model to obtain a fused feature map; classifying the fused feature map through the classification network in the classification model to obtain a predicted object classification result of the training sample, determining a loss value based on the predicted object classification result of the training sample and the sample label of the training sample (i.e., the real object classification result of the training sample); and adjusting the parameters of the classification model using an optimization algorithm based on the loss value.

[0170] In this embodiment, in the process of adjusting the parameters of the classification model using the optimization algorithm according to the loss value, the parameters of the classification model can be adjusted by back propagation. After multiple iterative training, the classification model tends to converge, and finally a classification model with better classification effect is obtained.

[0171] It should be noted that the classification process of the classification model for training samples can refer to the classification process of the classification model for target remote sensing images, which will not be described in detail here.

[0172] Optionally, a first dataset is obtained, and data augmentation is performed on the first dataset to obtain a second dataset; the second dataset is divided into a training dataset, a validation test set, and a test dataset. The first dataset is a public dataset and includes remote sensing images of multiple land feature types (e.g., agriculture, airplanes, baseball fields, beaches, buildings, and forests). Data augmentation of the first dataset may include performing one or more of rotation, translation, and inversion on the images in the first dataset to increase the richness of the data in the dataset.

[0173] Optionally, the optimization algorithm uses the Adaptive Moment Estimation (Adam) algorithm to improve optimization results. During training, the Adam algorithm's hyperparameters can be initialized, including the number of iterations and the batch size (i.e., the number of samples used in each training session). To fully train the classification model to learn multimodal features, a larger number of iterations can be set. These hyperparameters include the learning rate and the weight decay coefficient.

[0174] For example, the initial learning rate is 0.03, the weight decay coefficient is 0.0005, the batch size is 64, and the number of iterations is 10,000.

[0175] Optionally, the predicted feature classification results of the training samples include probability values ​​of the training samples belonging to multiple feature types, and the sample labels of the training samples indicate the feature types to which the training samples actually belong. Based on this, the loss function of the classification model is expressed as:

[0176] ;

[0177] in, Indicates the probability value of the training sample belonging to the i-th type of ground object in the predicted ground object classification result; if the sample label of the training sample indicates that the training sample actually belongs to the i-th type of ground object, then y i =1, otherwise y i = 0. C represents the number of feature types.

[0178] Exemplary devices

[0179] Correspondingly, an embodiment of the present application also provides a device for classifying land features in a remote sensing map.

[0180] See also Figure 5 In an exemplary embodiment, a device 500 for classifying objects in a remote sensing map is provided. The device 500 includes an acquisition unit 501, a spatial domain to frequency domain conversion unit 502, a spatial domain feature extraction unit 503, a frequency domain feature extraction unit 504, a feature fusion unit 505, and a object classification unit 506. Specifically:

[0181] The acquisition unit 501 is used to acquire the target remote sensing image; the spatial domain-frequency domain transformation unit 502 is used to transform the target remote sensing image from the spatial domain to the frequency domain to obtain a frequency domain image; the spatial domain feature extraction unit 503 is used to extract features from the target remote sensing image to obtain a spatial domain feature map; the frequency domain feature extraction unit 504 is used to extract features from the frequency domain image to obtain a frequency domain feature map; the feature fusion unit 505 is used to fuse the spatial domain feature map and the frequency domain feature map to obtain a fused feature map; the object classification unit 506 is used to classify the fused feature map to obtain the object classification result of the target remote sensing image.

[0182] In one possible implementation, the spatial domain feature extraction unit 503 is specifically used to: input the target remote sensing image and the frequency domain image into the classification model; perform feature extraction on the target remote sensing image through the first feature extraction network in the classification model to obtain a spatial domain feature map. The frequency domain feature extraction unit 504 is specifically used to: perform feature extraction on the frequency domain image through the second feature extraction network in the classification model to obtain a frequency domain feature map. The feature fusion unit 505 is specifically used to: fuse the spatial domain feature map and the frequency domain feature map through the feature fusion network in the classification model to obtain a fused feature map. The land object classification unit 506 is specifically used to: classify the fused feature map through the classification network in the classification model to obtain a land object classification result of the target remote sensing image.

[0183] In one possible implementation, the first feature extraction network is a residual network, which includes a convolution module and multiple series-connected residual blocks. The spatial domain feature extraction unit 503 is specifically used to: perform feature extraction on the target remote sensing image through the convolution module to obtain a convolution feature map of the target remote sensing image; and perform multiple downsampling on the convolution feature map through multiple series-connected residual blocks to obtain a spatial domain feature map.

[0184] In one possible implementation, the second feature extraction network is a two-layer routing attention network, which includes a convolution module and an attention module. The attention mechanism of the attention module adopts a two-layer routing attention mechanism. The frequency domain feature extraction unit 504 is specifically used to: perform feature extraction on the frequency domain image through the convolution module to obtain a convolution feature map of the frequency domain image; perform cross-region feature interaction based on the two-layer routing attention mechanism on the convolution feature map through the attention module to obtain a first attention feature map; and obtain a frequency domain feature map based on the first attention feature map.

[0185] In one possible implementation, the convolution module includes a convolution layer and a feature splicing layer. The position coding matrix in the feature splicing layer is a learning parameter in the training process. The frequency domain feature extraction unit 504 is specifically used to: divide the frequency domain image into blocks to obtain multiple non-overlapping image blocks; through the convolution layer, perform feature extraction on the multiple non-overlapping image blocks respectively to obtain feature maps corresponding to the multiple non-overlapping image blocks; in the feature splicing layer, sort the feature maps corresponding to the multiple non-overlapping image blocks according to the position order of the multiple non-overlapping image blocks, add position information to the sorted feature maps according to the position coding matrix, and obtain a convolution feature map.

[0186] In one possible implementation, the attention module includes an attention layer, and the frequency domain feature extraction unit 504 is specifically used to: perform window division on the convolution feature map to obtain multiple feature windows; determine the double-layer routing between the multiple feature windows based on the feature similarity between each of the multiple feature windows, and the double-layer routing indicates the associated feature windows corresponding to the multiple feature windows in the dimension of feature similarity; determine the query vectors corresponding to the multiple feature windows, the key vectors corresponding to the multiple feature windows, and the value vectors corresponding to the multiple feature windows based on the double-layer routing and the feature representations corresponding to the multiple feature windows on the convolution feature map; in the attention layer, determine the attention weight matrix based on the query vectors corresponding to the multiple feature windows, the key vectors corresponding to the multiple feature windows, the value vectors corresponding to the multiple feature windows, and the weight parameters of the attention layer, and add the attention weight matrix to the convolution feature map to obtain a first attention feature map.

[0187] In one possible implementation, the attention module also includes layer normalization and a feedforward neural network, and the frequency domain feature extraction unit 504 is also used to: perform preliminary processing on the first attention feature map through layer normalization; and enhance the attention feature map after preliminary processing through a feedforward neural network.

[0188] In one possible implementation, the frequency domain feature extraction unit 504 is specifically used to: perform dimension adjustment on the first attention feature map; perform cross-region feature interaction based on the double-layer routing attention mechanism on the dimension-adjusted first attention feature map through the attention module to obtain a second attention feature map; and obtain a frequency domain feature map based on the second attention feature map.

[0189] In one possible implementation, the feature fusion unit 505 is specifically used to: in the feature fusion network, use a dynamic exponential sliding average to fuse the spatial domain feature map and the frequency domain feature map to obtain a fused feature map, and the smoothing factor in the exponential sliding average is related to the training parameters of the feature fusion network.

[0190] The remote sensing map feature classification device 500 provided in this embodiment is based on the same concept as the remote sensing map feature classification method provided in the aforementioned embodiments of this application. It can execute the remote sensing map feature classification method provided in any of the aforementioned embodiments of this application and has the corresponding functional modules and beneficial effects of executing the remote sensing map feature classification method. For technical details not fully described in this embodiment, please refer to the specific processing content of the remote sensing map feature classification method provided in the aforementioned embodiments of this application and will not be repeated here.

[0191] The functions implemented by each unit in the above device can be implemented by the same or different processors, which is not limited in the embodiments of the present application.

[0192] It should be understood that the units in the above devices can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and the memory stores instructions. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device. The processor can be a general-purpose processor, such as a CPU or a microprocessor, and the memory can be a memory within the device or a memory outside the device. Alternatively, the units in the device can be implemented in the form of hardware circuits. The functions of some or all units can be realized by designing the hardware circuits. The hardware circuit can be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units can be realized by designing the logical relationships between the components within the circuit. For another example, in another implementation, the hardware circuit can be implemented by a PLD. For example, an FPGA can include a large number of logic gate circuits. The connection relationships between the logic gate circuits are configured through a configuration file to realize the functions of some or all of the above units. All units of the above devices can be implemented entirely in the form of a processor calling software, or entirely in the form of hardware circuits, or partially in the form of a processor calling software, with the remaining parts implemented in the form of hardware circuits.

[0193] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and execute instructions, such as a CPU, a microprocessor, a GPU, or a DSP. In another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit may be fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the configuration of the hardware circuit can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0194] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0195] In addition, the various units in the above apparatus may be fully or partially integrated together, or may be implemented independently. In one implementation, these units are integrated together and implemented in the form of a system-on-chip (SOC). The SOC may include at least one processor for implementing any of the above methods or implementing the functions of the various units of the apparatus. The at least one processor may be of different types, such as a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0196] Exemplary electronic devices

[0197] Another embodiment of the present application also provides an electronic device. Figure 6 As shown, the electronic device may include: a memory 600 and a processor 610; wherein the memory 600 is connected to the processor 610 for storing programs; the processor 610 is used to implement the method for classifying land objects in remote sensing maps disclosed in any of the above embodiments by running the program stored in the memory 600.

[0198] Specifically, the electronic device may further include: a bus, a communication interface 620 , an input device 630 and an output device 640 .

[0199] The processor 610, the memory 600, the communication interface 620, the input device 630 and the output device 640 are interconnected via a bus.

[0200] A bus may include a pathway that transfers information between components of a computer system.

[0201] Processor 610 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, or the like. It can also be an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component.

[0202] The processor 610 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0203] The memory 600 stores a program for executing the technical solution of the present application, and may also store an operating system and other key services. Specifically, the program may include program code, which includes computer operating instructions. More specifically, the memory 600 may include read-only memory (ROM), other types of static storage devices that can store static information and instructions, random access memory (RAM), other types of dynamic storage devices that can store information and instructions, disk storage, flash memory, and the like.

[0204] The input device 630 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0205] Output device 640 may include devices that allow information to be output to a user, such as a display screen, printer, speakers, etc.

[0206] The communication interface 620 may include any device such as a transceiver to communicate with other devices or communication networks, such as Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc.

[0207] The processor 610 executes the program stored in the memory 600 and calls other devices, which can be used to implement each step of any remote sensing map feature classification method provided in the above embodiments of the present application.

[0208] An embodiment of the present application also proposes a chip, which includes a processor and a data interface. The processor reads and runs the program stored in the memory through the data interface to execute any one of the remote sensing map feature classification methods provided in the above embodiments. The specific processing process and its beneficial effects can be found in the above-mentioned embodiment introduction of the remote sensing map feature classification method.

[0209] Exemplary computer program products and storage media

[0210] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method for classifying features in remote sensing maps according to various embodiments of the present application described in any of the above-mentioned embodiments of this specification.

[0211] The computer program product may be written in any combination of one or more programming languages ​​to implement the program code for performing the operations of the embodiments of the present application, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0212] In addition, an embodiment of the present application may also be a storage medium on which a computer program is stored, and the computer program is executed by a processor to execute the steps of the method for classifying features in remote sensing maps according to various embodiments of the present application described in any of the above embodiments of this specification.

[0213] For the sake of simplicity, the aforementioned method embodiments are described as a series of action combinations. However, those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0214] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similarities between the various embodiments can be referred to in conjunction with each other. For device embodiments, since they are generally similar to method embodiments, their description is relatively simple, and for relevant details, reference can be made to the description of the method embodiments.

[0215] The steps in the methods of each embodiment of the present application can be adjusted in sequence, merged, and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0216] The modules and sub-modules in the devices and terminals in the various embodiments of the present application can be merged, divided, and deleted according to actual needs.

[0217] In the several embodiments provided in this application, it should be understood that the disclosed terminals, devices, and methods can be implemented in other ways. For example, the terminal embodiments described above are merely illustrative. For example, the division of modules or submodules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or module, which can be electrical, mechanical or other forms.

[0218] The modules or submodules described as separate components may or may not be physically separate, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules may be selected to achieve the purpose of this embodiment according to actual needs.

[0219] In addition, each functional module or submodule in each embodiment of the present application may be integrated into a processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into a single module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or software functional modules or submodules.

[0220] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0221] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, software executed by a processor, or a combination of the two. The software may be stored in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0222] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0223] The above description of the disclosed embodiments will enable those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is to be construed in the widest manner consistent with the principles and novel features disclosed herein.

Claims

1. A method for classifying objects in remote sensing images, characterized in that: include: Acquire target remote sensing images; Transforming the target remote sensing image from the spatial domain to the frequency domain to obtain a frequency domain image; Performing feature extraction on the target remote sensing image through a convolution module in a first feature extraction network to obtain a convolution feature map of the target remote sensing image; Performing feature extraction on the convolution feature map of the target remote sensing image through a plurality of residual blocks connected in series in the first feature extraction network to obtain a spatial domain feature map; Performing feature extraction on the frequency domain image through a convolution module in a second feature extraction network to obtain a convolution feature map of the frequency domain image; performing, by the attention module in the second feature extraction network, cross-region feature interaction based on a two-layer routing attention mechanism on the convolutional feature map of the frequency domain image to model the spatial dependency relationship between the frequency domain features, thereby obtaining a first attention feature map; Obtaining a frequency domain feature map according to the first attention feature map; Fusing the spatial domain feature map and the frequency domain feature map to obtain a fused feature map; The fused feature map is classified to obtain a ground object classification result of the target remote sensing image.

2. The method for classifying objects in remote sensing images according to claim 1, wherein: The convolution module in the first feature extraction network includes a convolution layer and a feature splicing layer. The position encoding matrix in the feature splicing layer is a learning parameter in the training process. The convolution module in the first feature extraction network is used to extract features from the target remote sensing image to obtain a convolution feature map of the target remote sensing image, including: Dividing the frequency domain image into blocks to obtain a plurality of non-overlapping image blocks; Performing feature extraction on each of the plurality of non-overlapping image blocks through the convolution layer to obtain feature maps corresponding to each of the plurality of non-overlapping image blocks; In the feature splicing layer, the feature maps corresponding to the multiple non-overlapping image blocks are sorted according to the position order of the multiple non-overlapping image blocks, and position information is added to the sorted feature maps according to the position encoding matrix to obtain the convolution feature map of the target remote sensing image.

3. The method for classifying objects in remote sensing images according to claim 1, wherein: The attention module includes an attention layer. The attention module in the second feature extraction network performs cross-region feature interaction based on a double-layer routing attention mechanism on the convolution feature map of the frequency domain image to model the spatial dependency relationship between the frequency domain features, thereby obtaining a first attention feature map, including: Performing window division on the convolution feature map of the frequency domain image to obtain a plurality of feature windows; Determining, based on feature similarities between any two of the plurality of feature windows, a two-layer routing between any two of the plurality of feature windows, wherein the two-layer routing indicates associated feature windows corresponding to the plurality of feature windows in a dimension of feature similarity; Determining, based on the feature representations corresponding to the multiple feature windows respectively on the double-layer routing and the convolution feature map of the frequency domain image, query vectors corresponding to the multiple feature windows respectively, key vectors corresponding to the multiple feature windows respectively, and value vectors corresponding to the multiple feature windows respectively; In the attention layer, the attention weight matrix is ​​determined based on the query vectors corresponding to the multiple feature windows, the key vectors corresponding to the multiple feature windows, the value vectors corresponding to the multiple feature windows and the weight parameters of the attention layer, and the attention weight matrix is ​​added to the convolution feature map of the frequency domain image to obtain the first attention feature map.

4. The method for classifying objects in remote sensing images according to claim 3, wherein: The attention module further includes layer normalization and a feedforward neural network. After the attention module in the second feature extraction network performs cross-region feature interaction based on a double-layer routing attention mechanism on the convolutional feature map of the frequency domain image to model the spatial dependency relationship between frequency domain features, and obtains the first attention feature map, the method further includes: Performing preliminary processing on the first attention feature map by the layer normalization; The attention feature map after preliminary processing is enhanced through the feedforward neural network.

5. The method for classifying objects in remote sensing images according to claim 1, wherein: The step of obtaining the frequency domain feature map according to the first attention feature map includes: Performing dimension adjustment on the first attention feature map; Through the attention module, the first attention feature map after dimension adjustment is subjected to cross-region feature interaction based on the double-layer routing attention mechanism to obtain a second attention feature map; According to the second attention feature map, the frequency domain feature map is obtained.

6. The method for classifying objects in remote sensing images according to any one of claims 1 to 5, characterized in that: fusing the spatial domain feature map and the frequency domain feature map to obtain a fused feature map; And, classifying the fused feature map to obtain a ground object classification result of the target remote sensing image, including: The spatial domain feature map and the frequency domain feature map are fused through a feature fusion network to obtain the fused feature map; The fused feature map is classified through a classification network to obtain a ground object classification result of the target remote sensing image.

7. The method for classifying objects in remote sensing images according to claim 6, wherein: The step of fusing the spatial domain feature map and the frequency domain feature map through a feature fusion network to obtain the fused feature map includes: In the feature fusion network, a dynamic exponential sliding average is used to fuse the spatial domain feature map and the frequency domain feature map to obtain the fused feature map. The smoothing factor in the exponential sliding average is related to the training parameters of the feature fusion network.

8. An electronic device, characterized in that: including memory and processor; The memory is connected to the processor and is used to store programs; The processor is configured to implement the method for classifying objects in remote sensing images as claimed in any one of claims 1 to 7 by running the program in the memory.

9. A computer program product, characterized in that The method comprises a computer program, which, when executed by a processor, implements the method for classifying objects in remote sensing images as claimed in any one of claims 1 to 7.

Images