Multi-frequency gated chart LCZ classification method based on multi-view satellite image

Abstract

The invention relates to a multi-frequency gated chart LCZ classification method based on a multi-view satellite image, and belongs to the field of remote sensing image scene classification. The method comprises the steps that multi-angle features are added into an original multispectral image data set, a multi-angle image is obtained and input into a designed frequency domain gating module, image space features and image frequency domain features are extracted through parallel processing of a deformable gating branch and a frequency domain transformation branch, and the two features are fused through a fusion module; and inputting the fusion features into a graph convolution module, remodeling the fusion features into a graph structure, performing graph convolution coding, and finally obtaining an LCZ classification result. According to the method, a double-branch structure and graph convolution are designed for the first time and used for feature extraction of multi-view remote sensing images, a multi-angle stereoscopic feature extraction module is designed to fully capture three-dimensional features of ground features, and LCZ classification precision is greatly improved.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image classification and relates to a multi-frequency gated image LCZ classification method based on multi-view satellite images. Background Technology

[0002] Cities are the centers of human production and life, and with the development of urbanization, cities are expanding both horizontally and vertically. Urban land use layout and spatial structure have a significant impact on the urban environment, climate formation, and residents' quality of life. To further study urban thermal environment issues, Stewart and Oke proposed the Local Climate Zones (LCZ) land cover classification framework in 2012. LCZs subdivide the land surface into 10 built environment categories and 7 natural cover categories. Specifically, the built environment categories (LCZ 1-10) divide built-up areas into compact high-rise (LCZ 1), compact mid-rise (LCZ 2), compact low-rise (LCZ 3), open high-rise (LCZ 4), open mid-rise (LCZ 5), open low-rise (LCZ 6), simple low-rise (LCZ 7), large low-rise (LCZ 8), very open low-rise (LCZ 9), and heavy industrial areas (LCZ 10) based on building height, density, and materials. The natural cover categories (LCZ AG) are divided into densely wooded areas (LCZ A), sparsely wooded areas (LCZ B), shrubland areas (LCZ C), low vegetation areas (LCZ D), hardened surfaces (LCZ E), bare sandy areas (LCZ F), and water areas (LCZ G) based on natural surface features. The LCZs provide researchers with a detailed and standardized land cover classification scheme, which is helpful for further research on the impact of urban layout and spatial structure on the climate environment.

[0003] Large-scale LCZ mapping primarily relies on satellite remote sensing technology. Projects such as the World Cities Database and Access Tool (WUDAPT) provide users with a free platform for LCZ mapping. WUDAPT uses Landsat medium-resolution remote sensing data and performs supervised classification based on image spectral features and a random forest classifier to ultimately achieve large-scale LCZ mapping. However, due to the relatively simple data and models used in this approach, the accuracy of LCZ mapping remains low in complex urban environments.

[0004] In recent years, deep learning methods, especially convolutional neural networks (CNNs), have been widely used in Local Climate Zone (LCZ) mapping research due to their advantages in automatically learning and capturing remote sensing image information. For example, the MSMLA-Net method integrates multi-level attention modules on the basis of traditional CNNs, significantly improving spectral and spatial representation capabilities; LCZNet extracts spatial features through multi-scale filtered convolutional layers and integrates channel features by combining SE-Residual modules, achieving a classification accuracy of nearly 20% compared to the WUDAPT method in three economic regions of China. Nevertheless, existing methods still suffer from low network structure efficiency and insufficient depth when dealing with complex scene classification tasks such as LCZ. Specifically, the connections between modules in complex network designs often lack optimization, leading to low feature extraction efficiency; while simple networks struggle to fully exploit global and local neighborhood features, limiting further improvements in LCZ classification accuracy; in addition, existing studies often lack consideration of three-dimensional information, and accurate classification of built-up area types, such as high, middle, and low layers, remains a significant challenge. To address these network model bottlenecks, this patent proposes an innovative architecture. This architecture features two distinct branches, focusing on spatial and spectral feature extraction from remote sensing images to improve extraction efficiency. Simultaneously, a GCNConv-based module is introduced to enhance the extraction of local and hidden features, supporting the construction of deeper networks for more advanced feature learning. Addressing the issue of missing stereo information, this study, for the first time, designs a dual-branch flow structure and graph convolution for feature extraction from multi-view remote sensing images. A deformable gating module is also specifically designed for efficient multi-angle feature extraction, effectively solving the problem of insufficient feature extraction capabilities for specific image sources in existing networks. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a multi-frequency gated graph LCZ classification method based on multi-view satellite imagery. By designing an efficient dual-branch network structure and introducing the GCNConv module, it overcomes the limitations of CNN models in terms of depth and efficiency under single-view remote sensing images. Simultaneously, it innovatively utilizes multi-view imagery and its dedicated feature extraction module, significantly improving the network's ability to process complex LCZ tasks, enabling it to utilize multi-angle features more scientifically and efficiently. Furthermore, it incorporates a graph convolutional network (GCN) to overcome specific difficulties faced by CNNs in building category classification.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A multi-frequency gated map LCZ classification method based on multi-view satellite imagery, the method comprising:

[0008] The original image is acquired and input into the frequency domain gating module for processing.

[0009] The image is processed in parallel by the deformable gating branch and the frequency transform branch of the frequency domain gating module to obtain spatial and frequency domain features respectively.

[0010] The SE module of the frequency domain gated module fuses the output features of the deformable gated branch with the output features of the frequency domain transform branch to obtain the fused features;

[0011] The fused features are input into the graph convolution module, which reshapes the fused features into a graph structure.

[0012] Then, graph convolution encoding is performed on the nodes in the reshaped graph structure, and Chebyshev polynomials are used to fit the convolution kernel in the graph convolution process.

[0013] Global average pooling is used to classify the output of the graph convolution process to obtain the final classification result.

[0014] Furthermore, the frequency domain gating module FRDG is used to extract the channel spectral features and global features of the image, while adapting to the multi-angle features of the input image. The frequency domain gating module includes a deformable gating branch, a frequency domain transform branch, and an SE module. The deformable gating branch and the frequency domain transform branch process in parallel to extract their respective features. The extracted features are synchronously input into the SE module for fusion to obtain fused features.

[0015] Furthermore, in the deformable gating branch of the frequency domain gating module, the acquired image features F1 and the multi-angle features F2 of the dataset are concatenated dimensionally. First, downsampling is performed, and then deformable convolution is applied to the downsampled features. This includes obtaining the offset using deformable 2D convolution and adding the offset to the deformable pooling process, as shown below:

[0016] For each position p on the output feature map y 12 ,have:

[0017]

[0018] Where p n To enumerate the positions in R, x1 and x2 are the image features and multi-angle features, respectively, and Δp n p is the offset. n +Δp n Indicates the offset position of the current sampling point; offset Δp n Determined by bilinear interpolation as follows:

[0019]

[0020] Where p represents any position p0+p n +Δp n q represents the spatial location of all integers in the feature map x, and G(·,·) is bilinear interpolation;

[0021] In deformable pooling, adding the offset to space b, we have:

[0022]

[0023] The output is a k*k feature map y, where n is the number of pixels in b. Similar to the formula above, the decimal offset Δp is used. ij Add it to position b; finally, the deformable convolutional feature map y is obtained.

[0024] The obtained feature map y is input into the HorBlock module. The HorBlock module mainly uses recursive gated convolution gnConv to perform spatial long-distance dependency optimization. Let x∈R be the input feature, then the output y is expressed as:

[0025]

[0026] y = φ out (P1)=φ ott [f(q0)⊙P0]

[0027] Where, φ in and φ out It is a linear projection layer that performs channel blending, f(·) is a depthwise separable convolution, and C is the dimension that blends the image spectrum and multi-angle features;

[0028] Based on the first-order interaction of gnConv mentioned above, using φ in Get P0 and q k Then, multi-space interaction operations are performed through recursive gated convolution, where scaling is controlled by 1 / α; {f k} is a set of depthwise separable convolutions, {g k} is used to match dimensions, and finally the output q of the last recursive step is used. n Input to φ out Obtain the output result.

[0029] Furthermore, in the frequency domain transform branch of the domain gate module, feature map preprocessing is first performed on the image input Conv2D and GRLU layers;

[0030] Then, the processed image is subjected to two-dimensional discrete Fourier transform (2D-FFT), where the 2D-FFT alternately performs shifted one-dimensional discrete Fourier transform (DFT) in two dimensions. Based on the one-dimensional DFT, the two-dimensional DFT of the real input x[m,n] satisfies the conjugate symmetry property X[Mu,Nv]=X[u,v]. The process of the one-dimensional DFT is as follows:

[0031] Given a sequence of N complex numbers x[n], 0≤n≤N-1, a one-dimensional DFT can be transformed to the frequency domain using the following formula:

[0032]

[0033] Where j is the imaginary unit and W N =e -j(2π / N) Take the values ​​of X[k] at N consecutive points k = 0, 1, ..., N-1;

[0034] Recovering the original spatial domain through inverse DFT:

[0035]

[0036] The real and imaginary parts are separated on the original 2D FFT, and the nonlinear transformation is achieved by combining the advantages of CNN. Expanded using Euler's formula, it is as follows:

[0037]

[0038] The input is a sequence of N complex numbers x[n], and f() is a convolutional layer. Finally, two features are obtained: one is the low-frequency information of the real part cosine wave component, and the other is the high-frequency information of the imaginary part sine wave component.

[0039] Then, the two pieces of information are processed by Conv2D and GELU respectively, and then subjected to 2D IFFT transformation to obtain the frequency domain transformation output features.

[0040] Furthermore, the SE module in the frequency domain gating module is used to transform the output characteristics DG of the deformable gating branch. g Output characteristics FR of frequency domain transform branch s Feature fusion is performed, which is represented as:

[0041] F = f SE [DG g +FR s ]

[0042] Among them, f SE [·] indicates the fusion operation of the SE module.

[0043] Furthermore, in the graph convolution module, the fused features are reshaped into a graph structure G = (V, E, A), where V and E are the sequences of vertices and edges of the graph, respectively, and A ∈ R × N × N is the adjacency matrix of the graph; D is defined as a matrix of the same size as A, yielding the Laplacian matrix L:

[0044]

[0045] Where I is the identity matrix.

[0046] Furthermore, during the encoding process, convolutional kernel g is used. θ Encode the nodes, g θ =diag(θ),θ∈R N Graph convolution is calculated as follows:

[0047] x*g θ =Ug θ (Λ)U T x

[0048] Where Λ and U are the normalized Laplace matrix L. n The eigenvalues ​​of the eigenvector matrix; x is the input, and g is the eigenvalues ​​of the eigenvector matrix; θ It is a function of Λ; then Chebyshev polynomial T is used. k To fit g θ (Λ), as follows:

[0049]

[0050] in

[0051] Therefore, the migration formula for GCN is derived as follows:

[0052]

[0053] Among them, G l and W l They are l th The layer's output and learnable weights, where σ is the activation function;

[0054] After completing the graph convolution, a 1x1 convolution is used to enhance the information, resulting in the output of the graph convolution.

[0055] Furthermore, the output from the graph convolution module is subjected to global average pooling to obtain the final classification result.

[0056] A computer-readable storage medium is also proposed, on which a computer program is stored. When the computer program is executed by a processor, it implements the multi-frequency gated map LCZ classification method based on multi-view satellite imagery as described above.

[0057] A computer program product is also proposed, comprising a computer program that, when executed by a processor, implements the aforementioned multi-frequency gated map LCZ classification method based on multi-view satellite imagery.

[0058] The beneficial effects of this invention are as follows:

[0059] This invention introduces multi-view feature images into LCZ classification, effectively capturing image objects and multi-angle features through deformable gating branches. This solves the problem of insufficient depth and efficiency of CNN models under single-view remote sensing images, significantly improving the network's ability to handle complex tasks. The frequency domain transformation branch of this invention addresses the multispectral characteristics of remote sensing images, improving spectral recognition accuracy by optimizing high-frequency and low-frequency information processing. Simultaneously, it extracts global image features, enhancing the model's ability to utilize three-dimensional features of ground objects.

[0060] The graph convolution module of this invention enhances the extraction of local features and CNN latent features through graph convolution encoding and Chebyshev multinomial fitting of convolution kernels, supports the construction of deeper networks, achieves deeper feature learning, and improves classification accuracy.

[0061] This invention increases the diversity of remote sensing images by fusing multi-view and multispectral images, overcoming the dependence of traditional methods on single-view or high-cost images. At the same time, the designed dual-branch network structure and GCNConv module meet the network depth requirements of high-precision mapping and promote the development of LCZ classification technology.

[0062] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0063] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0064] Figure 1 This is a schematic diagram of the overall process of the multi-frequency gated image LCZ classification method based on multi-view satellite images according to an embodiment of the present invention;

[0065] Figure 2 This is a diagram illustrating the multi-view feature extraction process according to an embodiment of the present invention;

[0066] Figure 3 This is a schematic diagram of the network structure of the deformable gated branch and frequency domain transform branch according to an embodiment of the present invention, wherein, Figure 3 (b) is a deformable gated branch. Figure 3 (a) is the frequency domain transform branch;

[0067] Figure 4 This is an example of the LCZ classification result of the method in this embodiment of the invention, wherein, Figure 4 (a) is the original ZY-3 image of the Chongqing area. Figure 4 (b) shows the result after LCZ mapping;

[0068] Figure 5 Examples of local image LCZ classification results between the present invention and other existing methods are provided in this embodiment. Figure 5 (a) is the original local image. Figure 5 (b) is the LCZ local classification image of this method. Figure 5 (c) is a local classification image of the existing SwingTransformer method. Detailed Implementation

[0069] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0070] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0071] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0072] Please see Figures 1-5This is a multi-frequency gated map LCZ classification method based on multi-view satellite imagery.

[0073] To comprehensively improve the model's overall performance, this invention, for the first time, designs a dual-branch flow structure and graph convolution for feature extraction from multi-view remote sensing images. This invention innovatively designs two core modules for processing multi-angle, multispectral images. First, a Frequency Domain Gated (FRDG) module contains two branches: the DG branch enhances long-range dependency modeling and deformable object extraction capabilities, effectively improving the capture of image object features and multi-angle features; the FREQ branch, targeting the multispectral characteristics of remote sensing images, significantly improves spectral recognition accuracy by optimizing high-frequency and low-frequency information processing. Second, a GCN module (GP) contains a GCN-based module that strengthens the extraction capabilities of local features and CNN latent features. These three modules work together to construct an efficient and accurate remote sensing image classification system.

[0074] Specifically, such as Figure 1 As shown, a multi-scale interactive cross-layer cross-decoding method (FGGP-Net) is proposed, which includes a frequency-domain gated module (FRDG) and a GCN module (GP). This network performs only one downsampling step to retain more spatial and spectral information of the image. Simultaneously, two parallel branches are set up to process their respective features independently, and then the processed features are fused to form an FRDG module. The fused features are reshaped into a graph structure and a variable index is constructed to adapt to the input of the GP module.

[0075] Based on the above, in this embodiment, the specific process of the multi-frequency gated map LCZ classification method based on multi-view satellite imagery includes:

[0076] S1. Acquire the original image and input it into the frequency domain gated module FRDG for processing;

[0077] S2. The image is processed in parallel by the deformable gating branch and the frequency transformation branch of the frequency domain gating module to obtain spatial and frequency domain features respectively.

[0078] S3. The deformable gated branch output features and the frequency domain transform branch output features are fused through the SE module of the frequency domain gated module to obtain the fused features;

[0079] S4. Input the fused features into the graph convolution module, which reshapes the fused features into a graph structure.

[0080] S5. Next, graph convolution encoding is performed on the nodes in the reshaped graph structure using the graph convolution module, and the convolution kernel g in the graph convolution process is fitted using Chebyshev polynomials. θ (Λ);

[0081] S6. Global average pooling is used to classify the output after graph convolution to obtain the final classification result.

[0082] In step S1 of this embodiment, the original image acquired includes urban multispectral imagery and urban multi-angle imagery;

[0083] After receiving the original image input, the image first enters the Frequency Domain Gated (FRDG) module to extract the channel spectral features and global features of the image, while adapting to the multi-angle features of the input image. This module mainly consists of three parts: ① Deformable-Gated mechanism, ② Frequency Domain Transformation, and ③ Squeeze-and-Excitation (SE) module. Among them, the Deformable-Gated mechanism and the Frequency Domain Transformation are processed in parallel to extract spatial and frequency domain features respectively. The extracted features are synchronously input into the SE module for fusion to obtain the fused features.

[0084] In step S2 of this embodiment, the deformable gating branch of the frequency domain gating module specifically includes the following steps:

[0085] For the acquired image features and the multi-angle features of the dataset used, such as Figure 3 As shown in (a), a downsampling is first performed, and deformable convolution is then applied to the downsampled features. This includes obtaining the offset using deformable 2D convolution and adding the offset to the deformable pooling process. Specifically, it is expressed as follows:

[0086] For each position p on the output feature map y 12 ,have:

[0087]

[0088] Where p n To enumerate the positions in R, Δp n This is the offset; the sampling point is now at the offset position p. n +Δp n Above. Due to the offset Δp n If it is a decimal, it can be achieved using bilinear interpolation as follows:

[0089]

[0090] Where p represents any position p0+p n +Δp n q represents the spatial location of all integers in the feature map x, and G(·,·) is bilinear interpolation.

[0091] In deformable pooling, adding the offset to space b, we have:

[0092]

[0093] The output is a k*k feature map y, where n is the number of pixels in b. Similar to the formula above, the decimal offset Δp is used. ij Add it to position b; finally, the deformable convolutional feature map y is obtained.

[0094] The obtained feature map y is input into the HorBlock module. The HorBlock module mainly uses recursive gated convolution gnConv for spatial long-distance dependency optimization. Let x∈R be the input feature, then the output y can be expressed as:

[0095]

[0096] y = φ out (P1)=φ out [f(q0)⊙P0]

[0097] Where φ in and φ out It is a linear projection layer that performs channel blending, with depthwise separable convolutions of f(·). The above gnconv is a first-order interaction because each... Only with An interaction was performed, so to further improve the model's capabilities, we first used φ. in Get P0 and q k Then, multi-space interaction operations are performed through recursive gated convolutions, where the output is scaled by multiplying by 1 / α to stabilize training. k} is a set of depthwise separable convolutions, {g k} is used to match dimensions, and finally the output q of the last recursive step is used. n Input to φ out Obtain the output result.

[0098] In this embodiment, as Figure 3 As shown in (b), in the deformable gating branch, a Conv2D layer, a LayerNorm layer, and two consecutive gating units are set in sequence, which include a Deformable Conv layer and a HorBlock layer.

[0099] Another branch of FRDG is the frequency domain branch, which applies spatial domain features to the frequency domain, utilizing the distinguishability of spectral features in the frequency domain. The frequency domain transform branch of the domain-gated module specifically includes the following steps:

[0100] First, the image is input into the Conv2D and GRLU layers for feature map preprocessing;

[0101] Then, a two-dimensional discrete Fourier transform (2D FFT) is performed on the processed image. The 2D FFT can be viewed as alternating shifts in a one-dimensional discrete Fourier transform (DFT) across two dimensions. The two-dimensional DFT of real-number input x[m,n] satisfies the conjugate symmetry property X[Mu,Nv]=X[u,v]. Specifically, the one-dimensional DFT process is as follows:

[0102] Given a sequence of N complex numbers x[n], 0≤n≤N-1, a one-dimensional DFT can be transformed to the frequency domain using the following formula:

[0103]

[0104] Where j is the imaginary unit and W N =e -j(2π / N) Since X[k] repeats over an interval of length N, we only need to take the values ​​of X[k] at N consecutive points k = 0, 1, ..., N-1. Furthermore, for subsequent steps, the original spatial domain can be recovered using the inverse DFT.

[0105]

[0106] In this embodiment, the real and imaginary parts are separated on the original 2D FFT, and a nonlinear transformation is achieved by combining the advantages of CNN, which can be expanded using Euler's formula as follows:

[0107]

[0108] The input is a sequence of N complex numbers x[n], and f() is a convolutional layer. Finally, two features are obtained: one is the low-frequency information of the real part cosine wave component, and the other is the high-frequency information of the imaginary part sine wave component.

[0109] Then, the two pieces of information are processed by Conv2D and GELU respectively, and then subjected to 2D IFFT transformation to obtain the frequency domain transformation output features.

[0110] In step S3 of this embodiment, the SE module in the frequency domain gating module is used to convert the output feature DG of the deformable-gated branch. g Output characteristics FR of frequency domain transform branch s Feature fusion is performed, which is represented as:

[0111] F = f SE [DG g +FR s ]

[0112] Among them, f SE [·] indicates the fusion operation of the SE module.

[0113] In step S4 of this embodiment, the features from the image and spectrum are first reconstructed into a graph structure G = (V, E, A), where V and E are the sequences of vertices and edges of the graph, respectively, and A ∈ R×N×N is the adjacency matrix of the graph; D is defined as a matrix of the same size as A, and the Laplacian matrix L can be obtained:

[0114]

[0115] Where I is the identity matrix.

[0116] In step S5 of this embodiment, the convolution kernel g is then used. θ Encode the nodes, g θ =diag(θ),θ∈R N Graph convolution can be calculated as follows:

[0117] x*g θ =Ug θ (Λ)U T x

[0118] Where Λ and U are the normalized Laplace matrix L. n The eigenvalues ​​of the eigenvector matrix; x is the input, and g is the eigenvalues ​​of the eigenvector matrix; θ It is a function of Λ. Then, Chebyshev polynomials T are used. k To fit g θ (Λ), as follows:

[0119]

[0120] in Based on the above formula, the migration formula for GCN can be derived as follows:

[0121]

[0122] Among them G l and W l They are l th The layer's output and learnable weights, where σ is the activation function. After completing graph convolution, 1x1 convolutions can be used to enhance information.

[0123] In step S6 of this embodiment, global average pooling is used to classify the output after graph convolution processing to obtain the final classification result.

[0124] Example 2

[0125] This embodiment 1 focuses on the specific classification method given in embodiment 2. To evaluate the performance of the FGGP-Net network in the method, it is compared with two existing cutting-edge methods, including: (1) MSMLA: a multi-scale, multi-level attention network; (2) Swin-Transformer: a hierarchical Transformer implemented with shift windows; and (3) ResNet: a classic network based on residual connections. The specific process includes:

[0126] Experimental Dataset: ZY-3 satellite imagery is a high-resolution, multi-view image dataset. This dataset utilizes the ZY-3 panchromatic tri-line array (TLC) camera, including one orthophoto (NAD) sensor and two tilting sensors (Forward, FWD and Backward, BWD). The study area includes five cities with typical geographical and social environments: Beijing, Chongqing, Guangzhou, Wuhan, and Kunming. A total of 16,309 images are included, each set to 50×50 pixels as input to the network.

[0127] The network training is implemented as follows:

[0128] Loss function: The cross-entropy loss function is adopted, which has good numerical stability and efficient gradient calculation to ensure model convergence.

[0129] The AdamW optimizer was used on the dataset, and the model was trained for 300 epochs with a learning rate of 0.0002 to ensure model convergence.

[0130] In the multi-angle dataset, we cropped it to 50×50 and added the multi-angle features of the image to the channels, thus expanding it to 11 channels.

[0131] During training, data augmentation strategies are employed, including random flipping, random rotation, and adding Gaussian noise to increase the diversity of the data.

[0132] In the prediction section, ZY-3 remote sensing images covering the urban area of ​​Chongqing were used as the experimental area, such as... Figure 4 As shown in (a), LCZ classification is performed using the method of this invention, and the final LCZ category thematic map is obtained, as shown in (a). Figure 4 As shown in (b).

[0133] To further illustrate the effectiveness of the method of the present invention, Figure 5 The magnified view shows the classification effect of LCZ in a local area, where... Figure 5 (b) shows the results of this method. Figure 5 (c) shows the results of the comparison method Swing Transformer. It can be seen that the present invention can identify building types more accurately and has fewer misclassifications.

[0134] Our method achieved Precision, F1, and Recall scores of 0.8768, 0.8681, and 0.8673, respectively, on a multi-angle dataset (including five cities: Beijing, Chongqing, Guangzhou, Wuhan, and Kunming). Other comparison networks, including MSMLA-Net (0.8461, 0.8351, 0.8350), Swin Transformer (0.7917, 0.7668, 0.7679), and ResNet3 (0.8463, 0.8321, 0.8344), significantly outperformed our method.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multi-frequency gated image LCZ classification method based on multi-view satellite imagery, characterized in that: The method includes: The original image is acquired and input into the frequency domain gating module for processing. The image is processed in parallel by the deformable gating branch and the frequency transform branch of the frequency domain gating module to obtain spatial features and frequency domain features respectively. The SE module of the frequency domain gated module fuses the output features of the deformable gated branch with the output features of the frequency domain transform branch to obtain the fused features; The fused features are input into the graph convolution module, which reshapes the fused features into a graph structure. Then, graph convolution encoding is performed on the nodes in the reshaped graph structure, and Chebyshev polynomials are used to fit the convolution kernel in the graph convolution process. Global average pooling is used to classify the output of the graph convolution process to obtain the final classification result.

2. The multi-frequency gated image LCZ classification method based on multi-view satellite imagery according to claim 1, characterized in that: The system receives multi-angle images as input and uses the frequency domain gating module FRDG to extract spatial stereo features and image frequency domain features. The frequency domain gating module includes a deformable gating branch, a frequency domain transform branch, and an SE module. The deformable gating branch and the frequency domain transform branch are processed in parallel to extract their respective features. The extracted features are synchronously input into the SE module for fusion to obtain fused features.

3. The multi-frequency gated image LCZ classification method based on multi-view satellite imagery according to claim 1, characterized in that: In the deformable gating branch of the frequency domain gating module, the acquired image features F1 and the multi-angle features F2 of the dataset are concatenated dimensionally. First, downsampling is performed, and deformable convolution is applied to the downsampled features. This includes obtaining the offset using deformable 2D convolution and adding the offset to the deformable pooling process, as shown below: For each position p on the output feature map y 12 ,have: Where p n To enumerate the positions in R, x1 and x2 are the image features and multi-angle features, respectively, and Δp n p is the offset. n +Δp n Indicates the offset position of the current sampling point; offset Δp n Determined by bilinear interpolation as follows: Where p 12 Represents any position p0+p n +Δp n q represents the spatial location of all integers in the feature map x, and G(·,·) is bilinear interpolation; In deformable pooling, adding the offset to space b, we have: The output is a k*k feature map y, where n is the number of pixels in b. Similar to the formula above, the decimal offset Δp is used. ij Add it to position b; finally, the deformable convolutional feature map y is obtained. The obtained feature map y is input into the HorBlock module. The HorBlock module mainly uses recursive gated convolution gnConv to perform spatial long-distance dependency optimization. Let x∈R be the input feature, then the output y is expressed as: y=φ out (P1)=φ out [f(q0)⊙P0] Where, φ in and φ out It is a linear projection layer that performs channel blending, f(·) is a depthwise separable convolution, and C is the dimension that blends the image spectrum and multi-angle features; Based on the first-order interaction of gnconv described above, using φ in Get P0 and q k Then, multi-space interaction operations are performed through recursive gated convolution, where scaling is controlled by 1 / α; {f k } is a set of depthwise separable convolutions, {g k } is used to match dimensions, and finally the output q of the last recursive step is used. n Input to φ out Obtain the output result.

4. The multi-frequency gated image LCZ classification method based on multi-view satellite imagery according to claim 1, characterized in that: In the frequency domain transform branch of the frequency domain gate module, the feature map is first preprocessed in the Conv2D and GRLU layers of the image input; Then, the processed image is subjected to two-dimensional discrete Fourier transform (2D-FFT), where the 2D-FFT alternately performs shifted one-dimensional discrete Fourier transform (DFT) in two dimensions. Based on the one-dimensional DFT, the two-dimensional DFT of the real input x[m,n] satisfies the conjugate symmetry property X[Mu,Nv]=X[u,v]. The process of the one-dimensional DFT is as follows: Given a sequence of N complex numbers x[n], 0≤n≤N-1, a one-dimensional DFT can be transformed to the frequency domain using the following formula: Where j is the imaginary unit and W N =e -j(2π / N) Take the values ​​of X[k] at N consecutive points k = 0, 1, ..., N-1; Recovering the original spatial domain through inverse DFT: The real and imaginary parts are separated on the original 2D FFT, and the nonlinear transformation is achieved by combining the advantages of CNN. The result is expanded using Euler's formula: The input is a sequence of N complex numbers x[n], and f(·) is a convolutional layer. Finally, two features are obtained: one is the low-frequency information of the real part cosine wave component, and the other is the high-frequency information of the imaginary part sine wave component. Then, the two pieces of information are processed by Conv2D and GELU respectively, and then subjected to 2D-IFFT transformation to obtain the frequency domain transformation output features.

5. The multi-frequency gated image LCZ classification method based on multi-view satellite imagery according to claim 1, characterized in that: The output characteristics DG of the deformable gated branch are obtained by using the SE module in the frequency domain gating module. g Output characteristics FR of frequency domain transform branch s Feature fusion is performed, which is represented as: F<f SE [DG g +FR s ] Among them, f SE [·] indicates the fusion operation of the SE module.

6. The multi-frequency gated image LCZ classification method based on multi-view satellite imagery according to claim 1, characterized in that: In the graph convolution module, the fused features are reshaped into a graph structure G = (V, E, A), where V and E are the sequences of vertices and edges of the graph, respectively, and A ∈ R × N × N is the adjacency matrix of the graph; D is defined as a matrix of the same size as A, yielding the Laplacian matrix L: Where I is the identity matrix.

7. The multi-frequency gated image LCZ classification method based on multi-view satellite imagery according to claim 1, characterized in that: During the encoding process, convolution kernel g is used. θ Encode the nodes, g θ =diag(θ),θ∈R N Graph convolution is calculated as follows: x*g θ =Ug θ (Λ)U T x Where Λ and U are the normalized Laplacian matrices L and U, respectively. n The eigenvalues ​​of the eigenvector matrix; x is the input, and g is the eigenvalues ​​of the eigenvector matrix; θ It is a function of Λ; then Chebyshev polynomial T is used. k To fit g θ (Λ), as follows: in Therefore, the migration formula for GCN is derived as follows: Among them, G l and W l They are l th The layer's output and learnable weights, where σ is the activation function; After completing the graph convolution, a 1x1 convolution is used to enhance the information, resulting in the output of the graph convolution.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the multi-frequency gated image LCZ classification method based on multi-view satellite imagery as described in any one of claims 1-7.

9. A computer program product, characterized in that: Includes a computer program that, when executed by a processor, implements the multi-frequency gated map (LCZ) classification method based on multi-view satellite imagery as described in any one of claims 1-7.

Images