Remote sensing ground feature classification extraction network and drawing method in complex scene

Abstract

The invention relates to the technical field of remote sensing image processing, and discloses a remote sensing ground feature classification extraction network and drawing method in a complex scene, and the method comprises the following steps: S1, obtaining high-resolution satellite remote sensing image data, carrying out the preprocessing of an image, and generating a multi-scene ground feature data set; s2, constructing a two-layer U-shaped network comprising an encoder and a decoder, wherein the two-layer U-shaped network is used for multi-scale feature extraction and ground feature boundary segmentation; s3, inputting the preprocessed remote sensing image into a two-layer U-shaped network, and carrying out ground feature classification and segmentation; s4, performing splicing processing on the segmentation result, and generating a ground feature type spatial distribution map; and S5, performing noise elimination and post-processing operation on the generated distribution map. The multi-scale features are selectively extracted through the feature selection module (FSM), the attention of the network to small target ground features can be enhanced, and the segmentation precision of the small-scale ground features is remarkably improved by selecting the features containing important details.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a remote sensing object classification extraction network and mapping method in complex scenes. Background Art

[0002] With the development of remote sensing technology and the widespread application of high-resolution satellite images, remote sensing object classification and mapping have received increasing attention in the fields of land use planning, farmland resource management, urban planning, and ecological environment monitoring. Obtaining object classification results through remote sensing images can not only quickly and accurately identify different types of objects on the surface, but also provide large-scale and dynamic object distribution information. However, with the improvement of remote sensing image resolution, object classification and mapping tasks in complex scenes face a series of challenges, especially in the extraction of fine-grained small targets and the segmentation of complex object boundaries.

[0003] Existing technologies usually face the problem of insufficient segmentation accuracy when dealing with small target objects (such as narrow roads, building edges, ditches, etc.). Due to the small area of ​​small objects, their edges and details are easily ignored in traditional machine learning or shallow convolutional networks. Especially in complex scenes, the spatial features between different objects overlap and the boundaries are blurred, making it difficult for existing technologies to achieve accurate segmentation. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a remote sensing object classification extraction network and mapping method in complex scenes, which solves the problem of insufficient segmentation accuracy commonly faced by the prior art when processing small target objects.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: a remote sensing object classification extraction network and mapping method under complex scenes, comprising the following steps:

[0006] S1: Acquire high-resolution satellite remote sensing image data, pre-process the images, and generate multi-scene ground object datasets;

[0007] S2: Construct a two-layer U-shaped network including an encoder and a decoder for multi-scale feature extraction and object boundary segmentation;

[0008] S3: Input the preprocessed remote sensing image into a two-layer U-shaped network for object classification and segmentation;

[0009] S4: splicing the segmentation results and generating a spatial distribution map of the ground object types;

[0010] S5: performing noise removal and post-processing operations on the generated distribution map.

[0011] Preferably, the high-resolution satellite remote sensing image data comes from GF-1, GF-2 and GF-6 satellites, and the image data includes multi-source remote sensing data with resolutions of 0.3 meters, 1 meter and 2 meters.

[0012] Preferably, the encoder gradually extracts multi-scale features of the remote sensing image by downsampling, and the decoder restores the image spatial resolution by upsampling.

[0013] Preferably, a feature selection module is added to the encoder, and the feature selection module is used to select fine-grained features from multi-scale features.

[0014] Preferably, a feature alignment module is added to the decoder, and the feature alignment module is used to align the up-sampled high-level features with the low-level features in the encoding stage.

[0015] Preferably, the stitching process uses a sliding window technique to crop the remote sensing image.

[0016] Preferably, the voting method in the stitching process is used to process the pixel values ​​of multiple overlapping areas during stitching, and the final stitching result is determined by weighted calculation.

[0017] Preferably, the post-processing operation includes:

[0018] Use majority filter to eliminate noise and isolated pixels in the segmentation result;

[0019] Smooth the boundaries of features.

[0020] Preferably, the feature alignment module is based on deformable convolution technology for dynamically adjusting the spatial offset between high-level and low-level features.

[0021] Preferably, the two-layer U-shaped network uses a cross entropy loss function and combines a stochastic gradient descent optimizer for model training.

[0022] The present invention provides a remote sensing object classification extraction network and mapping method in complex scenes. It has the following beneficial effects:

[0023] 1. The present invention uses a feature selection module (FSM) to selectively extract multi-scale features, which can enhance the network's attention to small target objects. By selecting features that contain important details, the present invention significantly improves the segmentation accuracy of small-scale objects and solves the problem that the existing technology is difficult to effectively extract small target objects.

[0024] 2. The present invention achieves the full extraction and integration of multi-scale features by constructing a two-layer U-shaped deep convolutional neural network (SAU2-Net). The encoder and decoder can extract local details while maintaining global semantic information through multiple up- and down-sampling operations. This design improves the segmentation effect of large and small objects, and is particularly suitable for complex scenes with large differences in object scales.

[0025] 3. The present invention introduces a feature alignment module (FAM) in the decoding process, which automatically aligns feature maps of different resolutions using deformable convolution technology to solve the problem of feature misalignment. This module can effectively improve the segmentation accuracy of the boundaries of objects in complex scenes, especially at the junctions of farmland and forests, buildings and roads, etc., avoiding the boundary confusion phenomenon common in the prior art.

[0026] 4. The present invention solves the pixel value selection problem of the splicing area by using voting, ensuring that no obvious splicing traces will appear in the splicing process of large-scale images. By weighted calculation of the pixel values ​​of the overlapping areas, a seamless splicing feature classification map is finally generated, which improves the coherence and consistency of the mapping results.

[0027] 5. The present invention uses a majority filter to eliminate noise from the segmentation results in the post-processing stage and processes isolated pixels to ensure the continuity and consistency of the results. In addition, the boundary jagged phenomenon generated during the classification process is removed through boundary smoothing, making the boundary of the object more natural and clear, and significantly improving the visual effect and spatial consistency of the classification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a geographical location distribution map of the research object image of the present invention;

[0029] Figure 2 This is a partial data and label example diagram of the present invention;

[0030] Figure 3 It is a remote sensing image diagram under different scenes of the present invention;

[0031] Figure 4 It is a flow chart of the research and mapping of remote sensing image object type segmentation of the present invention;

[0032] Figure 5 The SAU of the present invention 2 -Network structure diagram;

[0033] Figure 6 It is a schematic diagram of a feature alignment module of the present invention;

[0034] Figure 7 It is a schematic diagram of a feature selection module of the present invention;

[0035] Figure 8 A flow chart of remote sensing mapping of spatial distribution of land feature types of the present invention;

[0036] Fig. 9 This is a comparison chart of the segmentation results of large images between the advanced semantic segmentation network of the present invention and the method of this paper on the test set;

[0037] Fig.10 It is a segmentation result diagram of an image in the test set of the Loveda data set of the present invention;

[0038] Fig.11 A visual feature map generated on the test set of the present invention;

[0039] Fig.12 It is a semantic segmentation result diagram of 2m and 1m spatial resolution images in the complex background object type data of the present invention;

[0040] Fig.13 This is a semantic segmentation result diagram of a 0.3m spatial resolution image in the LoveDA test set of the present invention;

[0041] Fig.14 It is the result map of ground object classification of images in mountainous area, rural area and town scene of the present invention;

[0042] Fig.15 It is a result diagram of ground object classification of images in different crop planting scenes of the present invention;

[0043] Fig.16 This is a schematic diagram of random sampling and fishing net sampling of the present invention;

[0044] Fig.17 It is a detailed comparison diagram of the prediction results before and after post-processing of the present invention;

[0045] Fig.18 This is the remote sensing spatial distribution map of land feature types in Qi County, Hebi City in two periods of the present invention. DETAILED DESCRIPTION

[0046] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Please see attached Figure 1 - Attachment Fig.18 The embodiment of the present invention provides a remote sensing object classification extraction network and mapping method in a complex scene, including the following steps:

[0048] S1: Acquire high-resolution satellite remote sensing image data, pre-process the images, and generate multi-scene ground object datasets;

[0049] S2: Construct a two-layer U-shaped network including an encoder and a decoder for multi-scale feature extraction and object boundary segmentation;

[0050] S3: Input the preprocessed remote sensing image into a two-layer U-shaped network for object classification and segmentation;

[0051] S4: splicing the segmentation results and generating a spatial distribution map of the ground object types;

[0052] S5: performing noise removal and post-processing operations on the generated distribution map.

[0053] Specifically, the above content mainly describes the overall process of a remote sensing feature classification extraction network and mapping method in a complex scene, including five main steps: data acquisition, network construction, image classification and segmentation, splicing processing and post-processing. The core of this method is to obtain and process high-resolution remote sensing image data, and use the construction and optimization of deep learning networks to finally generate a high-precision spatial distribution map of feature types.

[0054] In step S1, the acquisition and processing of high-resolution remote sensing image data is the basis of the entire process. By selecting high-resolution satellite images such as GF-1, GF-2, and GF-6, clear ground object information can be obtained. After being processed by radiation correction and orthorectification, these image data provide high-quality input data sets for subsequent classification and segmentation.

[0055] The S2 step emphasizes the design of the network, using a deep convolutional neural network (two-layer U-shaped network structure), which is characterized by the ability to extract multi-scale features from remote sensing images in complex scenes. The encoder is responsible for extracting features of objects at different resolutions, and the decoder restores spatial details through upsampling, so that the network can not only accurately classify large targets, but also finely segment small targets and complex object boundaries.

[0056] In step S3, the pre-processed images are input into the network to automatically classify and segment the objects. Compared with traditional manual classification or machine learning methods, deep learning networks can automatically extract useful features through massive data training, reduce manual intervention, and greatly improve the efficiency and accuracy of object classification.

[0057] Step S4 solves the problem of stitching large-scale images through stitching. Since high-resolution remote sensing images are usually very large, block processing is necessary. In the stitching process, the pixel values ​​of the overlapping areas are processed by voting to ensure seamless stitching, which is very critical in large-scale remote sensing mapping.

[0058] Step S5 is the post-processing step, which uses techniques such as majority filtering to remove noise and isolated pixels in the segmentation results and smooth the boundaries to generate a high-precision spatial distribution map of landform types. The overall method can provide detailed spatial information and is widely used in application scenarios such as land use planning, farmland management, and urban planning.

[0059] High-resolution satellite remote sensing image data comes from GF-1, GF-2 and GF-6 satellites, and the image data include multi-source remote sensing data with resolutions of 0.3 meters, 1 meter and 2 meters.

[0060] Specifically, the data source and resolution are further defined, and it is clarified that the high-resolution remote sensing image data comes from GF-1, GF-2 and GF-6 satellites. Each satellite has different imaging characteristics and resolution, so that the method can be effectively applied at different spatial scales.

[0061] The GF-1 satellite provides a spatial resolution of 2 meters, which is suitable for extracting basic land features in large areas, such as the distribution of farmland and forests. The 1-meter resolution image of the GF-2 satellite can capture more detailed land feature information, especially land features with clear structures such as buildings and roads. The GF-6 satellite has a higher resolution of 0.3 meters. Such ultra-high resolution enables small-scale land features (such as narrow roads, ditches in farmlands, etc.) to be clearly identified and segmented. By combining different data sources from multiple satellites, the generalization and applicability of the method are enhanced, especially in complex scenes, it can take into account both the overall classification of large areas and the precise segmentation of local details.

[0062] The encoder gradually extracts the multi-scale features of the remote sensing image through downsampling, and the decoder restores the image spatial resolution through upsampling.

[0063] Specifically, the basic working principles of the encoder and decoder are explained in detail. The encoder is responsible for converting the input high-resolution image into a series of multi-scale feature maps through downsampling operations. These feature maps retain the semantic information of the image at different spatial scales, allowing the network to identify large objects (such as farmland and forests) and small objects (such as roads and buildings).

[0064] The decoder restores the feature map output by the encoder to the resolution of the original input image through upsampling operations, and fuses it with the low-level features in the encoder through jump connections to ensure that spatial detail information is not lost during the upsampling process. This design enables the network to capture global semantic information while ensuring the accuracy of local details. Through this step, complex scenes in the image (such as buildings and road networks in cities, crop distribution in farmland, etc.) can be clearly distinguished and accurately segmented.

[0065] A feature selection module is added to the encoder, which is used to select fine-grained features from multi-scale features, reduce redundant information and enhance the accuracy of object classification.

[0066] Specifically, the function and role of the feature selection module (FSM) in the encoder are further explained. The feature selection module automatically selects the features with the most spatial details and semantic information by analyzing feature maps of different scales to reduce the amount of computation required by the network to process redundant information. This process enhances the segmentation capability of the network, especially the segmentation accuracy of small target objects (such as narrow roads, building edges, etc.) by selectively retaining features that contribute to the target objects.

[0067] The addition of the feature selection module can effectively reduce the complexity of the model, while improving the accuracy and efficiency of the model. For processing large-scale high-resolution remote sensing image data, this method can effectively improve the performance of the network and ensure the classification effect of small-scale targets and complex object boundaries.

[0068] A feature alignment module is added to the decoder. The feature alignment module is used to align the upsampled high-level features with the low-level features in the encoding stage to reduce feature misalignment and improve the accuracy of object boundary segmentation.

[0069] Specifically, the role of the feature alignment module (FAM) is particularly important in the decoding process. When the decoder is upsampling, the low-resolution feature map needs to be aligned with the high-resolution feature map extracted in the encoder to ensure the accuracy of information fusion. The feature alignment module uses deformable convolution technology to automatically align the feature maps of the upper and lower layers by learning the offset between feature maps of different layers, thereby avoiding the problem of feature misalignment caused by resolution differences.

[0070] The core goal of this module is to ensure that the precise segmentation of object boundaries is not affected by feature misalignment during the decoding process, thereby improving the accuracy of object classification, especially when dealing with complex scenes (such as densely built-up areas in cities or trail networks in forests).

[0071] The stitching process uses sliding window technology to crop the remote sensing images so that the stitched segmented images can be seamlessly integrated.

[0072] Specifically, the sliding window technology used in the stitching process is expanded, emphasizing the key role of this technology in processing large-scale remote sensing images. Due to the large amount of data in high-resolution remote sensing images, the images need to be divided into small pieces for cropping and processing when processing in the network. The sliding window technology ensures that each part of the image can be cropped and processed step by step, thus avoiding the problem of memory overflow caused by too large an image.

[0073] The seamless stitching of the sliding window during stitching ensures that there will be no obvious stitching marks in the final segmentation map, thereby ensuring the coherence and integrity of the final spatial distribution map of land feature types.

[0074] The voting method in the stitching process is used to process the pixel values ​​of multiple overlapping areas during stitching, and the final stitching result is determined by weighted calculation.

[0075] Specifically, the voting method in the splicing process is emphasized, which is used to solve the problem of pixel value conflicts in the overlapping areas of the segmented images. When processing remote sensing images, images of different blocks may overlap at the edges. If the pixel values ​​of these areas are not handled properly, inconsistent classification results may result. Through the voting method, weighted calculations can be performed based on the pixel values ​​of the overlapping areas to ensure that the final results are more reliable;

[0076] The voting method not only improves the classification accuracy of the mosaic area, but also makes the final mosaic smoother and more consistent. This processing method is of great significance for ensuring the classification results of large-scale images, especially in large-scale remote sensing applications.

[0077] Post-processing operations include:

[0078] Use majority filter to eliminate noise and isolated pixels in the segmentation result;

[0079] Smooth the boundaries of features.

[0080] Specifically, the filtering and boundary smoothing processes in the post-processing operations are further described in detail. The majority filter can effectively remove noise and isolated pixels in the classification results, which may be due to small errors in the segmentation process. The filter counts the neighborhood around each pixel and selects the pixel value with the most occurrences as the value of the current pixel to eliminate abnormal pixels.

[0081] The boundary smoothing operation is used to process the continuity of the feature boundary and prevent jagged or irregular phenomena from appearing on the feature boundary. Through the smoothing operation, the segmentation result of the feature boundary is more refined and continuous, especially in the case of fuzzy boundaries between feature types (such as the edge of farmland and forest, the junction of buildings and roads), this smoothing process is particularly important.

[0082] The feature alignment module is based on the deformable convolution technique to dynamically adjust the spatial offset between high-level and low-level features.

[0083] Specifically, further technical details of the feature alignment module are described, which uses deformable convolution technology to dynamically adjust the spatial offset between high-level features and low-level features. This module ensures that high-resolution and low-resolution features can be accurately aligned during upsampling by learning the offset of each feature layer.

[0084] Through deformable convolution technology, the network can adaptively adjust features of different scales to make the final segmentation result more accurate, especially in the classification of complex land boundaries and irregular areas (such as curved roads and broken farmland). The feature alignment module can significantly improve the classification effect.

[0085] The two-layer U-shaped network uses the cross entropy loss function and combines the stochastic gradient descent optimizer for model training.

[0086] Specifically, the training process of the network is expanded, especially the technical details of training using the cross entropy loss function and the stochastic gradient descent (SGD) optimizer. The cross entropy loss function is a common loss function for multi-classification tasks, which measures the difference between the network output and the true label. By minimizing the cross entropy loss, the network can gradually adjust parameters to improve classification accuracy.

[0087] The stochastic gradient descent (SGD) optimizer is used to update the weight parameters of the network. Compared with the traditional gradient descent method, SGD can handle large-scale data and speed up training, avoiding falling into local optimality. Combining the cross entropy loss function with the SGD optimizer, the network can effectively learn the complex features of the objects and enhance the accuracy and efficiency of segmentation.

[0088] Study area and data

[0089] Overview of the Study Area

[0090] This paper selects representative areas in 12 cities in Henan Province as the research area, with a total area of ​​about 1.16×104km2. Figure 1 Schematic diagram of data distribution in the study area. The images of the 6 rectangular red frame areas in the figure are used as model training data. The areas are manually annotated with high-quality labels for training the land feature classification model and verifying the classification accuracy. The study area has scattered settlements and a large area of ​​farmland, which has important reference value for the classification of land features dominated by farmland in my country. The images of the 9 county areas in the blue frame in the figure are used as spatial distribution mapping test data, with a total area of ​​about 7.1×103km2. This area is used to verify the accuracy of model mapping. The above 9 counties are spatially independent and contain rich terrain, so they are suitable as test areas for evaluating the accuracy of spatial distribution maps.

[0091] Data Source

[0092] The GF-1, GF-2, and GF-6PMS satellite remote sensing images used in this invention are from the China Resources Satellite Application Center (https: / / data.cresda.cn / ), and the red, green, and blue bands are selected. The image date is from August 2022 to April 2024. Table 1 is the detailed information of the image data used in the corresponding area. Referring to the "China Land Use Classification Standard" (GB / T210102017), the types of land objects are divided into six categories: farmland, forest, water, building, road, and background. In order to study the land object classification effect of remote sensing scenes under different influencing factors, the challenges in the land object classification task in the study area are analyzed from the perspectives of three scenes: different spatial resolutions, geographical environments, and crop phenological periods, and experimental data are selected. There are obvious differences in the same type of land objects under different terrains and landforms. In order to verify the adaptability of the research method under different terrains and landforms, the selected images are geographically distributed in 14 counties in Henan Province, containing rich mountainous areas, cities, and plain terrain and landform information, so that the research data has a wide distribution. Due to changes in seasons, temperature, light, and time, the spectral responses of objects in the same area show obvious differences. For example, there are usually large differences in crop types in farmland. In order to verify the classification effectiveness of the research method in different periods, images were obtained from the same county at different times to make the research data multi-temporal. The higher the spatial resolution, the richer the detailed information of the objects contained in the remote sensing image, and the target morphology will be different. In order to verify the object segmentation effect of the research method at different spatial resolutions, images were obtained from GF-1, GF-2, GF-6 and Google Earth, including three resolutions of 1m, 2m, and 0.3m, so that the research data has multiple sources.

[0093] Table 1 Detailed information of satellite images

[0094]

[0095]

[0096] Dataset construction

[0097] LoveDA dataset

[0098] The LoveDA dataset contains 5987 high spatial resolution (0.3m) images in both urban and rural geographical environments, collected in July 2016, with a total coverage area of ​​536.15km2. The dataset contains samples under 7 complex backgrounds, which has made a great contribution to the remote sensing land cover classification task. The purpose is to test the generalization ability of the model.

[0099] Complex scene object classification dataset

[0100] After the original data was downloaded, ENVI5.3 (annotation company) was used to perform radiometric calibration, orthorectification (RPC), image fusion (NND), band selection, cropping and other operations. After a series of processing, the experimental data was obtained. Based on the field survey results, the training data was manually outlined and visually interpreted. The annotated original image and label are shown in Figure 2. Figure 2 The obtained data category ratio is shown in Table 2. The images and labels are cropped into 515×512 images, and data enhancement is performed by randomly flipping, scaling, changing brightness, contrast, and saturation. The complex scene object type dataset has a total of 9432 images, which are divided into training set, validation set, and test set according to 8:1:1.

[0101] Table 2 Distribution of samples of different landform types

[0102]

[0103] In order to verify the segmentation effect of objects in different scenes, appropriate images were selected from the county maps that had not been trained as prediction images. 30 images were randomly cropped from the 2m and 1m images of Shunhe Hui Autonomous District, Kaifeng, to verify the segmentation effect at different resolutions. 30 images were randomly cropped from the mountainous part of Qi County, Hebi City, the rural area of ​​Fan County, Puyang City, and the densely populated urban area of ​​Suiyang District, Shangqiu City, to verify the segmentation effect under different geographical environments. 30 images were randomly cropped from the August corn jointing period, the March wheat jointing period, and the November bare land images of Wancheng District, Nanyang City, to verify the segmentation effect under different crop phenological periods.

[0104] Figure 3 Figures (1)-(3) are examples of images in remote sensing scenes with different resolutions. (1) is an image with a spatial resolution of 2m taken by the GF-1 satellite, (2) is an image with a resolution of 1m taken by the GF-2 image, and (3) is an image with a spatial resolution of 0.3m obtained from Google Earth. Figures (4)-(6) are image data in remote sensing scenes with different topography and landforms. (4) is a mountainous image. The farmland in the area is fragmented and irregular, and the roads are tortuous. (5) is a rural image. The farmland in this area is large and flat, but it is easy to be confused with other forests (such as grasslands, forests, etc.). (6) is an urban image. The buildings in this area are dense and regular, which is obviously different from the buildings in the countryside. The roads are wide and there are almost no farmlands. Figures (7)-(10) are image data in different crop phenological periods. (7) is an image of the corn jointing stage, (8) is an image of the wheat jointing stage, (9) contains a large area of ​​bare land, and (10) contains mixed planting of multiple crops. From the picture, we can see that crops are usually green, corn is lighter than wheat, other crops are even lighter, and bare land is khaki. Visually, the color difference is large and the texture is different.

[0105] Land feature classification and mapping methods

[0106] Feature classification and mapping process

[0107] The main research process of the present invention is as follows Figure 4 As shown. It includes three parts: data set construction, model training, model reasoning and evaluation. Image data of different resolutions are preprocessed, and a ground object classification data set is generated through image fusion and sliding window cropping technology. The semantic segmentation model is trained using the existing data set to achieve effective segmentation of ground object types, and the effectiveness of the model is verified through comparative experiments and ablation experiments. Combined with the spatial distribution mapping method proposed in this invention, the county images are divided into blocks and input into the model for prediction, and the spatial distribution map of ground object types is generated after sequential splicing.

[0108] SAU2-Net object classification network

[0109] This paper proposes a new feature extraction network SAU2-Net suitable for high-resolution remote sensing image object classification tasks. Figure 5 As shown. SAU2-Net is a two-layer nested U-shaped structure, in which an RSU residual block of different heights is embedded in each stage. This structure can obtain multi-scale features of each stage without reducing the resolution of the feature map and realize multi-level feature aggregation through jump connections. The overall structure of the network is a 6-stage encoder from bottom to top and a 5-stage decoder from top to bottom. The rightmost side outputs 6 feature maps of different scales and merges them into the final output prediction result. The SAU2-Net network adds a feature selection module at the jump connection to make it pay more attention to the classification of target boundaries. In the upsampling process, a feature alignment module is added to align the high-level features after upsampling and the rich detail features in the encoding stage. SAU 2 The -Net network removes the sigmoid function used at the end of the U2-Net network and replaces the 3×3 convolution in the original operation with a 1×1 convolution to adapt to multi-classification semantic segmentation tasks.

[0110] Feature Alignment Module

[0111] In U2-Net, the outer U-shaped structure fuses the high-level features of the corresponding positions with the low-level features through jump connections. The network decoding stage restores the resolution of the feature map through multiple upsampling, but there will be feature misalignment in the upsampled feature map, and the misalignment problem becomes more serious as the number of upsampling increases. Directly fusing the feature map with the high-resolution feature map will result in poor boundary information prediction. Adding a feature alignment module, namely FAM, can align the 2x upsampled features with the bottom-up features of their corresponding positions, so that more accurate boundary information can be extracted. Figure 6 As shown, the mathematical expression of feature alignment is:

[0112]

[0113] in represents the spatial difference between low-level features and corresponding high-level features, f a (·) and f o (·) respectively denote the learning space difference and alignment feature offset (Δ i+1 ), where the learning of the offset is achieved through deformable convolution.

[0114] Feature Selection Module

[0115] Feature selection before feature fusion can enhance important fine-grained feature maps containing a lot of spatial information and suppress redundant feature maps, such as Figure 7 The module workflow is shown in Figure 1. The input and output channels are represented by L i =[l1,l2,…,l D ], u=[u1,u2,...,u D ] is the feature importance vector, indicating the importance of each channel. Each input feature map l i Obtain global information z through global average pooling operation i , through the feature importance modeling layer f m (·) Learn the information of each channel and perform importance modeling, and output the feature importance vector u. The feature importance vector is then scaled and added to the input feature map, called the rescaled feature map. The original input feature map is fused with the rescaled feature map through a skip connection and fed into f s (·) function discards unimportant feature maps to reduce the number of channels and achieve feature selection. FSM can be used in the network neck to enhance multi-scale feature aggregation, and its feature selection results can be used as a reference for FAM to make the feature alignment module more effective. The mathematical expression of FSM is:

[0116]

[0117] u=f m (z)

[0118] Where z = [z1, z2, …, z D ], H, W represent the height and width of the image and are calculated by the following formula:

[0119]

[0120] The loss function used in the model of the present invention is the cross entropy loss function

[41] . The loss is calculated for multi-scale output features using a multi-supervisory approach. The training loss is calculated as follows:

[0121]

[0122] l side (m) and l fuse Represents the loss of each feature map and the feature map after the final fusion of features at each stage. side (m) and w fuse Represents the loss weight of each layer output and the final fused output. The calculation formula for each loss is:

[0123]

[0124] Where (r, c) represents the coordinates of the pixel, (H, W) represents the height and width of the image, M represents the number of categories, and P G(r，c)a Indicates the true category of the sample point (r, c), which takes 1 when it is equal to a, otherwise it takes 0, P S(r，c)a Represents the predicted probability that the sample point (r, c) belongs to category a.

[0125] Remote sensing spatial distribution mapping method

[0126] First, obtain the county remote sensing image, use a sliding pane cropping with a repetition rate to crop the image to a size that the model can handle, and then input the model for prediction. The output result is a single-channel image. The prediction result is spliced ​​back to the original image in the cropping order, and the affine matrix information of the original image is added to the splicing result to restore the geographic information. In the splicing process, there are repeated parts in the splicing boundary, and the pixel prediction result is determined by voting. This method can reduce the splicing traces. After obtaining the predicted large image, post-processing is required. First, the majority filter is used to replace the value of each pixel with the majority in its surrounding area to eliminate noise or isolated points in the image. Secondly, the irregular class boundaries are smoothed and the regions are aggregated by class. Finally, a spatial distribution map of land feature types that meets the requirements is generated. Figure 8 Shown is a flow chart of the remote sensing mapping method for the spatial distribution of land feature types.

[0127] Geographic information writing

[0128] Splicing overlap processing

[0129] Evaluation indicators

[0130] The evaluation indicators used in the present invention include mean intersection over union (mIoU), mean recall (mRecall), mean precision (mAP), overall classification accuracy (OA), Kappa coefficient, and Dice coefficient. The calculation formulas are (6)-(11).

[0131]

[0132] Among them, TP i Represents the count of true positives, FPi Represents the false positive count, TN i Represents the count of true negatives, FN i represents the count of false negatives, P o represents the accuracy, P e It represents the sum of the product of the actual and predicted quantities for all categories, divided by the square of the total number of samples.

[0133] Experimental results analysis

[0134] The operating system of the experiment is Ubuntu 20.04. The main required packages include python 3.8, Paddle 2.7, GDAL 3.5, etc. The CPU is Intel (R) Xeon (R) Silver 4114 CPU @ 2.20 GHz, the memory is 64 GB, and the GPU is NVIDIA Corporation GP104 GL (16 GB video memory). The SGD (Stochastic Gradient Descent) stochastic gradient descent method is used as the optimizer. The SGD momentum is set to 0.9, the weight decay is set to 4.0e-5 to avoid overfitting, and the loss weights are set to 1. Use the polynomial learning rate iteration strategy and turn on the linear learning rate warm-up. The initial learning rate is set to 1.0e-6, and the learning rate is adjusted normally after 1500 iterations.

[0135] The performance of the SAU2-Net model proposed in this paper is verified on two object classification datasets. During the training of the complex scene object classification dataset, the batch size is set to 12 and the number of iterations is set to 15k. At this time, the mIoU, Accuracy and loss of the model tend to be stable. The same experimental settings as in the original paper are used on the LoveDA dataset to ensure the comparability of the experimental results. Due to GPU performance issues, the number of training iterations is set to 20k and the batch size is 12.

[0136] Comparison with different semantic segmentation model performance ()

[0137] The SAU2-Net network proposed in this invention is compared with other advanced networks under the same experimental environment. The best results in the following table are highlighted in bold, and the suboptimal results are underlined.

[0138] Comparative experiments were conducted on self-constructed datasets. The semantic segmentation networks compared include PSPNet, Deeplabv3+, U-Net, UperNet, DANet, BiSeNetV2, and DDRNet. Table 3 shows that compared with other advanced segmentation models, the four indicators of the unpre-trained SAU2-Net have achieved the best results. Compared with the best model (DDRNet), mIoU increased by 2.72 percentage points and mRecall increased by 1.09 percentage points.

[0139] Table 3 Semantic segmentation results of different models on complex scene object classification dataset (%).

[0140]

[0141]

[0142] Table 4 shows the IoU of each category of the eight models in the test set. It can be seen from the table that SAU2-Net achieved the highest value in each category. Compared with the second best result, the farmland category increased by 1.02 percentage points, the forest category increased by 2.19 percentage points, and the road category increased by 4.1 percentage points.

[0143] Table 4 IoU (%) of each category of different models on the complex scene object classification dataset.

[0144]

[0145] like Fig. 9 The visual segmentation results on the test set are shown. The roads between farmlands are in the white boxes. They are generally small and curved, which requires the network to pay attention to multi-scale features. It can be clearly seen from the results that SAU2-Net has the best segmentation effect. According to the above typical image segmentation results, it can be concluded that SAU2-Net can effectively extract small-scale information and accurately extract the boundaries of easily confused land features.

[0146] Verification of the model’s generalization ability

[0147] In order to verify the generalization ability of the SAU2-Net model, the LoveDA dataset is used to compare it with other models commonly used in remote sensing. These are PSPNet

[42] , LinkNet

[47] , FarSeg

[48] , FactSeg

[49] , HRNet

[11] , DDRNet

[46] , TopFormer

[50] , and RTFormer

[17] . The experimental results are based on the results in the original article, and those marked with (*) are re-implemented locally. Table 5 shows that the model of the present invention has the best overall effect, especially in the bare land and farmland categories.

[0148] Table 5 IoU (%) and FPS of each category of different models on the LoveDA test set.

[0149]

[0150] Fig.10 This is the image segmentation result in the test set of the Loveda dataset. The lower left corner of the picture is the farmland area. Compared with other models, SAU2-Net has a smoother segmentation boundary between buildings and farmland. This proves that SAU2-Net's performance in ground object boundary segmentation has improved.

[0151] Ablation experiment

[0152] In order to verify the effectiveness of the two modules in improving the model, the two modules are decomposed to study their impact on the network separately. U2-Net means that only the original binary classification task is changed to multi-classification without adding any improved modules, AU2-Net means that FAM is used only after each upsampling in the decoding stage, and SAU2-Net means that the two modules of FSM and FAM are used, that is, the method proposed by the present invention. These three models are trained on complex scene object classification datasets to verify the effectiveness of the improved modules.

[0153] As shown in Table 6, after adding FAM, the evaluation index obtained has an improvement of 0.16%-2.52% compared with the improvement before. It can be concluded that using FAM alone can effectively improve the accuracy of the model in segmenting the type of objects, and using FSM before feature fusion can accurately locate fine-grained feature information, making FAM's improvement on the network more obvious. Mixing multiple resolution data together to retrain the network makes the model more versatile. It can be seen that all average indicators are above 80%, proving that the model can effectively extract the types of objects at two resolutions.

[0154] Table 6 Quantitative results (%) of ablation experiments on complex scene object classification dataset.

[0155]

[0156] As shown in Table 7, the IoU of the improved model and the original model in all categories is shown. From the data, it can be seen that the addition of FAM and FSM modules increases the segmentation IoU score, proving the effectiveness of the two modules. FAM narrows the gap between high-level semantic information and low-level semantic information, making the segmentation effect better. Compared with the original model, the model with the addition of FSM and FAM modules has an improvement of 1.46% in the forest class, 1.09% in the building class, and 1.06% in the road class. Forests and buildings are types of objects with large differences in texture information and scale between classes, and roads have different forms such as straight, curved, and narrow. The three categories are difficult to segment in the classification of objects, which fully proves the effectiveness of the two modules in improving the classification of objects.

[0157] Table 7 IoU (%) of each category in the ablation experiment on the complex scene object classification dataset.

[0158]

[0159] Gradient Weighted Class Activation Mapping (Grad-CAM) is a technique that can display the features learned by the neural network. We use this technique to visualize the gradients of the last convolutional layer of the three models to show the key areas of the segmented image. Fig.11 As shown, Grad-CAM generates a heat map on the original image. It can be seen that SAU2-Net pays more attention to small targets, and the heat map colors are darker in the small-scale water area in the first image and the buildings and curved roads in the second image. This further shows that FSM and FAM are better for feature extraction of multi-scale objects and segmentation details in remote sensing object type segmentation.

[0160] Model classification accuracy in different scenarios

[0161] Segmentation effect of the model at different spatial resolutions

[0162] like Fig.12 The following are the segmentation results of 2m and 1m spatial resolution images of the same area. Generally speaking, the higher the resolution, the clearer the details of the objects, and the better the segmentation effect. The improved model has improved the segmentation performance of roads and reduced the segmentation of many broken objects. SAU2-Net can correctly identify the buildings in the white box area in Figure (a), and the segmentation boundary of the trees outside the water area in Figure (b) is smoother.

[0163] like Fig.13The following is the segmentation result of a 0.3m image. Since we cannot obtain the labels in the test set, we evaluate the pros and cons of the method by observing the original image and the segmentation results of other models. In the segmentation results of the compared models, the narrow roads in the farmland are difficult to identify, and the farmland is easily confused with other forests. Since SAU2-Net can fuse multi-scale features and align misaligned features, better visualization effects are obtained at the segmentation boundaries and details.

[0164] Different spatial resolutions lead to obvious visual differences in objects. Higher spatial resolution makes the image clearer and the object division more detailed. Combining the object classification results under the above three spatial resolutions, it can be seen that the models designed by the present invention can effectively identify objects.

[0165] Segmentation effect under different terrain environments

[0166] The results of land feature classification in three types of landforms: mountainous area, rural area and urban area are as follows Fig.14 As shown in the figure. Overall, it is more difficult to identify farmland roads in mountainous areas than in plains. In row (a), it is more difficult to identify farmland in mountainous areas, and the improved model has better extraction effect. In row (b), there is confusion between the boundary between forest and farmland, and the improved model has a clearer segmentation boundary. In row (c), there is a small part of forest on the side of the road, which is mistakenly marked as farmland in the label. The improved model successfully identifies it, which shows that the model is fault-tolerant.

[0167] Effect of segmentation under different crop phenological periods (growth periods)

[0168] When different crops are planted in the farmland, the texture and color are different. Wheat has the darkest color and smooth texture, corn has obvious stripes, and when multiple crops are planted, the texture is blocky and the color is different. This is caused by the growth form of the crops themselves. In this invention, different crops and bare land are divided into farmland categories, and this category has large intra-category differences. Fig.15 SAU2-Net can effectively identify farmland in four forms. The white box in the figure is a clear contrast area of ​​the model segmentation effect after adding the improved module.

[0169] Remote sensing spatial distribution mapping of land feature types

[0170] In order to verify the segmentation effect of the model proposed in this invention in the actual county, the 9 county remote sensing images in the spatial distribution mapping test area were used for mapping experiments and the accuracy was calculated. The original image information was used to add geographic information to the prediction results, and the prediction results were post-processed to generate a spatial distribution map of the landform types. Two methods were used to generate accuracy test points for each county, namely, randomly creating 500 sampling points and generating fishing net sampling points with a pixel width and height of 1000m×1000m. Fig.16It is a schematic diagram of the sampling points. Each sampling point is manually marked and the prediction map result is extracted to calculate the accuracy. Table 8 shows the evaluation index values ​​obtained from the remote sensing image prediction mapping of each county in a specific period. The average producer accuracy is 88.48%, the average user accuracy is 93.28%, the average overall classification accuracy reaches 93.37%, and the kappa coefficient reaches 89.63%. The above results fully demonstrate the effectiveness of the high-resolution remote sensing feature type spatial distribution mapping method proposed in the present invention, and the semantic segmentation model trained with a small number of samples has good generalization.

[0171] Table 8 Evaluation results of cartographic accuracy in different periods in the study counties (%).

[0172]

[0173]

[0174] Fig.17 The following is a comparison of the details of the spatial distribution map before and after post-processing. Fig.16 (1) There are isolated noise points in the image, which are filled with surrounding pixel values ​​after post-processing; Fig.16 There are areas with poor road connectivity in (2), which are significantly improved after post-processing. Fig.18 The figure shows the spatial distribution map of remote sensing land object types in Qi County, Hebi City in two periods generated by the mapping method proposed in the present invention. The farmland feature of the remote sensing image on July 9, 2023 is bare land, and the color feature of the mountain is obviously green. The remote sensing image of wheat in the growth period on March 7, 2024 shows obvious farmland crop features, and the mountain forest features are not obvious. There are obvious feature differences between the two images, and the model can achieve effective recognition, proving that the generalization is good. It can be seen from the results that the segmentation effect of different land object types is good, and the segmentation edges are clear, which proves that the model proposed in the present invention is suitable for remote sensing land object classification tasks.

[0175] in conclusion

[0176] This paper uses orthophotos of 12 cities in Henan collected by GF-1, GF-2, and GF-6 satellites to construct a dataset of 6 types of land features in complex scenes, and designs a deep learning model SAU2-Net to automatically extract land feature information and conduct county-level mapping research. The following conclusions were drawn:

[0177] (1) In order to accurately extract the type information of land objects in remote sensing images in different scenarios, a feature selection and feature alignment U-type network (SAU2-Net) is proposed. This network is a simple bottom-up and top-down symmetrical structure that can predict and generate multi-scale features. It can accurately extract the information of land objects in different scenarios and provide an intuitive basis for the sustainable development of land resources and the prevention of the reduction of farmland.

[0178] (2) A feature selection module is added to the jump connection of the SAU2-Net model to enhance the spatial detail features. A feature alignment module is added to align the upsampled feature layer with the feature layer after feature selection to predict clearer ground object boundaries. In three sets of ablation experiments, it was found that the feature misalignment problem caused by multiple upsampling was improved, the boundary segmentation effect of farmland and other types was better, and the fine-grained feature extraction effect was effectively improved.

[0179] (3) The method of the present invention achieves mIoU of 79.31%, mRecall of 89.77%, and OA of 93.81% in complex scene object classification datasets. Compared with the best comparison model, the IoU of the road category increased by 4.1 percentage points, the forest category increased by 2.91 percentage points, and the water category increased by 1.46 percentage points. In the LoveDA dataset, the mIoU value reached 52.86%, which is 3.07 percentage points higher than HRNet, and the farmland category increased by 4.21 percentage points. SAU2-Net can more accurately identify small targets and the boundaries between farmland and other categories, which is crucial for the problem of object type segmentation.

[0180] (4) Input the remote sensing images in the actual county into the model proposed by the present invention for prediction, and obtain the block prediction results. The prediction results are spliced ​​and classified by the mapping method proposed by the present invention to obtain the county-level spatial distribution map of land feature types that meets the mapping requirements, and the overall classification accuracy reaches 93.37%.

[0181] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A remote sensing object classification extraction network and mapping method in complex scenes, characterized in that: The following steps are involved: S1: Acquire high-resolution satellite remote sensing image data, pre-process the images, and generate multi-scene ground object datasets; S2: Construct a two-layer U-shaped network including an encoder and a decoder for multi-scale feature extraction and object boundary segmentation; S3: Input the preprocessed remote sensing image into a two-layer U-shaped network for object classification and segmentation; S4: splicing the segmentation results and generating a spatial distribution map of the ground object types; S5: performing noise removal and post-processing operations on the generated distribution map.

2. According to the method for extracting remote sensing objects from complex scenes through classification network and mapping, the method is characterized in that: The high-resolution satellite remote sensing image data comes from GF-1, GF-2 and GF-6 satellites, and the image data includes multi-source remote sensing data with resolutions of 0.3 meters, 1 meter and 2 meters.

3. The remote sensing object classification extraction network and mapping method in complex scenes according to claim 1 is characterized in that: The encoder gradually extracts multi-scale features of the remote sensing image through downsampling, and the decoder restores the image spatial resolution through upsampling.

4. The remote sensing object classification extraction network and mapping method in complex scenes according to claim 1 is characterized in that: A feature selection module is added to the encoder, and the feature selection module is used to select fine-grained features from multi-scale features.

5. The remote sensing object classification extraction network and mapping method in complex scenes according to claim 1 is characterized in that: A feature alignment module is added to the decoder, and the feature alignment module is used to align the up-sampled high-level features with the low-level features in the encoding stage.

6. The remote sensing object classification extraction network and mapping method in complex scenes according to claim 1 is characterized in that: The stitching process uses a sliding window technique to crop the remote sensing image.

7. The remote sensing object classification extraction network and mapping method in complex scenes according to claim 1 is characterized in that: The voting method in the stitching process is used to process the pixel values ​​of multiple overlapping areas during stitching, and the final stitching result is determined by weighted calculation.

8. The remote sensing object classification extraction network and mapping method in complex scenes according to claim 1 is characterized in that: The post-processing operations include: Use majority filter to eliminate noise and isolated pixels in the segmentation result; Smooth the boundaries of features.

9. The remote sensing object classification extraction network and mapping method in complex scenes according to claim 5, characterized in that: The feature alignment module is based on deformable convolution technology to dynamically adjust the spatial offset between high-level and low-level features.

10. The remote sensing object classification extraction network and mapping method in complex scenes according to claim 1, characterized in that: The two-layer U-shaped network uses a cross entropy loss function and combines a stochastic gradient descent optimizer for model training.

Images