A monitoring system for natural resource changes

Abstract

The present invention discloses a monitoring system for natural resource changes, which is implemented based on WebGIS and includes a server side and a local browser side; the server side is deployed with a map service subsystem, a natural resource monitoring subsystem and a management subsystem. The natural resource monitoring subsystem includes a natural resource change monitoring module, which is configured to perform the following steps: obtaining an aerial remote sensing image collected by a drone; using PHOTOMOD software to perform single-piece image production on the aerial remote sensing image to obtain a plurality of single-piece image orthophotos; splicing a plurality of single-piece image orthophotos to obtain an orthophoto map of a target monitoring area; obtaining a historical orthophoto map of the target monitoring area; segmenting the orthophoto map based on a preset segmentation algorithm to obtain an image segmentation result; and obtaining a change detection result using a preset change analysis algorithm based on the historical orthophoto map and the image segmentation result.

Description

Technical Field

[0001] The invention belongs to the technical field of natural resource monitoring, and in particular relates to a monitoring system for natural resource changes. Background Art

[0002] Natural resources play an important role in human survival and production. However, with the rapid economic development, the degree of human exploitation and utilization of natural resources has been increasing, leading to the increasingly prominent problem of resource deterioration and exhaustion. Therefore, scientific and rational monitoring and management of natural resources is crucial to the sustainable development of mankind.

[0003] However, the applicant found that the current monitoring of natural resource changes faces many challenges and defects, including:

[0004] (1) Limited monitoring methods: Traditional human monitoring is time-consuming, labor-intensive and inefficient, and cannot meet the monitoring needs of large-scale natural resource changes. Human monitoring has limited coverage, low monitoring frequency, and cannot provide timely and comprehensive monitoring information.

[0005] (2) Limitations of remote sensing data: Satellite remote sensing data are affected by many factors, such as cloud cover, revisit cycle and climate factors, which lead to instability and uncertainty in data acquisition. The resolution and frequency of remote sensing data are relatively low, making it difficult to capture subtle and rapid changes in resource changes.

[0006] (3) Data processing and analysis challenges: Massive amounts of remote sensing data need to be efficiently processed and analyzed to extract useful information and discover patterns and trends in resource changes. However, current data processing technology and analysis methods are not yet mature enough, and there are problems such as slow data processing speed and low analysis accuracy.

[0007] In summary, there are many defects and challenges in the current monitoring of natural resource changes. It is necessary to improve the efficiency and accuracy of monitoring natural resource changes to provide better support for the management and monitoring of natural resources. Summary of the invention

[0008] The purpose of the present invention is to solve the above-mentioned technical problems and provide a monitoring system for natural resource changes.

[0009] In order to solve the above problems, the present invention is implemented according to the following technical solutions:

[0010] The present invention provides a monitoring system for natural resource changes, which is implemented based on WebGIS and includes a server side and a local browser side; the server side is deployed with a map service subsystem, a natural resource monitoring subsystem, and a management subsystem;

[0011] The natural resource monitoring subsystem includes a natural resource change monitoring module, which is configured to perform the following steps:

[0012] Acquire aerial remote sensing images collected by drones;

[0013] Using PHOTOMOD software to produce single-piece images of the aerial remote sensing images, and obtaining a plurality of single-piece image orthophotos;

[0014] The orthophotos of multiple single-piece images are stitched together to obtain the orthophotos of the target monitoring area;

[0015] Obtain historical orthophotos of the target monitoring area;

[0016] Segmenting the orthophoto image based on a preset segmentation algorithm to obtain an image segmentation result;

[0017] According to the historical orthophotos and image segmentation results, the change detection results are obtained using the preset change analysis algorithm.

[0018] Preferably, the preset segmentation algorithm includes a UPerNet network model, and the UPerNet network model adopts an FPN network with Swin Transformer as a backbone network in the feature extraction stage; Stage 4 in the Swin Transformer backbone network is connected to the PPM module of the UPerNet network model.

[0019] Preferably, the preset segmentation algorithm obtains the image segmentation result through the following segmentation steps:

[0020] The backbone network of Swin Transformer extracts the input image to be processed and constructs a multi-scale feature map;

[0021] The PPM module is used to perform pooling and convolution operations on the last layer of feature maps output by the backbone network of Swin Transformer, and then connect and fuse them with the feature maps of the same scale in the bottom-up branches, and then output feature maps of corresponding scales, thus forming a top-down multi-scale feature pyramid;

[0022] Use bilinear interpolation to adjust the feature map of each layer to 1 / 4 resolution, connect the corresponding layers, and then use the convolution layer to fuse the feature maps from different layers;

[0023] The result of feature map fusion is input into the detection head module. After a separate 3×3 convolution layer and classifier, the detection result is output to complete semantic segmentation.

[0024] Preferably, the preset change analysis algorithm comprises the following steps:

[0025] Acquire a historical image map of the target monitoring area, wherein the historical image map corresponds to the orthophoto map;

[0026] Input the historical image map and the orthophoto image map into the CNN backbone network, extract semantic features, and generate a historical image feature map and an orthophoto feature map of the same size as the original image;

[0027] The historical image feature map and the orthophoto feature map are respectively converted into a set of semantic label information, and global information modeling is performed through an encoder. After the global modeling information is processed by a decoder, it is mapped to the corresponding historical image feature map and orthophoto feature map to enhance the semantic feature representation;

[0028] Based on the enhanced historical image feature map and orthophoto feature map, a preliminary change prediction map is obtained by using the difference calculation method;

[0029] The image segmentation result is fused with the preliminary change prediction map to obtain the final single-element change detection result.

[0030] Preferably, the natural resource change monitoring module is configured to perform the following surface water monitoring steps:

[0031] Acquire optical satellite remote sensing images of the area to be monitored, wherein the optical satellite remote sensing images include 2m panchromatic images and 8m multispectral images of the same period;

[0032] The Pansharpening algorithm is used to fuse the 2m panchromatic image and the 8m multispectral image to obtain the current fused image with a spatial resolution of 2m.

[0033] Obtain historical fused images with a spatial resolution of 2m for the area to be monitored.

[0034] The historical fused images and current fused images of the area to be monitored are divided into 16 scenes at 1:50000, and a total of 16 historical images and 16 current images are obtained.

[0035] The trained DeepLabv3+ network is used to automatically extract the 16 scene images to obtain preliminary historical water body patches and preliminary current water body patches.

[0036] Compare the preliminary historical water body map with the preliminary current water body map to obtain the results of surface water body changes.

[0037] Preferably, the natural resource change monitoring module is configured to perform the following DeepLabv3+ network update steps:

[0038] Acquire optical satellite remote sensing images of the area to be monitored, wherein the optical satellite remote sensing images include 2m panchromatic images and 8m multispectral images of the same period;

[0039] The Pansharpening algorithm is used to fuse the 2m panchromatic image and the 8m multispectral image to obtain the current fused image with a spatial resolution of 2m.

[0040] Obtain historical fused images with a spatial resolution of 2m for the area to be monitored.

[0041] The historical fused images and current fused images of the area to be monitored are divided into 16 scenes at 1:50000, and a total of 16 historical images and 16 current images are obtained.

[0042] Randomly select 4 historical frame images and current frame images, as well as preliminary historical water body patches and preliminary current water body patches corresponding to the 4 historical frame images and current frame images;

[0043] Randomly select 2 historical frame images and a current frame image, and output them for manual trimming; wherein the 2 historical frame images and the current frame image used for manual trimming are different from the 4 historical frame images and the current frame image;

[0044] Obtain the historical framed images and the current framed images of the two scenes after artificial trimming, and obtain the real historical water body patches and the real current water body patches of the historical framed images and the current framed images of the two scenes;

[0045] The historical framed images and current framed images of the above four scenes, as well as the preliminary historical water body patches and preliminary current water body patches corresponding to the historical framed images and current framed images of the four scenes, are used to construct a training set;

[0046] The historical framed images and current framed images of the above two scenes, as well as the real historical water body patches and real current water body patches corresponding to the historical framed images and current framed images of the two scenes are used as the verification set;

[0047] Use the above training set and validation set to train the DeepLabv3+ network again.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] The present invention provides a natural resource change monitoring system, which is implemented based on WebGIS and includes a server side and a local browser side; the server side is deployed with a map service subsystem, a natural resource monitoring subsystem and a management subsystem.

[0050] In the present invention, the natural resource monitoring subsystem includes a natural resource change monitoring module, and the natural resource change monitoring module is configured to perform the following steps: obtaining aerial remote sensing images collected by a drone; using PHOTOMOD software to produce single-piece images of the aerial remote sensing images to obtain multiple single-piece image orthophotos; splicing multiple single-piece image orthophotos to obtain an orthophoto map of the target monitoring area; obtaining a historical orthophoto map of the target monitoring area; segmenting the orthophoto map based on a preset segmentation algorithm to obtain an image segmentation result; and obtaining a change detection result based on the historical orthophoto map and the image segmentation result using a preset change analysis algorithm.

[0051] The present invention utilizes a combination of advanced technologies such as drone aerial remote sensing imaging technology, GIS technology, and computer image interpretation technology to dynamically monitor natural resources through aerial photography, thereby effectively supervising with high efficiency and accuracy. It can timely discover and track changes in natural resources, provide important data support for resource management and protection, and provide better technical support for the management and monitoring of natural resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The specific embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings, wherein:

[0053] Figure 1 It is the technical roadmap of the monitoring system of the present invention;

[0054] Figure 2 is a platform architecture diagram of the monitoring system of the present invention;

[0055] Figure 3 This is a flow chart of the monitoring steps of the natural resource change monitoring module of the present invention:

[0056] Figure 4 It is a model architecture diagram of the UPerNet semantic segmentation network of the present invention;

[0057] Figure 5 It is a model structure diagram of the Swin Transformer network of the present invention;

[0058] Figure 6 It is a structural schematic diagram of a feature extraction module of the present invention;

[0059] Figure 7 is a schematic diagram of channel attention generation of the present invention;

[0060] Figure 8 is a schematic diagram of spatial attention generation of the present invention;

[0061] Fig. 9 It is a schematic diagram of the Semantic Tokens generation process of the present invention;

[0062] Fig.10 It is a schematic diagram of the structure of the input, encoding and output of the encoder of the present invention. DETAILED DESCRIPTION

[0063] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0064] At present, natural resource change monitoring faces many challenges and defects, including:

[0065] (1) Limited monitoring methods: Traditional human monitoring is time-consuming, labor-intensive and inefficient, and cannot meet the monitoring needs of large-scale natural resource changes. Human monitoring has limited coverage, low monitoring frequency, and cannot provide timely and comprehensive monitoring information.

[0066] (2) Limitations of remote sensing data: Satellite remote sensing data are affected by many factors, such as cloud cover, revisit cycle and climate factors, which lead to instability and uncertainty in data acquisition. The resolution and frequency of remote sensing data are relatively low, making it difficult to capture subtle and rapid changes in resource changes.

[0067] (3) Data processing and analysis challenges: Massive amounts of remote sensing data need to be efficiently processed and analyzed to extract useful information and discover patterns and trends in resource changes. However, current data processing technology and analysis methods are not yet mature enough, and there are problems such as slow data processing speed and low analysis accuracy.

[0068] In summary, there are many defects and challenges in the current monitoring of natural resource changes. It is necessary to improve the efficiency and accuracy of monitoring natural resource changes to provide better support for the management and monitoring of natural resources.

[0069] To this end, the present invention utilizes a combination of advanced technologies such as drone aerial remote sensing imaging technology, GIS technology, and computer image interpretation technology to dynamically monitor natural resources through aerial photography, thereby effectively supervising with high efficiency and accuracy. It can timely discover and track changes in natural resources, provide important data support for resource management and protection, and provide better technical support for the management and monitoring of natural resources.

[0070] In this paper, natural resources refer to various environmental factors and conditions that can be developed and utilized by humans under certain historical conditions to improve their welfare level or survival ability, have a certain scarcity, and are subject to social constraints. According to spatial distribution, they can be divided into ground resources, underground resources, and marine resources. According to the regeneration nature, they can be divided into renewable resources and non-renewable resources.

[0071] The natural resources mainly targeted by the monitoring system of the present invention include land resources, forest and grass resources, surface water resources, etc. It can also realize the monitoring of cross-border mining in mines, etc.

[0072] Embodiment 1

[0073] like Figure 1-2 As shown, the natural resource change monitoring system described in the present invention is implemented based on WebGIS, and the monitoring system includes a server side and a local browser side.

[0074] In the present invention, WebGIS (Web Geographic Information System) is a geographic information system that adopts B / S architecture operation mode. Compared with the traditional desktop geographic information system, it is based on the Internet, uses HTTP protocol, and the program runs on the server side and the local browser side respectively, getting rid of the platform limitations and improving the accessibility of geographic information services.

[0075] In the specific technology, the browser and the server exchange data through HTTP requests. The browser sends a service request to the GIS server. The GIS server receives and parses the service request from the browser, calls the relevant middleware to calculate and obtain the data for the response, and then returns it to the browser through the network. The browser then renders the returned data to the user. By adopting the WebGIS design, users can use the geographic spatial data application services published by the server anytime and anywhere through the browser.

[0076] In the present invention, the server side is deployed with a map service subsystem, a natural resource monitoring subsystem and a management subsystem.

[0077] In one implementation, mature WebGIS solutions can be used, such as the SuperMapIS.net platform provided by Beijing SuperMap, as well as open source solutions such as GeoServer.

[0078] In a specific implementation, Figure 1 The technical route of this monitoring system shown in Figure 2 The platform architecture of the system is shown.

[0079] In the present invention, the operation support layer includes the hardware and software facilities and network environment for the operation of the platform. The data layer includes remote sensing image tile data, the third national land survey data, geographic national conditions data, business data, real-life three-dimensional data, user data, etc. The service support layer is a supporting environment for supporting platform data storage, slicing, publishing, and program operation, including GeoServer, PostgreSQL, MySQL, Tomcat, Nginx, ArcGIS Server, etc. The service layer includes data resource services and platform general interface services. The data resource service is centered on compatible open space data protocols, and provides data management services, data processing services, feature query services, tile services, and permission services with interface interoperability capabilities. The platform general interface is used to uniformly encapsulate general basic functions, including query and analysis interfaces for front-end business function development and resource service interfaces for data access customization. At the business layer, the platform is based on business division and carries out thematic business construction for functions, namely, a three-dimensional natural resource map, remote sensing image data center, natural resource change monitoring, and natural resource dynamic supervision. At the presentation layer, users access the natural resource three-dimensional dynamic supervision platform through a browser.

[0080] Embodiment 2

[0081] The second embodiment provides a natural resource change monitoring system, and the system architecture and composition of the monitoring system are exactly the same as those of the first embodiment.

[0082] In this embodiment, if Figure 3 As shown in the flowchart, the natural resource monitoring subsystem of the present invention includes a natural resource change monitoring module, and the natural resource change monitoring module is configured to perform the following steps:

[0083] S100: Acquire aerial remote sensing images collected by drones.

[0084] S200: Using PHOTOMOD software to produce single-piece images of the aerial remote sensing image to obtain a plurality of single-piece image orthophotos.

[0085] S300: stitching together a plurality of single-image orthophotos to obtain an orthophoto of the target monitoring area.

[0086] S400: Acquire historical orthophoto images of the target monitoring area.

[0087] S500: Segmenting the orthophoto image based on a preset segmentation algorithm to obtain an image segmentation result.

[0088] S600: According to the historical orthophoto map and the image segmentation result, a change detection result is obtained using a preset change analysis algorithm.

[0089] Specifically, the present invention provides the following detailed descriptions of each step:

[0090] S100: Acquire aerial remote sensing images collected by drones.

[0091] In the present invention, aerial remote sensing images are obtained by real-time collection of target monitoring areas by drones, and are transmitted by drones or drone hangars to cloud servers for storage and for server call.

[0092] Through this design, the drone is flexible and maneuverable, less affected by air traffic control and climate, and can directly obtain photographic data in harsh environments. Even if the machine equipment fails, there will be no casualties, and it has high safety. The drone can fly at ultra-low altitudes under the clouds, making up for the defect of satellite remote sensing being affected by clouds, and can obtain images with higher resolution than satellite remote sensing. At the same time, low-altitude multi-angle photography can obtain high-resolution texture images of multiple sides of buildings.

[0093] On the other hand, drones fly at low altitudes of 50-1000m, which belongs to close-range aerial photogrammetry. The photogrammetry accuracy reaches sub-meter level, and the accuracy range is usually 0.1-0.5m, which greatly improves the efficiency and accuracy of operations for monitoring changes in natural resources.

[0094] In a specific implementation, before using a drone for aerial photography, it is necessary to base it on specific mission requirements, and at the same time be familiar with the natural geographical overview, topography and other conditions of the aerial photography area, and formulate a clear aerial photography plan based on the specific conditions of the aerial photography area. Before the start of aerial photography, the following specific things need to be done: 1) First, apply for the airspace of the aerial photography area; 2) Secondly, conduct on-site surveys to conduct safety risk assessments, route design and image control point layout, write flight plans, and formulate emergency plans; 3) Finally, try to choose weather with clear weather, sufficient light and atmospheric transparency for aerial photography to improve the quality of flight, the quality of photography and ensure the work cycle.

[0095] In a preferred implementation, considering that the lens of the UAV remote sensing device does not coincide with the mapping center during mapping, there will be deviation. Therefore, the lens error is calculated to compensate the mapping data to improve the accuracy of the data. The calculation formula is as follows:

[0096] V x =(xx 0 )(k 1 r 2 +k 2 r 4 +k 3 t 6 +L);

[0097] V y =(yy 0)(k 1 r 2 +k 2 r 4 +k 3 r 6 +L);

[0098] V z =(xx 0 )2(yy 0 )2

[0099] Where: Vx, Vy, Vz represent the deviations of the UAV remote sensing mapping in the horizontal, vertical and elevation directions; x and y represent the horizontal and vertical coordinates of the mapping point; x 0 ,y 0 Indicates the correction point deviation with the origin as the reference point; k 1 , k 2 , k 3 Represents the radial distortion coefficient. During the calculation process, usually only k is considered 1 ; r represents the image point radius. According to the above calculation results, the surveying and mapping data are preliminarily compensated.

[0100] Furthermore, based on the above, a linear regression equation is established, and this equation is used to process the remote sensing image as a whole. The equation is as follows: i =α+bX i +ε i ; Where: Y i represents the expression of the univariate linear regression processing equation; α represents the estimated value, and the value α is taken as a constant in the calculation; b represents the empirical regression coefficient; X i represents the linear prediction slope of the i-th feedback data; ε i Represents the random error of the i-th feedback data.

[0101] In a preferred implementation, the image control points are obtained through the previous imaging phase and then image control point joint measurement is performed, and the images taken by the UAV are sorted and subjected to aerial triangulation encryption processing according to the image control points obtained through the joint measurement.

[0102] In a specific implementation, three types of drones, namely, a P316 vertical take-off and landing fixed-wing drone equipped with remote sensing sensors, an X-Chimera composite wing drone, and a DJIPHANTOM 4RTK multi-rotor drone, are used to collect aerial data.

[0103] It should be noted that the layout of ground control points is not necessary. The images obtained by drone aerial photography have the characteristics of small frame and large number of photos. Then, they are quickly mosaicked through quick mosaic software, and the control points that meet the requirements are selected using the simple mosaic image map of the shooting area.

[0104] S200: Using PHOTOMOD software to produce single-piece images of the aerial remote sensing image to obtain a plurality of single-piece image orthophotos.

[0105] In a specific implementation, after the aerial image data is acquired, the image distortion is first corrected through aerial triangulation processing; secondly, the single-chip image is produced using PHOTOMOD software, and the single-chip result is corrected and evenly lit to form a single-chip image orthophoto; then, the processed single-chip image orthophotos in the monitoring area are spliced ​​to form an orthophoto map within the monitoring area; then, the orthophoto map within the monitoring area is image corrected and geometrically corrected to reduce the position error caused by terrain undulations; finally, the orthophoto map within the monitoring area is image modified, mosaicked, etc., to average the image position error and make sure there is no obvious color difference between different image areas, which is convenient for the application and management of later images.

[0106] In another preferred implementation, in view of the fact that the original image has distortions such as squeezing and twisting, it is very necessary to preprocess the original image. The preprocessing operation platform is ENVI software. First, the panchromatic image and multispectral image are orthorectified to correct the terrain distortion in the image; secondly, the multispectral image is geometrically corrected based on the panchromatic image to eliminate the geometric errors between the same-name points in the panchromatic image and the multispectral image; then the panchromatic image and the multispectral image are fused using the GS fusion method to improve the image interpretation accuracy and image quality; finally, the study area is cropped. Specifically:

[0107] (1) Orthorectification: Orthorectification of images requires the use of a terrain elevation model (DEM) to correct terrain distortion in the image so that the image meets the requirements of ortho projection. Orthorectification of aerial remote sensing images can correct the impact of geometric distortion caused by terrain on aerial remote sensing images. Based on RPC information, orthorectification is performed on one-meter panchromatic images and four-meter multispectral images. The operating platform of this experiment is ENVI. Combined with the cubic convolution resampling method, the RPC Orthorectification Workflow tool is used to orthorectify multispectral data and panchromatic data.

[0108] (2) Geometric correction: Geometric correction is to correct the geometric deformation such as squeezing, deformation, and distortion between the pixels on the image and the corresponding objects on the ground. Geometric correction can eliminate the geometric errors of the same-name pixels in the panchromatic image and the multispectral image. The multispectral image is corrected based on the panchromatic image of the aerial remote sensing image. This is achieved using the ImageRegistrationWorkflow tool in ENVI. The basic point of this operation is to fully align the multispectral image and the panchromatic image. First, manually or automatically select about 30 control points on the multispectral image and the panchromatic image, then check and delete the control points with large errors, and finally preview the corrected image and perform the correction operation.

[0109] (3) Data fusion and clipping

[0110] Remote sensing image fusion is beneficial to improving the temporal resolution and spatial resolution of the original image; remote sensing image fusion is beneficial to object recognition and information extraction; remote sensing image fusion improves image interpretation accuracy and image quality. The purpose of remote sensing image fusion is to integrate remote sensing information from different types of images to synthesize a new image with richer spatial features. For high-resolution images, Gram-Schmidt has a high image fidelity effect and is an efficient image fusion method.

[0111] S300: stitching together a plurality of single-image orthophotos to obtain an orthophoto of the target monitoring area.

[0112] In a specific implementation, orthophotos can be generated through ContextCapture, Pix4D, and PhotoScan, and then the generated orthophotos are imported into ArcGIS or Global Mapper for image stitching to obtain an orthophoto map of the target monitoring area.

[0113] S400: Acquire historical orthophoto images of the target monitoring area.

[0114] In the present invention, the historical orthophoto map and the orthophoto map are essentially remote sensing data from different periods, with the historical orthophoto map in the front and the orthophoto map in the back. By comparing the two, the monitoring of natural resource changes can be achieved.

[0115] In a specific implementation, the historical orthophoto map of the target monitoring area is historical data used for natural resource monitoring and has been stored in a server database. In another implementation, the historical orthophoto map can be historical aerospace remote sensing data or historical aerial remote sensing data.

[0116] S500: Segmenting the orthophoto image based on a preset segmentation algorithm to obtain an image segmentation result.

[0117] In the present invention, the preset segmentation algorithm includes a UPerNet network model, and the UPerNet network model adopts an FPN network with Swin Transformer as a backbone network in the feature extraction stage; Stage 4 in the Swin Transformer backbone network is connected to the PPM module of the UPerNet network model.

[0118] Specifically, the present invention describes the UperNet network model in detail as follows:

[0119] like Figure 4 As shown in the figure, it is the basic architecture of the basic UperNet network model. Its processing flow is as follows: (1) First, the backbone network in FPN extracts the input image and constructs a bottom-up multi-scale feature pyramid, and then the PPM module is added to the last layer of the backbone network. (2) Then, the PPM module is used to perform pooling and convolution operations on the last layer of feature maps output by the backbone network, and they are connected and fused with the feature maps of the same scale in the bottom-up branch, and then the feature maps of the corresponding scale are output, thereby forming a top-down multi-scale feature pyramid. (3) Then, in order to be able to fuse each layer of feature maps with different scales in the next step, UPerNet first adjusts each layer of feature maps to 1 / 4 resolution by using bilinear difference, and connects each corresponding layer, and then uses the convolution layer to fuse these feature maps from different layers, thereby reducing the number of channels. (4) Finally, the result of the feature map fusion is input into the detection head module, and after a separate 3×3 convolution layer and classifier, the detection result is output to complete the task of semantic segmentation.

[0120] In the natural resource segmentation and extraction task studied by the present invention, the types of surface objects in the study area are complex and diverse, usually including roads, land, rivers, vegetation, etc. Therefore, the UPerNet semantic segmentation network can be applied to such complex scenes with many types of objects and rich feature information. However, the applicant's research found that:

[0121] Although the UPerNet network model can achieve good segmentation effects on multi-level visual concepts in images in complex scenes, it still has the following difficulties when applied to the natural resource segmentation and extraction task of the present invention:

[0122] Compared with other images, drone remote sensing images have a high spatial resolution, resulting in richer ground object details and more complex and diverse feature information in the images. However, when extracting complex and large amounts of feature information from images, the backbone network of the UPerNet network model lacks effective capture of target feature information, has weak feature expression capabilities, and has low computational efficiency.

[0123] To address the above issues, the UperNet network model of this patent is an optimization of the basic UperNet network model. The core innovation is that the FPN network with Swin Transformer as the backbone network is used in the feature extraction stage. Figure 4 The bottom-up network structure on the left side is replaced by Figure 5 The improved model first uses Swin Transformer to extract and construct multi-scale hierarchical feature maps to represent the corresponding feature pyramid levels, then connects Stage4 in the Swin Transformer backbone network with the PPM module, and finally, after being processed by the PPM module, it is sent to the top-down branch of FPN and adopts the same downsampling rate as in the Swin Transformer backbone network. The subsequent network processing flow is consistent with the processing flow of the basic UPerNet network model. The feature pyramid structure of FPN in the UPerNet network model structure can be highly consistent with the multi-level feature expression structure obtained by the Swin Transformer backbone network, realizing the effective fusion of high, medium and low-level semantic information. At the same time, the multi-head self-attention mechanism module in the Swin Transformer network can enhance the global modeling ability and feature expression ability of the model while maintaining low computational complexity, and focus on the attention to target pixels such as ground objects and water bodies, effectively eliminating the influence of non-target noise on the network model, so that the improved network can achieve better semantic segmentation effect; it also has the following advantages:

[0124] (1) Using a hierarchical construction method similar to that used in CNN, a multi-scale, hierarchical feature expression can be constructed, and its receptive field gradually expands from shallow to deep layers, making the model suitable for pixel-level dense prediction visual tasks such as semantic segmentation;

[0125] (2) Self-attention is calculated locally in non-overlapping windows of the image to be segmented, and cross-window connections are introduced, which greatly reduces the computational complexity and improves the computational efficiency;

[0126] (3) The self-attention mechanism module based on moving windows is adopted to realize the transmission and interaction of information between different windows while maintaining low computational complexity, which greatly enhances the global modeling ability and feature expression ability of the model.

[0127] Compared with backbone networks such as ResNet and AlexNet, Swin Transformer as a backbone network can provide more compact feature expression and richer semantic information, thereby helping subsequent networks to better locate building target objects and enhance the modeling ability of the model while reducing computational complexity.

[0128] like Figure 5 As shown in the figure, first, the image of size [H, W, 3] is input into the block segmentation layer PatchPartition and divided into adjacent 4×4 pixels, and then flattened in the channel direction, so the image size becomes [H / 4, W / 4, 48]. Then, the network constructs feature maps of different scales through four consecutive stages: in Stage 1, the output result of the previous module is input into the linear embedding layer LinearEmbedding, and the channel data of each pixel is linearly transformed, the image size becomes [H / 4, W / 4, C], and then the image is input into the SwinTransformer module; the remaining three stages are different from Stage 1 in that they no longer need to pass through LinearEmbedding, but instead use a block merging layer PatchMerging to downsample the input feature map.

[0129] In another implementation, Stage 3 has 6 SwinTransformer modules instead of 2, which means that there are 3 pairs of Figure 5 (b) The structure shown in Figure 2. In the subsequent three stages, the image size will be changed to [H / 8, W / 8, 2C], [H / 16, W / 16, 4C], and [H / 32, W / 32, 8C], thereby constructing a multi-scale feature map.

[0130] Regarding the loss function of the UperNet network model of the present invention, the loss function refers to the difference between the predicted value and the true value of the training sample in the deep learning model. The smaller the difference, the smaller the loss, that is, the closer the predicted value is to the true value, the better the effect of the model.

[0131] In the present invention, a comprehensive loss function combining the binary cross entropy loss function BCE loss and Dice loss is used to improve the stability of model training. Comprehensive loss function: L = aL BCE +(1-a)L Dice ; Where a is the weight coefficient of BCE loss, its value range is [0,1], and it is determined to be 0.8.

[0132] (1) BCE loss: Cross entropy is mainly used to measure the difference between two probability distributions. The smaller the cross entropy value, the better the model effect. The definition of BCE loss is as follows: Among them, a is the model prediction value, y is the corresponding sample label true value, n is the total number of samples, and x is the sample value.

[0133] (2) Dice loss: The Dice coefficient is a widely used metric in the field of computer vision. It is mainly used to measure the similarity between two images. Its calculation formula is as follows: Among them, A and B represent the data sets of the model prediction results and the sample true value respectively, and the Dice coefficient is the similarity of the two data sets, and its value range is [0,1]. The larger the Dice value, the higher the overlap of the two data sets, the more similar they are, that is, the better the model effect. Define Diceloss as:

[0134] In a specific implementation, the preset segmentation algorithm obtains the image segmentation result through the following segmentation steps:

[0135] S510: The backbone network of Swin Transformer extracts the input image to be processed and constructs a multi-scale feature map;

[0136] S520: using the PPM module to perform pooling and convolution operations on the last layer feature map output by the backbone network of the Swin Transformer, connect and fuse it with the feature map of the same scale in the bottom-up branch, and then output the feature map of the corresponding scale, thereby forming a top-down multi-scale feature pyramid;

[0137] S530: using bilinear interpolation to adjust each layer of feature maps to 1 / 4 resolution, and connecting each corresponding layer, and then using a convolutional layer to fuse the feature maps from different layers;

[0138] S540: The result of the feature map fusion is input into the detection head module, and after passing through a separate 3×3 convolution layer and classifier, the detection result is output to complete the semantic segmentation.

[0139] S600: According to the historical orthophoto map and the image segmentation result, a change detection result is obtained using a preset change analysis algorithm.

[0140] In a specific implementation, the preset change analysis algorithm includes the following steps:

[0141] S610: Acquire a historical image map of the target monitoring area, where the historical image map corresponds to the orthophoto map.

[0142] In a specific implementation, the correspondence means that the historical image map and the orthophoto image map are remote sensing monitoring data of the target monitoring area in different periods.

[0143] S620: Input the historical image map and the orthophoto image map into the CNN backbone network, extract semantic features, and generate a historical image feature map and an orthophoto feature map of the same size as the original image.

[0144] In the specific implementation, the network structure for extracting features is as follows Figure 6 As shown in the figure, the feature extraction module uses the encoder of FRUNetbase proposed in this paper as the backbone network, followed by a two-fold upsampling and 1×1 convolution structure, and finally outputs a feature map F containing deeper information. In this way, the problem of losing some feature information due to the feature map being too small can be avoided, and the receptive field can be expanded to a certain extent.

[0145] In order to further represent the feature map, that is, to make the feature map highlight the area of ​​concern, the present invention adds a combination module of channel attention and spatial attention.

[0146] (1) The input of the channel attention generation module is the feature map F of the final output of the feature extraction network in the previous step. We obtain C×1×1 vectors through maximum pooling and average pooling respectively, and then use the same shallow MLP module with two convolutions to obtain two C×1×1 vectors. We add the two vectors as the input of the sigmoid activation function, making their mean 0 and variance 1, and finally get the channel attention. The calculation of adding the generated channel attention to the feature map is as follows: Features is the extracted feature map, W c (Features) is the channel attention weight parameter for network training, That is, all pixels of the feature map of each channel are multiplied by the corresponding weights, and Features′ is the feature map with channel attention added.

[0147] (2) Spatial attention: After a 1×1 convolution on the original C×1×1 feature map to obtain an M×H×W feature map (M H×W spatial attention weights), each channel uses the sigmoid activation function to generate a spatial attention matrix. The calculation method for adding the generated spatial attention to the feature map is as follows: W s (Features′) is the spatial attention weight parameter for network training, That is, the same pixel in all feature maps that obtain channel attention is multiplied by the corresponding weight, and Features′ is a map with channel attention and spatial attention added.

[0148] S630: Convert the historical image feature map and the orthophoto feature map into a set of semantic label information respectively, and perform global information modeling through an encoder. After the global modeling information is processed by a decoder, it is mapped to the corresponding historical image feature map and orthophoto feature map to enhance the semantic feature representation.

[0149] like Figure 8 As shown in Figure 1, the specific process of adding channel attention and spatial attention mechanisms and converting them into tokens is as follows: First, add channel attention to the original feature map to obtain Features CA , then the feature map of the spatial attention is transformed into M vectors of length HW after dimensionality reduction, and then combined with Features CA Multiply them together to finally get K semantic tags (SemanticTokens), which are used as the input of the encoder, similar to the feature vectors representing each word in the target sentence in the text translation task.

[0150] Tokens represents the feature vector of the image, which is calculated as follows:

[0151] MDR represents the feature map F that will receive channel attention. CA Expanded into matrix Me CA , MDR′ represents the spatial attention weight matrix W SA Expanded into matrix Me SA , Multiplying the two matrices together yields a semantic information label (a set of M sub-label vectors of length C).

[0152] Then, the semantic feature information-rich tokens obtained in the above steps are output to the encoder, and the features are analyzed for association through its self-attention mechanism (there are differences in the features represented by different geographies). The encoder input is the original feature map before and after and the output of the encoder. The information of each position in the original feature map is processed similarly to text translation, making the vector expression of the difference more significant, and finally outputting two transformed feature maps. The encoder structure is as follows: Fig.10 shown.

[0153] The task of the decoder is similar to the work of translation. The input is the serialization result of the original feature map, which is converted into a Q tensor through a linear transformation function. The tokens output by the encoder are converted into K and V tensors respectively, and then calculated through a multi-head cross attention module and a forward propagation module. Finally, after a linear transformation and an activation function, the output is a feature result map with enhanced differences that is consistent with the size of the feature map. The calculation is as follows:

[0154] MCA(F i Tokens new )=COncat(head 1 head 2 , ..., head N )W A ;

[0155]

[0156] Where N is the number of attention heads, is the linear mapping parameter matrix.

[0157] In the present invention, the key point of the encoder is the self-attention mechanism (Multi-Head SelfAttention), the main idea of ​​which is: first, each linear input is mapped into three tensors, namely Q (Query), K (Key), and V (Value), through a Linear function, which is calculated as follows:

[0158] Q=Tokens i W q

[0159] K=Tokens i W k

[0160] V=Tokens i W v ;

[0161] Among them, W q , W k , W v are three trainable parameter weight matrices, Q, K, V dimensions and tokens i For one of the input Tokens, the attention is calculated from all input Q, K and the current V respectively, and the calculation is as follows: The similarity between them is expressed by the dot product of Q and K. The purpose is to transform the attention matrix into a standard state distribution, normalized by softmax, so that the sum of the attention weights of each input and other inputs is 1.

[0162] S640: On the enhanced historical image feature map and the orthophoto image feature map, a preliminary change prediction map is obtained by using a difference calculation method;

[0163] In the specific implementation, the early and late aerial remote sensing images belonging to the same target monitoring area are output as two enhanced feature map sets after passing through the feature extraction module and the encoding and decoding module. On this basis, the difference calculation is performed on each feature map in the two feature sets, and the calculation is as follows: Represents the pixel value of the coordinate (i, j) on the c channel in the a, b and r feature maps respectively. The difference result is obtained by taking the difference and absolute value of the pixel values ​​of the corresponding channels and positions in a and b. After the difference calculation, the feature map is passed through a shallow convolution structure and a softmax function to generate a preliminary single-channel difference result map.

[0164] S650: Fusing the image segmentation result with the preliminary change prediction map to obtain a final single-element change detection result.

[0165] In the specific implementation, the output result only represents the difference part of the two remote sensing images. In order to realize the single-element change detection in the remote sensing image, the preliminary single-channel difference result map and the image segmentation result map are jointly judged, that is, only the areas that meet the two conditions of existence of changes and belonging to the image segmentation results are retained, and finally the single-element change detection result map is obtained.

[0166] In order to further improve the efficiency of change detection, the preliminary change detection result map is first judged among the image segmentation result maps, that is, its average pixel value is calculated. When the average pixel value is 0, it means that there is no change spot. At this time, there is no need to perform subsequent image segmentation and single-element change detection, and the preliminary change detection result is directly used as the final result.

[0167] Embodiment 3

[0168] This implementation three provides a natural resource change monitoring system, and the system architecture and composition of the monitoring system are exactly the same as those of implementations one and two.

[0169] In this embodiment, the natural resource change monitoring module is configured to perform the following surface water monitoring steps:

[0170] S1000: Acquire optical satellite remote sensing images of the area to be monitored, wherein the optical satellite remote sensing images include 2m panchromatic images and 8m multispectral images of the same period.

[0171] In the present invention, it is necessary to perform data preprocessing on the optical satellite remote sensing image, including orthorectification, registration fusion, regional adjustment, mosaic uniform color, downsampling and cutting. Among them, the 2m panchromatic image and the 8m multispectral image are fused using the Pansharpening algorithm to obtain a fused image with a spatial resolution of 2m. Data orthorectification, registration fusion, regional adjustment, and downsampling are completed using GXL software, and finally basic image data with the same geographic coordinates, consistent geometric position, and uniform color tone are obtained.

[0172] S2000: Use the Pansharpening algorithm to fuse the 2m panchromatic image and the 8m multispectral image to obtain the current fused image with a spatial resolution of 2m;

[0173] S3000: Obtain the historical fusion image of the area to be monitored with a spatial resolution of 2m;

[0174] S4000: The historical fused image and the current fused image of the area to be monitored are divided into 16 scene frames according to the 1:50000 frame, and a total of 16 historical frame images and 16 current frame images are obtained;

[0175] S5000: Use the trained DeepLabv3+ network to automatically extract 16 scene images to obtain preliminary historical water body maps and preliminary current water body maps.

[0176] In the present invention, the domestically produced GF-1 series optical satellite remote sensing images are used as the basic data source.

[0177] In the present invention, DeepLabv3+, which is widely used in deep learning semantic segmentation, is used as the basic network architecture. DeepLabv3+ is an encoder-decoder structure. The encoder is a deep convolutional network with an atrous spatial pyramid pooling (ASPP) module with dilated convolutions. This design enables DeepLabv3+ to effectively identify objects of different spatial scales in the image. Considering the multi-scale characteristics of lakes in remote sensing images, this paper uses DeepLabv3+ as a framework to extract water targets in GF1 series satellite images.

[0178] The DeepLabv3+ model structure includes an encoder and a decoder. Assume that the network input X is a remote sensing image of size N×N (the number of bands is 3, and the image width and height are both N pixels). For the convenience of description, this paper records the scale ratio of the input image to the feature map in the network as δ. For the encoding module, the input image is downsampled to achieve feature extraction. Specifically, the input image X first passes through a backbone network. The backbone network can choose any mainstream network, such as ResNet, Xception, etc. This paper uses the ResNet18 network as the backbone network of the encoder to reduce the dimension to obtain a feature map of size N16×N16 with 512 channels. Then, the feature map is passed through the ASPP module to introduce multi-scale information. The ASPP module mainly includes a 1×1 convolution layer, 3 3×3 dilated convolutions, and a global average pooling layer. Features of different scales are merged in the feature dimension, and then sent to the 1×1 convolution for feature fusion, thereby obtaining a new feature of 256-dimensional N16×N16.

[0179] For the decoding module, the feature map is upsampled to restore the segmentation map of the same size as the input image. Specifically, the feature map is firstly bilinearly upsampled with a scaling factor of 4 to obtain a 256-dimensional 4×N4 output feature (δ=4); then, the low-level features (such as Figure 3 The Low-LevelFeatures shown in the figure are merged to obtain a 256-dimensional N4×N4 output feature (δ=4); then, a 3×3 convolution and a bilinear upsampling with a scaling factor of 4 are used to obtain the final segmentation result. The output image of the network is a single-band binary classification result image with the same size as the input image. Among them, the pixels with a value of 1 are water target pixels recognized by the network, and the pixels with a value of 0 are background pixels recognized by the network.

[0180] S6000: Compare the preliminary historical water body map with the preliminary current water body map to obtain the surface water body change results.

[0181] In a specific implementation, the area size of the preliminary historical water body map and the preliminary current water body map is calculated and the numerical difference between the two is compared to obtain the surface water body change result. The preliminary historical water body map and the preliminary current water body map can also be overlapped to obtain a non-overlapping area, that is, the surface water body change area;

[0182] In a preferred implementation, the natural resource change monitoring module is configured to perform the following DeepLabv3+ network update steps:

[0183] S01: Acquire optical satellite remote sensing images of the area to be monitored, wherein the optical satellite remote sensing images include 2m panchromatic images and 8m multispectral images of the same period;

[0184] S02: Use the Pansharpening algorithm to fuse the 2m panchromatic image and the 8m multispectral image to obtain the current fused image with a spatial resolution of 2m;

[0185] S03: Acquire a historical fused image of the area to be monitored with a spatial resolution of 2 m.

[0186] S04: Divide the historical fused image and the current fused image of the area to be monitored into 16 scene frames according to 1:50000 frames, and obtain 16 historical frame images and 16 current frame images in total;

[0187] S05: Randomly select 4 historical frame images and current frame images, as well as preliminary historical water body patches and preliminary current water body patches corresponding to the 4 historical frame images and current frame images;

[0188] S06: randomly selecting two historical framed images and a current framed image, and outputting them for manual trimming; wherein the two historical framed images and the current framed image used for manual trimming are different from the four historical framed images and the current framed image;

[0189] S07: Acquire the historical framed images and the current framed images of the two scenes after artificial trimming, and obtain the real historical water body patches and the real current water body patches of the historical framed images and the current framed images of the two scenes;

[0190] S08: constructing a training set by using the above four historical framed images and current framed images, as well as the preliminary historical water body patches and preliminary current water body patches corresponding to the four historical framed images and current framed images;

[0191] S09: taking the above two historical framed images and the current framed images, as well as the corresponding real historical water body patches and real current water body patches of the two historical framed images and the current framed images, as a verification set;

[0192] S10: Use the above training set and validation set to train the DeepLabv3+ network again.

[0193] For other structures of the natural resource change monitoring system described in this embodiment, refer to the prior art.

[0194] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Therefore, any modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. A monitoring system for natural resource changes, characterized in that: The monitoring system is implemented based on WebGIS, and includes a server side and a local browser side; the server side is deployed with a map service subsystem, a natural resource monitoring subsystem, and a management subsystem; The natural resource monitoring subsystem includes a natural resource change monitoring module, which is configured to perform the following steps: Acquire aerial remote sensing images collected by drones; Using PHOTOMOD software to produce single-piece images of the aerial remote sensing images, and obtaining a plurality of single-piece image orthophotos; The orthophotos of multiple single-piece images are stitched together to obtain the orthophotos of the target monitoring area; Obtain historical orthophotos of the target monitoring area; Segmenting the orthophoto image based on a preset segmentation algorithm to obtain an image segmentation result; According to the historical orthophotos and image segmentation results, the change detection results are obtained using the preset change analysis algorithm.

2. A natural resource change monitoring system according to claim 1, characterized in that: The preset segmentation algorithm includes a UPerNet network model, which uses an FPN network with a Swin Transformer as a backbone network in the feature extraction stage; Stage 4 in the Swin Transformer backbone network is connected to the PPM module of the UPerNet network model.

3. A natural resource change monitoring system according to claim 2, characterized in that: The preset segmentation algorithm obtains the image segmentation result through the following segmentation steps: The backbone network of Swin Transformer extracts the input image to be processed and constructs a multi-scale feature map; The PPM module is used to perform pooling and convolution operations on the last layer of feature maps output by the backbone network of Swin Transformer, and then connect and fuse them with the feature maps of the same scale in the bottom-up branches, and then output feature maps of corresponding scales, thus forming a top-down multi-scale feature pyramid; Use bilinear interpolation to adjust the feature map of each layer to 1 / 4 resolution, connect each corresponding layer, and then use the convolution layer to fuse the feature maps from different layers; The result of feature map fusion is input into the detection head module. After a separate 3×3 convolution layer and classifier, the detection result is output to complete semantic segmentation.

4. A natural resource change monitoring system according to claim 3, characterized in that: The preset change analysis algorithm comprises the following steps: Acquire a historical image map of the target monitoring area, wherein the historical image map corresponds to the orthophoto map; Input the historical image map and the orthophoto image map into the CNN backbone network, extract semantic features, and generate a historical image feature map and an orthophoto feature map of the same size as the original image; The historical image feature map and the orthophoto feature map are respectively converted into a set of semantic label information, and global information modeling is performed through an encoder. After the global modeling information is processed by a decoder, it is mapped to the corresponding historical image feature map and orthophoto feature map to enhance the semantic feature representation; Based on the enhanced historical image feature map and orthophoto feature map, a preliminary change prediction map is obtained by using the difference calculation method; The image segmentation result is fused with the preliminary change prediction map to obtain the final single-element change detection result.

5. A natural resource change monitoring system according to claim 1, characterized in that: The natural resource change monitoring module is configured to perform the following surface water monitoring steps: Acquire optical satellite remote sensing images of the area to be monitored, wherein the optical satellite remote sensing images include 2m panchromatic images and 8m multispectral images of the same period; The Pansharpening algorithm is used to fuse the 2m panchromatic image and the 8m multispectral image to obtain the current fused image with a spatial resolution of 2m. Obtain historical fused images with a spatial resolution of 2m for the area to be monitored. The historical fused images and current fused images of the area to be monitored are divided into 16 scenes at 1:50000, and a total of 16 historical images and 16 current images are obtained. The trained DeepLabv3+ network is used to automatically extract the 16 scene images to obtain preliminary historical water body patches and preliminary current water body patches. Compare the preliminary historical water body map with the preliminary current water body map to obtain the results of surface water body changes.

6. A natural resource change monitoring system according to claim 5, characterized in that: The natural resource change monitoring module is configured to perform the following DeepLabv3+ network update steps: Acquire optical satellite remote sensing images of the area to be monitored, wherein the optical satellite remote sensing images include 2m panchromatic images and 8m multispectral images of the same period; The Pansharpening algorithm is used to fuse the 2m panchromatic image and the 8m multispectral image to obtain the current fused image with a spatial resolution of 2m. Obtain historical fused images with a spatial resolution of 2m for the area to be monitored. The historical fused images and current fused images of the area to be monitored are divided into 16 scenes at 1:50000, and a total of 16 historical images and 16 current images are obtained. Randomly select 4 historical frame images and current frame images, as well as preliminary historical water body patches and preliminary current water body patches corresponding to the 4 historical frame images and current frame images; Randomly select 2 historical frame images and a current frame image, and output them for manual trimming; wherein the 2 historical frame images and the current frame image used for manual trimming are different from the 4 historical frame images and the current frame image; Obtain the historical framed images and the current framed images of the two scenes after artificial trimming, and obtain the real historical water body patches and the real current water body patches of the historical framed images and the current framed images of the two scenes; The historical framed images and current framed images of the above four scenes, as well as the preliminary historical water body patches and preliminary current water body patches corresponding to the historical framed images and current framed images of the four scenes, are used to construct a training set; The historical framed images and current framed images of the above two scenes, as well as the real historical water body patches and real current water body patches corresponding to the historical framed images and current framed images of the two scenes are used as the verification set; Use the above training set and validation set to train the DeepLabv3+ network again.

Images