A high-resolution remote sensing target accurate detection method integrating semantic segmentation and edge

Abstract

A method for accurate detection of high-resolution remote sensing targets that integrates semantic segmentation and edges. First, samples are drawn according to the contour of the target object, and edge detection samples and semantic segmentation samples are produced. These samples are sent to the edge detection network and the semantic segmentation network for training to a fitting state, respectively, to obtain an edge detection model and a semantic segmentation model. The high-resolution remote sensing image is input into the trained edge detection model to obtain an edge intensity map of the object target, and the high-resolution remote sensing image is input into the trained semantic segmentation model to obtain semantic information and position information of the object target. Finally, the semantic segmentation results obtained are integrated, i.e., the accurate semantic type and position information of the object target, the processed edge intensity map of the object target is supplemented, the interrupted edge is repaired, and finally the accurate boundary of the object target is obtained, and the target boundary is vectorized into a polygonal element according to the task requirements. The present invention realizes accurate edge extraction of the object target in the high-resolution remote sensing image.

Description

Technical Field

[0001] The present invention relates to the technical field of high-resolution remote sensing image processing and the technical field of edge detection and semantic segmentation in the field of computer vision, and specifically to a method for realizing accurate detection of high-resolution remote sensing targets, which is suitable for accurate detection of ground objects in high-resolution remote sensing images. Background Art

[0002] In recent years, with the rapid development of earth observation technology, the large-scale preparation and application of high-spatial-resolution remote sensing satellites, supplemented by the earth observation of aircraft and drones, the remote sensing field has gradually entered the era of big data. The ground object information of high-resolution remote sensing images is richer and more accurate than that of medium- and low-resolution remote sensing images, and has greater practical significance in practical applications. How to effectively use its rich spatial spectrum, shape and texture feature information through various technologies to accurately interpret and analyze high-resolution remote sensing images, thereby avoiding the negative impact caused by redundant information, noise, etc. in practical applications, and more efficiently and accurately completing tasks such as classification, recognition and extraction of high-resolution remote sensing images has become the most challenging research frontier problem in the field of remote sensing. Inspired by deep learning and computer vision methods, the present invention uses semantic segmentation models and edge detection models to respectively extract semantic information and edge information of ground object targets in high-resolution remote sensing images, and fuses the semantic segmentation results and edge detection results to produce accurate target detection results that conform to human visual habits.

[0003] The edge of an object is the part of an image where the pixel value changes dramatically. It exists between the object and the background, and between objects. The deep learning edge detection algorithm extracts the edge information of the object to accurately locate the precise boundary of the object. Compared with the traditional object detection algorithm that only detects the bounding box of the object, the accurate edge of the object can better represent the real object. The wave of deep learning emphasizes the importance of automatic hierarchical feature learning in edge detection. RCF uses a deep learning model of a fully convolutional neural network and a deep supervision network. It uses the features of all convolutional layers and the multi-scale and multi-level information of the object to combine all meaningful target features to make predictions from image to image, effectively solving the inaccuracy and ambiguity problems in edge and target boundary detection. However, in high-resolution remote sensing images, due to resolution and imaging reasons, some ground object edges are difficult to provide effective semantic information. Even due to factors such as shadows or cloud occlusion, the convolutional neural network cannot accurately identify the edge of the ground object, and cannot guarantee the continuity and closure of the edge of the ground object, resulting in a large number of fragmented edges and edge disconnection. In actual work, the edge disconnection situation cannot be used as a complete extraction of the target, so how to complete the disconnection repair to improve the extraction accuracy of the target edge is the goal of the present invention.

[0004] With the introduction and rapid development of concepts such as semantic segmentation and instance segmentation in the field of computer vision, new solutions have been proposed for the extraction of ground objects in high-resolution remote sensing images. The semantic segmentation algorithm further determines the target category based on the semantic information of the target, and predicts the target mask. The target masks of the same semantic category are given the same color to achieve semantic target extraction. The fully convolutional network (FCN) uses the powerful feature learning ability of the convolutional neural network (CNN) on the image to propose a concept of full convolution, replacing all the fully connected layers of the network with convolutional layers, so that the final output result is a predicted image, achieving end-to-end semantic segmentation. U-Net expands and modifies the network structure of FCN, which can use less training data and combine the semantic information and feature information of the bottom and high layers to provide more refined features for target segmentation and achieve more accurate results. Mask R-CNN adds a branch in the region of interest (RoI) to predict the segmentation mask, and expands in parallel with the target classification and target detection bounding box regression to improve the accuracy of target feature extraction while retaining the precise position space of the target. With the successive introduction of network models such as FCN, U-Net, and Mask-RCNN, the semantic segmentation results of targets have achieved increasingly higher accuracy. However, semantic segmentation models are currently mostly used for natural image processing. Compared with natural images, remote sensing images have richer spatial spectrum, texture features and other complex information, which leads to inaccurate target categories and target boundaries when semantic segmentation models extract ground objects from remote sensing images. There are deviations from the actual ground object boundaries, making it difficult to apply directly. Summary of the invention

[0005] The present invention aims to overcome the problems of inaccurate boundary extraction of ground object targets in semantic segmentation, discontinuity and non-closure of ground object target edges extracted by edge detection, a large number of fine edges and broken lines in the prior art, and realize fine extraction of ground object target edges.

[0006] The semantic segmentation model provides the accuracy of the semantic category and position accuracy of the target, and the edge detection model provides relatively accurate edge information. The semantic segmentation prediction results and edge detection prediction results are integrated to improve the accuracy of the extracted ground object target type and edge, and finally achieve accurate edge extraction of ground object targets in high-resolution remote sensing images. The technical solution of the present invention includes the processes of deep learning network model construction, training, prediction, post-processing, etc.

[0007] The method for accurately detecting high-resolution remote sensing targets by integrating semantic segmentation and edge includes the following steps:

[0008] Step 1: Select a semantic segmentation network model and an edge detection network model suitable for extracting ground objects from high-resolution remote sensing images;

[0009] Step 2: Create high-resolution remote sensing image samples, manually draw the precise boundaries of the sample objects and determine their semantic types, and generate corresponding line label samples and surface label samples;

[0010] Step 3: Use the line label samples in step 2 to train the edge detection model, and use the surface label samples to train the semantic segmentation model until they are in a fitting state;

[0011] Step 4: Use the edge detection model and semantic segmentation model trained in step 3 to predict the high-resolution remote sensing images of the production area, and obtain the edge intensity map and semantic segmentation prediction map of the ground objects in the area;

[0012] Step 5: Fuse the semantic segmentation prediction map and edge strength map in step 4, process the wrong target boundaries and missed target boundaries in the edge strength map, extract the ground object target edges predicted by semantic segmentation, and repair the broken lines in the edge strength map to obtain a more accurate target edge map;

[0013] Step 6: Use the target edge map obtained after step 5 to perform vectorization processing, construct the polygon element vector corresponding to the ground object target, and complete the accurate extraction of the ground object target boundary.

[0014] To supplement the above steps:

[0015] In step 1, according to the characteristics of high-resolution remote sensing images, the semantic segmentation network model can be Mask RCNN or D-Linknet, and the edge detection network model can be RCF or BDCN.

[0016] When selecting samples in step 2, multiple areas of the same size are cut from the large-area high-resolution remote sensing image. The area should contain a variety of ground objects. After selecting the samples, outline the boundaries of the ground objects and generate binary line label samples and surface label samples. If the sample data volume is small, data enhancement methods can be used to expand the sample set.

[0017] In step 3, the edge detection model is trained using line label samples, and the semantic segmentation model is trained using face label samples. Appropriate parameters and hyperparameters need to be selected during training, and the model is trained until it reaches a fitting state.

[0018] In step 4, the high-resolution remote sensing images of the area to be produced are respectively input into the trained semantic segmentation model to obtain the semantic segmentation prediction map and edge strength map.

[0019] In step 5, the prediction map obtained in step 4 needs to be processed to achieve the requirement of accurate edge extraction. The specific steps are as follows:

[0020] Step 5.1: Select a suitable threshold to binarize the semantic segmentation prediction map;

[0021] Step 5.2: Perform median filtering on the edge intensity map, select a suitable threshold for binarization, and use the skeleton extraction method to refine it to a single pixel width. The median filter formula is as follows, where f(x, y) and g(x, y) are the original image and the processed image, respectively. W is a two-dimensional template, usually a 3*3 area, and med represents the median filter operation.

[0022] g(x,y)=med{f(xk,yl),(k,l∈W)}

[0023] Step 5.3: Perform connected domain analysis on the single-pixel width edge map obtained in step 5.2 to remove some redundant and short broken lines;

[0024] Step 5.4: The edge map processed in step 5.3 is superimposed on the binary semantic map obtained in step 5.1, and the boundary of the semantic map is used to supplement the edge map. At this time, most of the broken lines form a closed surface. The watershed algorithm is used to extract the edge of the closed surface, and the line segments that have not been closed are retained.

[0025] Step 5.5: There are still some broken lines in the edge graph after step 5.4. Use the eight-neighborhood algorithm to find the breakpoints and use the shortest path connection algorithm to form a closed edge.

[0026] Step 5.6: The edge map obtained in step 5.5 may still have some broken lines or redundant lines, which can be identified and deleted using the eight-neighborhood algorithm.

[0027] Step 5.7: Fill the holes in the edge map obtained in step 5.6 and overlay the single-pixel edge map in step 5.2 to retain important edge information and use the watershed algorithm to obtain the edge. The hole filling formula is as follows, A is the original image, A c is the complement of image A, B is the structural element, usually 3*3 in size, X k represents the kth element that needs to be filled, and ⊕ represents a B to X k-1 The expansion operation of , which represents the filling process of an element.

[0028]

[0029] Step 6: Vectorization processing. The edge map obtained in the above steps has basically met the requirements for accurate detection of ground objects. Finally, vectorization is performed to generate a vector line layer.

[0030] According to the above operation plan, accurate detection of high-resolution remote sensing targets integrating semantic segmentation and edges is achieved.

[0031] The present invention respectively integrates the advantages of semantic segmentation model and edge detection model to extract the edge of ground object target in high-resolution remote sensing image. The semantic segmentation model is used to segment the remote sensing image to obtain accurate target ground object type and location information, and the edge detection model is used to accurately extract the edge of the ground object target, and the target semantic information and target edge information are integrated, which overcomes the problems of inaccurate ground object target boundary extraction in semantic segmentation and discontinuity and non-closure of ground object target extraction edges and a large number of fine edges in edge detection, and finally realizes accurate detection of ground object targets in high-resolution remote sensing images.

[0032] The present invention has the following advantages: the present invention overcomes the problems of inaccurate extraction of ground object boundaries in semantic segmentation, discontinuity and non-closure of ground object edges extracted by edge detection, a large number of fine edges, and broken lines, and improves and ensures the accuracy and integrity of ground object edge extraction. The ground object boundaries extracted by the above operation are more in line with the visual specifications of the human eye and can partially replace manually drawn samples. The present invention can also select different deep learning network models according to different tasks, with wide applicability, strong robustness, and can adapt to a variety of task requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of the process of the present invention;

[0034] Figure 2a to Figure 2b is a sample graph of a semantic segmentation model in an embodiment of the present invention, wherein Figure 2a It is a sample image of the original high-resolution remote sensing image. Figure 2b It corresponds to Figure 2a Sample map of face labels;

[0035] Figure 3a to Figure 3b is a sample diagram of the edge detection model in an embodiment of the present invention, wherein Figure 3a It is a sample image of the original high-resolution remote sensing image. Figure 3b It corresponds to Figure 3a Sample graph of line labels;

[0036] Figure 4a to Figure 4b is an example diagram of the prediction results of the semantic segmentation model in an embodiment of the present invention, wherein Figure 4a It is the original image of high-resolution remote sensing image. Figure 4b It is the semantic segmentation result map predicted by the semantic segmentation model;

[0037] Figure 5a to Figure 5b is an example diagram of the prediction results of the edge detection model in an embodiment of the present invention, wherein Figure 5a It is the original image of high-resolution remote sensing image. Figure 5b It is the edge strength result map predicted by the edge detection model;

[0038] Figure 6a to Figure 6bis an example diagram of the final boundary result comparison in an embodiment of the present invention, wherein Figure 6a This is the result image after the edge intensity result image is vectorized and superimposed on the original image image. Figure 6b It is a result image obtained by vectorizing the edge result image processed by the present invention and superimposing the original image image. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0040] Figure 1 Schematic diagram of a high-resolution remote sensing target edge fine extraction method assisted by semantic segmentation;

[0041] Reference Figure 1 As shown in FIG. 1 , the high-resolution remote sensing target accurate detection method integrating semantic segmentation and edge of the present invention comprises the following steps:

[0042] (1) Select a suitable deep learning network model structure. In this example, the semantic segmentation model is Mask RCNN and the edge detection model is RCF.

[0043] (2) Drawing high-resolution remote sensing image samples includes the following three steps:

[0044] (2-1) Preparing high-resolution remote sensing images: Determine the production area to be studied and obtain high-resolution remote sensing images of the area. The present invention is aimed at visible light-near infrared multi-spectral satellite image data or true color aerial image data, and it is necessary to simultaneously determine the resolution of the image, the number of bands, and the type of ground object to be extracted;

[0045] (2-2) Cropping remote sensing images: Cropping small area samples in the high-resolution remote sensing image area. In this embodiment, samples of 1000*1000 size are cropped. The samples should contain typical ground objects (buildings in this example), and the sample coverage should be as wide as possible, covering the entire image area as much as possible. The number should be as large as possible, generally about 100 (the number can be adjusted according to the specific task). The cropped high-resolution remote sensing image original image sample map is as follows: Figure 2a , Figure 3a As shown;

[0046] (2-3) Draw sample labels: Use ArcGis software to manually mark the edges of the ground objects, strictly follow the real target boundaries, and assign semantic type attribute values ​​to the ground objects to obtain the corresponding vector files. Rasterize the vector files into surface labels and line labels respectively. The surface label samples are as follows: Figure 2b As shown, the line label samples are as follows Figure 3b If the sample data volume is small, data enhancement methods can be used to expand the sample set.

[0047] (3) Training the semantic segmentation model and edge detection model includes the following two steps:

[0048] (3-1) Initialize weights and set model hyperparameters: Use the pre-trained parameters of VGG-16 to initialize the RCF network weights, and use the pre-trained parameters on the COCO dataset to initialize the Mask RCNN parameters. Set model hyperparameters, learning rate, number of iterations, convolution step size, etc.

[0049] (3-2) Training the network model: Input the face label samples into the Mask RCNN semantic segmentation model for training until it is in a fitting state, thereby obtaining a semantic segmentation model for the task; input the line label samples into the RCF edge detection model for training until it is in a fitting state, thereby obtaining an edge detection model for the task;

[0050] (4) Obtaining semantic segmentation prediction maps and edge extraction prediction maps: Input the high-resolution remote sensing image of the production area into the trained semantic segmentation model to obtain the semantic segmentation prediction map, such as Figure 4b As shown; the high-resolution remote sensing image of the production area is input into the trained edge detection model to obtain the edge intensity map, such as Figure 5b As shown; the original high-resolution remote sensing image of the area to be produced is as follows Figure 4a , Figure 5a Shown

[0051] (5) Fusion of semantic segmentation prediction map and edge strength map, post-processing, including the following steps:

[0052] (5-1): Select an appropriate threshold to binarize the semantic segmentation prediction map. In this example, the threshold is 70 to retain the semantic information;

[0053] (5-2): Perform median filtering on the edge intensity map, select a suitable threshold for binarization, and use the skeleton extraction method to refine it to a single pixel width. This example is an automatic threshold processing. The median filter formula is as follows, where f(x, y) and g(x, y) are the original image and the processed image, respectively. W is a two-dimensional template, usually a 3*3 area, and med represents the median filter operation;

[0054] g(x,y)=med{f(xk,yl),(k,l∈W)}

[0055] (5-3): Perform connected domain analysis on the single-pixel width edge map obtained in (5-2) to remove some redundant and short broken lines;

[0056] (5-4): The processed edge map of (5-3) is superimposed on the binary semantic map obtained by (5-1), and the boundary of the semantic map is used to supplement the edge map. At this time, most of the broken lines form closed surfaces. The watershed algorithm is used to extract the edges of the closed surfaces, and the line segments that have not yet been closed are retained.

[0057] (5-5): (5-4) There are still some unclosed line segments in the processed edge graph. The eight-neighborhood algorithm is used to find the breakpoints, and the shortest path connection algorithm is used to form closed edges.

[0058] (5-6): The edge map obtained by (5-5) will still have some broken lines or redundant lines, which can be identified and deleted using the eight-neighborhood algorithm.

[0059] (5-7): Fill the holes in the edge map obtained in (5-6) and overlay the single-pixel edge map in (5-2) to retain important edge information and use the watershed algorithm to obtain the edge. The hole filling formula is as follows, A is the original image, A c is the complement of image A, B is the structural element, usually 3*3 in size, X k represents the kth element that needs to be filled, and ⊕ represents a B to X k-1 The expansion operation of , which represents the filling process of an element.

[0060]

[0061] (6) Vectorization processing: The edge map obtained in the above steps is basically sufficient for accurate detection of ground objects. Finally, vectorization is performed to generate a vector line layer. The vector layer is superimposed on the original image to obtain a complete and accurate extraction of the ground object boundary. Figure 6b As shown, Figure 6a The vector layer is obtained after processing the edge strength map predicted by RCF. By comparing the two, it can be found that the present invention overcomes the problems of inaccurate boundary extraction of ground object targets in semantic segmentation and discontinuity and disconnection of ground object target edges extracted by edge detection, and improves and ensures the accuracy and integrity of ground object target edge extraction.

[0062] The present invention integrates semantic segmentation and edge detection to achieve accurate detection and extraction of ground objects in high-resolution remote sensing images. Specifically, the edge detection model is used to predict the edge strength of ground objects, the semantic segmentation model is used to predict the semantic category and position of the target ground object, and the ground object information predicted by the semantic segmentation is integrated to process the edge strength map predicted by the edge detection, thereby achieving accurate edge extraction of remote sensing ground objects. The present invention utilizes the prediction result maps after edge detection and semantic segmentation to fuse and restrict each other, while overcoming the inaccuracy problem of the edge detection model in semantic category, the non-closure problem caused by disconnection during prediction, and the inaccuracy of the semantic segmentation model in predicting the edge of the ground object, thereby improving the accuracy of edge extraction of high-resolution remote sensing ground objects and achieving accurate detection of high-resolution remote sensing targets.

[0063] The example of the present invention is implemented on a PC platform equipped with a GPU. Experiments have shown that the present invention can obtain a relatively ideal segmentation effect, which is more consistent with the visual delineation effect and can improve the accuracy of target edge detection. The method mentioned in the present invention can be widely used in the extraction of target objects in high-resolution remote sensing images, greatly improving the extraction accuracy. The method mentioned in the present invention can be widely used in the image processing, analysis, classification, and recognition of high-resolution remote sensing images.

[0064] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms described in the embodiments. The protection scope of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A high-resolution remote sensing target accurate detection method integrating semantic segmentation and edge, comprising the following steps: Step 1: Select a semantic segmentation network model and an edge detection network model suitable for extracting ground objects from high-resolution remote sensing images; Step 2: Draw high-resolution remote sensing image samples, manually draw their precise boundaries and determine their semantic types according to the sample's ground objects, and generate corresponding line label samples and surface label samples; Step 3: Use the line label samples in step 2 to train the edge detection model, and use the surface label samples to train the semantic segmentation model until they are in a fitting state; Step 4: Use the edge detection model and semantic segmentation model trained in step 3 to predict the high-resolution remote sensing image of the production area, and obtain the edge intensity map and semantic segmentation prediction map of the ground object target in the high-resolution remote sensing image; Step 5: Fuse the semantic segmentation prediction map and edge strength map in step 4, process the wrong target boundaries and missed target boundaries in the edge strength map, extract the ground object target edges predicted by semantic segmentation, and repair the broken lines in the edge strength map to obtain a more accurate target edge map; Step 6: Use the target edge map obtained after step 5 to perform vector processing, construct the polygon element vector corresponding to the ground object target, and complete the accurate extraction of the ground object target boundary; In step 1, according to the characteristics of high-resolution remote sensing images, the semantic segmentation network model uses Mask RCNN or D-Linknet, and the edge detection network model uses RCF or BDCN; When selecting samples in step 2, multiple areas of the same size are cut from the large-area high-resolution remote sensing image, and the area should contain a variety of ground objects. After selecting the samples, the boundaries of the ground objects are outlined to generate binary line label samples and surface label samples. If the sample data volume is small, the data enhancement method can be used to expand the sample set. In step 3, the edge detection model is trained using line label samples, and the semantic segmentation model is trained using face label samples. Appropriate parameters and hyperparameters need to be selected during training, and the model is trained until it is in a fitting state, including: Step 3.1 Initialize weights and set model hyperparameters: Use the pre-trained parameters of VGG-16 to initialize the RCF network weights, and use the pre-trained parameters on the COCO dataset to initialize the Mask RCNN parameters; set model hyperparameters, learning rate, number of iterations, and convolution step size; Step 3.2: Train the network model: Input the face label samples into the Mask RCNN semantic segmentation model and train it until it is in a fitting state to obtain a semantic segmentation model for the task; input the line label samples into the RCF edge detection model and train it until it is in a fitting state to obtain an edge detection model for the task; Step 4: Input the high-resolution remote sensing images of the production area into the trained semantic segmentation model to obtain the semantic segmentation prediction map and edge strength map; In step 5, the prediction graph obtained in step 4 needs to be processed. The specific steps are as follows: Step 5.1: Select a suitable threshold to binarize the semantic segmentation prediction map; Step 5.2: Perform median filtering on the edge intensity map, select a suitable threshold for binarization, and use the skeleton extraction method to refine it to a single pixel width; The median filter formula is as follows, where f(x, y) and g(x, y) are the original image and the processed image respectively; W is a two-dimensional template, which is a 3*3 area, and med represents the median filter operation; g(x,y)=med{f(xk,yl),(k,l∈W)} Step 5.3: Perform connected domain analysis on the single-pixel width edge map obtained in step 5.2 to remove some redundant and short broken lines; Step 5.4: The edge map processed in step 5.3 is superimposed on the binary semantic map obtained in step 5.1, and the boundary of the semantic map is used to supplement the edge map. At this time, most of the broken lines form a closed surface. The watershed algorithm is used to extract the edge of the closed surface, and the line segments that have not been closed are retained; Step 5.5: After step 5.4, there are still some broken lines in the edge graph. Use the eight-neighborhood algorithm to find the breakpoints and use the shortest path connection algorithm to form a closed edge. Step 5.6: The edge map obtained in step 5.5 still has some broken lines or redundant lines, which can be identified and deleted using the eight-neighborhood algorithm; Step 5.7: Fill the holes in the edge map obtained in step 5.6 and overlay the single-pixel edge map in step 5.2 to retain important edge information and use the watershed algorithm to obtain the edge; the hole filling formula is as follows, A is the original image, A c is the complement of image A, B is the structural element, with a size of 3*3, and X k represents the kth element that needs to be filled, and ⊕ represents a B to X k-1 The expansion operation of , which represents the filling process of an element; X k =(X k-1 ⊕B)∩A c k=1,2,3… Step 6 is vectorization processing, which vectorizes the edge map obtained in step 6 to generate a vector line layer.

Images