A remote sensing image segmentation method and system based on edge auxiliary information

Abstract

The invention belongs to the technical field of remote sensing image segmentation, and provides a remote sensing image segmentation method and system based on edge auxiliary information. First, a remote sensing image is acquired, and a plurality of local images are obtained after image preprocessing; then a remote sensing image segmentation model is used to predict the local image to obtain a prediction result that each pixel belongs to a class; wherein the remote sensing image segmentation model comprises an encoder and a decoder; the encoder adopts a residual network to obtain a shallow feature map and a plurality of deep feature maps; the decoder is based on the shallow feature map and the plurality of deep feature maps, and obtains an edge feature map and a main feature map through upsampling, splicing and difference operation, and fuses the edge feature map with the main feature map to obtain a final feature map; the segmentation effect of the edge in the remote sensing image is improved, and the accuracy of remote sensing image segmentation is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing image segmentation, and in particular relates to a remote sensing image segmentation method and system based on edge auxiliary information. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, with the maturity of aerospace technology and the commercialization of drone technology, large-scale collection of high-resolution remote sensing images has been realized. High-resolution remote sensing images contain richer spatial geometric information, texture information, and color information. Intelligent interpretation of high-resolution remote sensing images is of great significance in many fields such as environmental monitoring, urban planning, and resource utilization. The most direct analysis of the information contained in them is to determine the object to which each pixel belongs, that is, to assign a category to each pixel, which coincides with the purpose of image semantic segmentation.

[0004] With the development of deep learning, especially with the introduction of the fully convolutional neural network FCN, the image semantic segmentation task has made milestone progress. Algorithms such as Unet, SegNet, PSPNet, and Deeplab series have gradually achieved very high accuracy in natural scene image segmentation tasks. When these algorithms are applied to high-resolution remote sensing images, they can guarantee a certain segmentation effect. However, due to the problem that remote sensing images have large intra-class differences and small inter-class differences, for example, different buildings belonging to the same building category will have differences in size, color, and texture, while roads and buildings have similarities. Therefore, image segmentation algorithms designed for natural scenes have category misclassification and inaccurate edges when extracting ground objects in remote sensing images, making them difficult to apply directly. Summary of the invention

[0005] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides a remote sensing image segmentation method and system based on edge auxiliary information, which extracts multi-scale information of the remote sensing image through a convolutional neural network, decouples the edge information on the basis of the multi-scale information, and uses the edge information and the main feature map to construct a feature map containing refined edge information, and uses this feature map for final segmentation prediction.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A first aspect of the present invention provides a remote sensing image segmentation method based on edge auxiliary information, comprising:

[0008] Acquire remote sensing images, and obtain multiple local images after image preprocessing;

[0009] The remote sensing image segmentation model is used to predict the local image and obtain the prediction results of each pixel belonging to each category;

[0010] Among them, the remote sensing image segmentation model includes an encoder and a decoder; the encoder adopts a residual network to obtain a shallow feature map and multiple deep feature maps; the decoder is based on the shallow feature map and the multiple deep feature maps, through upsampling, splicing and difference operations, to obtain an edge feature map and a main feature map, and the edge feature map and the main feature map are fused to obtain the final feature map.

[0011] Furthermore, the step of obtaining edge features and main features includes:

[0012] After multiple deep feature maps are upsampled to the size of shallow feature maps through linear interpolation, they are concatenated to obtain primary feature maps;

[0013] Concatenate the primary feature map and the shallow feature map to obtain a fused feature map;

[0014] Based on the fusion feature map and the shallow feature map, the edge feature map is decoupled in the form of difference;

[0015] Based on the edge feature map and the fusion feature map, the main feature map is decoupled in the form of difference.

[0016] Furthermore, it is characterized in that the loss function of the remote sensing image segmentation model is composed of a segmentation loss function and an auxiliary loss function;

[0017] The auxiliary loss function uses edge labels as supervisory information and uses binary cross entropy as the loss function;

[0018] The segmentation loss function uses the category label as supervision information and the cross entropy as the loss function.

[0019] Furthermore, the remote sensing image segmentation model also includes a predictor, which maps the final feature map to the same dimension as the number of remote sensing image categories.

[0020] Furthermore, the local image is obtained by cutting the remote sensing image using a sliding window method. Specifically, the sliding window is moved on the remote sensing image with a set window size and moving step to obtain multiple local images. For the part that is not large enough to be cut, the sliding window is rolled back to make it meet the window size.

[0021] Furthermore, the training set used in the training process of the remote sensing image segmentation model includes image blocks and their corresponding edge labels and category labels; wherein the image blocks in the training set need to be subjected to data enhancement operations.

[0022] Furthermore, the remote sensing image segmentation model outputs the prediction results of each pixel belonging to each category in the form of probability. For overlapping pixels between multiple local images, the probability of each overlapping pixel belonging to each category is averaged, and the average is used as the final prediction result of each overlapping pixel.

[0023] A second aspect of the present invention provides a remote sensing image segmentation system based on edge auxiliary information, comprising:

[0024] A preprocessing module is configured to: acquire a remote sensing image, and obtain a plurality of local images after image preprocessing;

[0025] A prediction module is configured to: use a remote sensing image segmentation model to predict the local image and obtain a prediction result of each pixel belonging to each class;

[0026] Among them, the remote sensing image segmentation model includes an encoder and a decoder; the encoder adopts a residual network to obtain a shallow feature map and multiple deep feature maps; the decoder is based on the shallow feature map and the multiple deep feature maps, through upsampling, splicing and difference operations, to obtain an edge feature map and a main feature map, and the edge feature map and the main feature map are fused to obtain the final feature map.

[0027] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the remote sensing image segmentation method based on edge auxiliary information as described above.

[0028] The fourth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the remote sensing image segmentation method based on edge auxiliary information as described above are implemented.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] The present invention provides a remote sensing image segmentation method based on edge auxiliary information, which overcomes the shortcomings of existing segmentation algorithms in the field of remote sensing images and improves the accuracy of remote sensing image segmentation.

[0031] The present invention provides a remote sensing image segmentation method based on edge auxiliary information, which improves the network's ability to process objects of different scales by introducing multi-scale features. At the same time, edge auxiliary supervision information is added during the training process, which effectively improves the problem of inaccurate segmentation at the edge of the existing segmentation algorithm and improves the segmentation effect of the edge in the remote sensing image. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0033] Figure 1 This is an overall flow chart of a remote sensing image segmentation method based on edge auxiliary information according to an embodiment of the present invention;

[0034] Figure 2 4 is a structural diagram of a feature extraction network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0036] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0037] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0038] Embodiment 1

[0039] like Figure 1-2 As shown, this embodiment provides a remote sensing image segmentation method based on edge auxiliary information, which can be applicable to the segmentation task of high-resolution remote sensing images. The method extracts multi-scale information of the remote sensing image through a convolutional neural network, decouples the edge information based on the multi-scale information, and uses the edge information and the main feature map to construct a feature map containing refined edge information, and uses this feature map for the final segmentation prediction.

[0040] The present embodiment provides a remote sensing image segmentation method based on edge auxiliary information. First, a remote sensing image is acquired, and multiple local images are obtained after image preprocessing; then, a remote sensing image segmentation model is used to predict the local image to obtain the prediction results of each pixel belonging to each category; wherein, the remote sensing image segmentation model includes an encoder and a decoder; the encoder uses a residual network to obtain a shallow feature map and multiple deep feature maps; the decoder is based on the shallow feature map and multiple deep feature maps, and obtains an edge feature map and a main feature map through upsampling, splicing and difference operations, and fuses the edge feature map with the main feature map to obtain the final feature map. Specifically, it includes 6 steps:

[0041] 1. Obtain remote sensing images

[0042] As an implementation method, the original data set is divided into a training set and a test set in a ratio of 7:3, wherein the training set is used to train the remote sensing image segmentation model, and the validation set is used to evaluate the model effect.

[0043] In actual use, the acquired remote sensing images or remote sensing images in the test set can be directly input into the trained remote sensing image segmentation model after preprocessing.

[0044] Among them, the remote sensing image segmentation model includes feature extraction network and predictor, such as Figure 2 As shown, the feature extraction network includes an encoder and a decoder.

[0045] 2. Image preprocessing

[0046] (1) The remote sensing image is cut using a sliding window method to obtain multiple local images. Specifically, the window size and moving step of the sliding window are set, and the moving step is smaller than the window size. The sliding window is moved on the remote sensing image with the set window size and moving step to obtain multiple local images. For the part that is not large enough to be cut, the sliding window is rolled back to make it meet the window size.

[0047] As an implementation method, a high-resolution remote sensing image is cut into multiple local images using a sliding window method, wherein the sliding window size is 896*896 pixels, and it moves 512 pixels each time. For the part that is not large enough to be cut, the window is rolled back to make it 896*896 pixels to ensure that every part of the original image is utilized.

[0048] (2) The image blocks in the training set need to undergo data enhancement operations, such as horizontal and vertical random flipping and random 90-degree rotation of the local image.

[0049] For the training set, before image preprocessing, the seg2bmap method is used to generate edge labels based on category labels. The labels take values ​​from 0 to 1. If a pixel is at the edge of an object, its label value is 1, otherwise, its label value is 0. Therefore, the training set used by the remote sensing image segmentation model during training includes image blocks and their corresponding edge labels and category labels.

[0050] Compared with natural scene images, high-resolution remote sensing images are generally very large in size and cannot be directly fed into the remote sensing image segmentation model for training. Therefore, high-resolution remote sensing images are segmented in advance to meet the limitations of computing resources.

[0051] The present invention uses a pre-training model in the training process, so the present invention also standardizes the remote sensing image according to the statistical data of the pre-training model.

[0052] 3. Encoding process

[0053] First, the present invention selects the residual network ResNet-50 as the encoder. ResNet-50 constructs a residual structure between input and output, making the training of deep networks possible.

[0054] The encoder uses a residual network to obtain multiple feature maps of different scales, including shallow feature maps and multiple deep feature maps.

[0055] As an implementation method, the encoder ResNet-50 is divided into four stages to generate feature information of four scales respectively. Taking a local image of 896*896 pixels as an example, the image is input into the feature extractor, and four feature maps of different scales are obtained, with sizes of 224*224*256, 128*128*512, 64*64*1024 and 32*32*2048 respectively. The last number represents the number of channels of the current feature map. The four feature maps are recorded as P2, P3, P4, and P5 respectively.

[0056] 4. Decoding process - edge information decoupling

[0057] The feature map extracted by the shallow layer of the network ResNet-50 has a larger resolution and contains more spatial information. As the number of layers of the network ResNet-50 increases, the feature map resolution gradually decreases and contains more semantic information.

[0058] The decoder obtains edge features and main features based on shallow feature maps and multiple deep feature maps through upsampling, splicing and difference operations. The specific steps include:

[0059] (1) Multiple deep feature maps P3, P4, and P5 are upsampled to the size of the shallow feature map P2 through linear interpolation to form feature The above three features are concatenated to obtain a primary feature map containing rich semantic information.

[0060] (2) By concatenating primary feature maps With the shallow feature map P2 containing more spatial information, we get the fused feature map Therefore, P contains both edge information and main information, and concat represents the concatenation operation.

[0061] (3) Based on the fusion feature map and the shallow feature map, the edge feature map is decoupled in the form of difference

[0062] In the training process of remote sensing image segmentation model, P edge To predict the edge of the object in the image, and use the edge label obtained in the image preprocessing as the supervision information, the binary cross entropy is used as the loss function, which is recorded as the auxiliary loss function L edge , the calculation formula is as follows:

[0063]

[0064] Where N is the number of samples, that is, the number of pixels in each local image; i represents the i-th pixel in the image, and its value range is {1,2,3,...,N}; y i is the category label corresponding to the i-th pixel. When the pixel is located at the edge of the object, y i =1, otherwise y i =0; p(y i ) is the probability that the i-th pixel belongs to the boundary.

[0065] (4) At this point, a feature map P representing edge information is constructed. edge And the fusion feature map P containing both the main information and the edge information is obtained by difference in the same way as decoupling the edge information. body =PP edge .

[0066] 5. Decoding process - fusion of edge information and main information

[0067] By decoupling the fusion feature P, the present invention introduces auxiliary supervision information into the remote sensing image segmentation model. The remote sensing image segmentation model will pay more attention to the edge part of the object, making the remote sensing image segmentation model more accurate in predicting the edge.

[0068] The present invention combines the supervised edge feature map with the main feature map to obtain the final feature map P pred =P edge +P body .

[0069] 6. Final prediction

[0070] The predictor contains a convolution layer with a convolution kernel size of 1*1. The predictor converts the final feature map P pred Map it to the same dimension as the number of remote sensing image categories, and use it with the image label to calculate the cross entropy loss function Loss.

[0071] The segmentation loss function uses the category label as supervision information and cross entropy as the loss function. The formula for calculating the segmentation loss function is as follows:

[0072]

[0073] Where N is the number of samples, which is the number of pixels in each image block; M is the number of categories; i represents the i-th pixel in the image block, and its value range is {1,2,3,...,N}; j represents the j-th category, and its value range is {1,2,...,M}; y ij is a symbolic function, i.e., a category label. If the true category of sample i is equal to j, it takes 1, otherwise it takes 0; p ij is the probability that sample i belongs to category j.

[0074] In addition to supervising the segmentation results, the present invention also sets auxiliary supervision information by introducing an auxiliary loss function. That is, the loss function of the remote sensing image segmentation model is composed of the segmentation loss function and the auxiliary loss function. Therefore, the calculation method of the total loss function Loss is as follows:

[0075] Loss=Loss seg +λLoss edge

[0076] Among them, λ is a hyperparameter of the auxiliary loss function. Preferably, the value of λ is set to 1.

[0077] The present invention uses the SGD optimizer to optimize the remote sensing image segmentation model, sets its momentum to 0.9, the weight decay coefficient to 0.0001, and takes 0.007 as the initial learning rate. During the training process, the learning rate decays in a polynomial manner until it reaches 0. Limited by computing resources, the present invention sets the batch training size to 4 and sets 80 epochs. The present invention does not perform verification during the training process, but observes the change curve of the loss function Loss, and selects the model after the loss function Loss converges as the final remote sensing image segmentation model.

[0078] The remote sensing image segmentation model is used to predict the local image and obtain the prediction results of each pixel belonging to each category.

[0079] As an implementation method, in the test phase, the present invention keeps the image cropping consistent with the training process, that is, the test image is cropped into multiple image blocks by a sliding window method, wherein the sliding window size is 896*896 pixels, and each slide is 512 pixels. Given any test image I, during the test process, the position information of each local image relative to the image I is recorded, and then each local image is sent to the remote sensing image segmentation model. The remote sensing image segmentation model will output the prediction result of each pixel belonging to each category in the form of probability. For the overlapping pixels between the two local images, the present invention averages the probability of each overlapping pixel belonging to each category based on the position information of the local image relative to the image I, and uses the average as the final prediction result of each overlapping pixel. If the pixel is covered by multiple image blocks, the final prediction result is calculated according to the principle of averaging. Further, all the image blocks cropped from the image I are combined according to the above principles to form the final segmentation result of the image I.

[0080] The remote sensing image segmentation method based on edge auxiliary information of the present invention improves the ability of the network to process objects of different scales by introducing multi-scale features, and at the same time adds edge auxiliary supervision information in the training process, effectively improving the problem of inaccurate segmentation at the edge of the existing segmentation algorithm, improving the segmentation effect of the edge in the remote sensing image, overcoming the shortcomings of the existing segmentation algorithm in the field of remote sensing images, and improving the accuracy of remote sensing image segmentation.

[0081] Embodiment 2

[0082] This embodiment provides a remote sensing image segmentation system based on edge auxiliary information, which specifically includes the following modules:

[0083] A preprocessing module is configured to: acquire a remote sensing image, and obtain a plurality of local images after image preprocessing;

[0084] A prediction module is configured to: use a remote sensing image segmentation model to predict the local image and obtain a prediction result of each pixel belonging to each class;

[0085] Among them, the remote sensing image segmentation model includes an encoder and a decoder; the encoder adopts a residual network to obtain a shallow feature map and multiple deep feature maps; the decoder is based on the shallow feature map and the multiple deep feature maps, through upsampling, splicing and difference operations, to obtain an edge feature map and a main feature map, and the edge feature map and the main feature map are fused to obtain the final feature map.

[0086] It should be noted here that each module in this embodiment corresponds to each step in Example 1 one by one, and the specific implementation process is the same, which will not be repeated here.

[0087] Embodiment 3

[0088] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the remote sensing image segmentation method based on edge auxiliary information as described in the first embodiment above are implemented.

[0089] Embodiment 4

[0090] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the remote sensing image segmentation method based on edge auxiliary information as described in the first embodiment above are implemented.

[0091] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0092] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0093] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0094] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0095] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0096] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A remote sensing image segmentation method based on edge auxiliary information, characterized in that: include: Acquire remote sensing images, and obtain multiple local images after image preprocessing; The remote sensing image segmentation model is used to predict the local image and obtain the prediction results of each pixel belonging to each category; The remote sensing image segmentation model includes an encoder and a decoder; the encoder uses a residual network to obtain a shallow feature map and multiple deep feature maps; the decoder obtains an edge feature map and a main feature map based on the shallow feature map and the multiple deep feature maps through upsampling, splicing and difference operations, and fuses the edge feature map with the main feature map to obtain a final feature map; The step of obtaining edge features and main features comprises: After multiple deep feature maps are upsampled to the size of shallow feature maps through linear interpolation, they are concatenated to obtain primary feature maps; Concatenate the primary feature map and the shallow feature map to obtain a fused feature map; Based on the fusion feature map and the shallow feature map, the edge feature map is decoupled in the form of difference; Based on the edge feature map and the fusion feature map, the main feature map is decoupled in the form of difference.

2. A remote sensing image segmentation method based on edge auxiliary information as claimed in claim 1, characterized in that: The loss function of the remote sensing image segmentation model is composed of a segmentation loss function and an auxiliary loss function; The auxiliary loss function uses edge labels as supervisory information and uses binary cross entropy as the loss function; The segmentation loss function uses the category label as supervision information and the cross entropy as the loss function.

3. A remote sensing image segmentation method based on edge auxiliary information as claimed in claim 1, characterized in that: The remote sensing image segmentation model also includes a predictor that maps the final feature map into the same dimension as the number of remote sensing image categories.

4. The remote sensing image segmentation method based on edge auxiliary information as claimed in claim 1, characterized in that: The local image is obtained by cutting the remote sensing image using a sliding window method. Specifically, the sliding window is moved on the remote sensing image with a set window size and moving step to obtain multiple local images. For the part that is not large enough to be cut, the sliding window is rolled back to make it meet the window size.

5. The remote sensing image segmentation method based on edge auxiliary information as claimed in claim 1, characterized in that: The training set used by the remote sensing image segmentation model during the training process includes image blocks and their corresponding edge labels and category labels; wherein the image blocks in the training set need to be subjected to data enhancement operations.

6. The remote sensing image segmentation method based on edge auxiliary information as claimed in claim 1, characterized in that: The remote sensing image segmentation model outputs the prediction results of each pixel belonging to each class in the form of probability. For overlapping pixels between multiple local images, the probability of each overlapping pixel belonging to each class is averaged, and the average is used as the final prediction result of each overlapping pixel.

7. A remote sensing image segmentation system based on edge auxiliary information, characterized in that: Implementing a remote sensing image segmentation method based on edge auxiliary information as described in any one of claims 1 to 6, comprising: A preprocessing module is configured to: acquire a remote sensing image, and obtain a plurality of local images after image preprocessing; A prediction module is configured to: use a remote sensing image segmentation model to predict the local image and obtain a prediction result of each pixel belonging to each class; Among them, the remote sensing image segmentation model includes an encoder and a decoder; the encoder adopts a residual network to obtain a shallow feature map and multiple deep feature maps; the decoder is based on the shallow feature map and the multiple deep feature maps, through upsampling, splicing and difference operations, to obtain an edge feature map and a main feature map, and the edge feature map and the main feature map are fused to obtain the final feature map.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in a remote sensing image segmentation method based on edge auxiliary information as described in any one of claims 1 to 6 are implemented.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the remote sensing image segmentation method based on edge auxiliary information as described in any one of claims 1 to 6 are implemented.

Images