Remote sensing image target segmentation method based on instance segmentation network and voting mechanism

Through the instance segmentation network and voting mechanism, combined with the sliding window overlapping cropping and mean method, the accuracy problem of dense distribution of cages in remote sensing image segmentation is solved, and efficient cage image segmentation is achieved, which reduces manual intervention and improves segmentation accuracy .

CN114627135BActive Publication Date: 2024-11-19HAINAN UNIV
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202210018598.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-08
Publication Date
2024-11-19
Estimated Expiration
2042-01-08

AI Technical Summary

Technical Problem

The existing remote sensing image segmentation method is not effective when processing dense distribution of cages, and the traditional method requires manual parameters adjustment, which is time-consuming and labor-intensive, making it difficult to achieve high-precision cage image segmentation.

Method used

Using the method based on instance segmentation network and voting mechanism, through sliding window overlapping cropping method and mean method, combining data augmentation and three-channel separation, nine training sets were generated and nine models were generated, and the image blocks were segmented, and finally two rounds were passed. The vote was given the final judgment.

Benefits of technology

The precise segmentation of images of remote sensing cages of different densities is achieved, which reduces manpower and material consumption and improves segmentation accuracy, especially the segmentation effect under dense distribution of cages.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114627135B_ABST
    Figure CN114627135B_ABST
Patent Text Reader

Abstract

The present invention provides a remote sensing image target segmentation method based on an instance segmentation network and a voting mechanism: step 1, construct a training data set, perform data enhancement and three-channel separation operations on the training samples, and generate nine training sets; step 2, introduce instance segmentation network training and generate nine models; step 3, reasonably fill the samples to be detected, and use the sliding window overlapping cropping method to cut the high-resolution samples to be detected into image blocks in sequence; step 4, use nine trained models to perform segmentation, and splice the segmented image blocks using the mean method to obtain nine segmented images of the same original test sample; step 5, for each test sample image, vote twice for each pixel through the nine segmented images to obtain the final judgment result. The method of the present invention can have a good target segmentation effect on remote sensing images with different distribution densities, and solves the problems of manual segmentation, high cost and high error in surveying and mapping in traditional methods.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the fields of deep learning and image processing, and in particular to a remote sensing image target segmentation method based on an instance segmentation network and a voting mechanism. Background Art

[0002] Too high cage density and unreasonable cage layout can easily lead to an imbalance between algae and bacteria in the aquaculture environment, causing serious damage to the ecological environment system and being detrimental to the sustainable development of marine aquaculture. At present, most of the research in the field of marine aquaculture cage density is based on the statistics of fish density in cages. However, not only the aquaculture density, but also the distribution density and layout of the cages can affect the normal growth of fish.

[0003] With the continuous efforts of scientific researchers, domestic and foreign scholars have made great development and progress in the field of target segmentation. Early segmentation methods mainly used threshold processing, edge detection, region growing, etc. to segment the target area. However, the above traditional methods have low segmentation accuracy, and the relevant threshold parameters need to be manually adjusted due to different environments, which is time-consuming and laborious. With the emergence of deep learning networks, its powerful feature learning ability has been verified in many fields. It can learn objects in different scenes and has good feature extraction capabilities for learned objects. Fully Convolutional Networks (FCN) is one of the earliest applications of deep learning methods in the field of segmentation. Compared with non-deep learning methods, its powerful segmentation ability has attracted much attention. Image segmentation tasks have entered a rapid development stage. Subsequently, U-Net, Mask RCNN, Deep Lab and other increasingly complex deep learning segmentation algorithms have been proposed and widely used in various fields.

[0004] There are many research branches in the image segmentation task. Among them, the remote sensing image segmentation task has been widely studied because it can segment large areas of targets from a macro perspective, greatly reducing labor costs. Although non-deep learning methods have good segmentation effects on targets in specific scenes, the final segmentation effect is generally sensitive to the target scene, the hyperparameters of the method, and the imaging conditions. They generally have weak generalization capabilities and there is still a lot of room for improvement in segmentation accuracy. Therefore, deep learning methods have been applied to remote sensing image segmentation tasks. Huang Y. et al. (Huang Y., Tang LB, Li Z., Long T. (2019). Research on Peanut Planting Area Classification Techonology Using Remote Sensing Image Based Deep Learning. Journal of Signal Processing, 35 (4), 617-622.) used a deep learning network to determine the statistical data of peanut planting areas in remote sensing images. This reduces the problems of poor timeliness and human resource consumption of traditional statistical methods. Zi WJ et al. (Zi WJ, Xiong W., Chen H., Li J., Jing N. (2021), SGA-Net: Self-Constructing Graph Attention Neural Network for Semantic Segmentation of Remote Sensing Images. REMOTE SENSING, 13 (21). DOI: 10.3390 / rs13214201.) proposed a novel self-constructing graph attention neural network (SGA- Net). It uses a graph neural network to capture global context representation and uses an attention mechanism to focus on extracting useful feature information. Liu R. et al. (Liu R., Mi L., Chen ZZ (2021). AFNet: Adaptive Fusion Network for Remote Sensing Image Semantic Segmentation. IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, 59 (9), 7871-7886.) proposed a novel adaptive fusion network (AFNet) for improving the performance of very high resolution (VHR) remote sensing image segmentation. It designs a multi-level architecture with a Scale Feature Attention Module (SFAM) to learn more representative features.In addition, deep learning methods for remote sensing image semantic segmentation tasks include DDLNet, ResU-Net, etc. However, the above deep learning methods have not yet been studied for remote sensing cage images, and direct application may not be applicable. Yu Chuang et al. (Yu C., Liu YP, Hu ZH, Xia X. (2021). Accuratesegmentation of remote sensing cages based on U-Net and voting mechanism. Proc. Seventh Asia Pacific Conference on Optics Manufacture (COM21), Accepted for Publication, Not Yet Published.) proposed a remote sensing cage density statistics method based on Mask R-CNN. However, its application has certain limitations. This method does not work well when the cages are densely distributed. Summary of the invention

[0005] The purpose of the present invention is to provide a remote sensing image target segmentation method based on an instance segmentation network and a voting mechanism, which realizes accurate segmentation of remote sensing cage images through a sliding window overlapping cropping method, a mean method and a two-round voting mechanism, thereby solving the problem that manual measurement in traditional methods consumes a lot of manpower and material resources.

[0006] The technical solution of the present invention is to provide a remote sensing image target segmentation method based on an instance segmentation network and a voting mechanism, comprising the following steps:

[0007] Step 1: construct a remote sensing image target training data set (such as remote sensing cage images), perform data enhancement and three-channel separation operations on the training samples, and generate nine training sets consisting of three image resolutions and three single channels;

[0008] Step 2: Introduce an instance segmentation network (such as SegNet) to train nine training sets and generate nine models;

[0009] Step 3: Fill the sample images to be detected reasonably, and use the sliding window overlapping cropping method with a sliding step size of half the window size to cut the high-resolution sample images to be detected into image blocks in sequence;

[0010] Step 4: Use nine trained models to segment the image blocks, and splice the segmented image blocks using the mean method to obtain nine segmented images of the same original test sample;

[0011] Step 5: For each test sample image, vote twice for each pixel through nine segmented images to obtain the final judgment result.

[0012] The beneficial effects of the present invention are:

[0013] (1) The present invention is an innovative application of the combination of traditional methods and methods based on deep learning and voting mechanisms. After preprocessing the image, the deep learning instance segmentation network is used for training, and finally a judgment vote is performed to obtain a segmented image. The method proposed in the present invention can achieve good segmentation effects on remote sensing cage images of different densities. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a summary figure of a remote sensing image target segmentation method based on an instance segmentation network and a voting mechanism of the present invention;

[0015] Figure 2 is a diagram of a remote sensing cage image segmentation scheme of SegNet and a voting mechanism in an embodiment of the present invention;

[0016] Figure 3 are training samples from the RSCS dataset and their corresponding labels in an embodiment of the present invention;

[0017] Figure 4 is an image enhanced by data in an embodiment of the present invention;

[0018] Figure 5 It is the filling and cropping of images when they are non-overlapping in an embodiment of the present invention;

[0019] Figure 6 Filling and cropping images when overlapping in an embodiment of the present invention;

[0020] Figure 7 is a schematic diagram of a voting mechanism in an embodiment of the present invention;

[0021] Figure 8 It is the cage segmentation result of Hainan Lingshui Li'an Port in the embodiment of the present invention;

[0022] Fig. 9 It is the cage segmentation result of Hainan Lingshui Xincun Port in the embodiment of the present invention;

[0023] Fig.10 is the cage segmentation result of the Potou Port in Wanning, Hainan in the embodiment of the present invention; DETAILED DESCRIPTION

[0024] The following will be combined with the embodiment and Figure 1-10The concept, specific steps and technical effects of the present invention are clearly and completely described to fully understand the purpose, features and effects of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. The equivalent transformations or substitutions in functions, methods or structures made by ordinary technicians in the field according to these implementations are all within the protection scope of the present invention.

[0025] The embodiment of the present invention uses the SegNet+ voting mechanism to segment cage instances in remote sensing cage images. The remote sensing images in the RSCS dataset come from "Google Earth", mainly by searching for sea areas with marine aquaculture cages in Hainan, Guangdong, Guangxi, Shandong and other Chinese cities. Due to the special conditions required for marine aquaculture, the number of collected images is relatively scarce.

[0026] like Figure 2 As shown, this example provides a remote sensing image target segmentation method based on an instance segmentation network and a voting mechanism. In the figure, "256(R, G, B)" represents three single images (R, G, B) formed by splicing image blocks of 256×256 pixels; "512(R, G, B)" represents three single images (R, G, B) formed by splicing image blocks of 512×512 pixels; "1024(R, G, B)" represents three single images (R, G, B) formed by splicing image blocks of 512×512 pixels.

[0027] The method comprises the following steps:

[0028] 1. Construct a remote sensing image target training dataset (such as remote sensing cage images), perform data augmentation and three-channel separation operations on the training samples, and generate nine training sets consisting of three image resolutions and three single channels.

[0029] In order to annotate the dataset, the “labelme” (Torralba et al., 2010) annotation software was used to obtain the mask image, which was then converted into a binary image in uint8 format (Alberto et al., 2017) as a label. The labeled dataset was used as a training set.

[0030] For the selection of loss function, the embodiment of the present invention uses a binary cross entropy loss function, which is expressed as follows:

[0031]

[0032] where y i Represents the label i of the sample, the positive class is 1 and the negative class is 0. iRepresents the probability that sample i is predicted as a positive class. L represents the generated loss value, which is continuously updated during network training and gradually approaches es 0.

[0033] For the evaluation of the segmentation effect, the embodiment of the present invention uses IoU. Its expression is as follows:

[0034]

[0035] Where A∩B represents the sum of regions correctly predicted as objects. A∪B represents the sum of regions predicted as objects or marked as objects.

[0036] Figure 3 The training sample images and their corresponding label images are shown. The resolution is 5956×5002 pixels. The cages in the image are densely distributed and there are some abandoned cages.

[0037] On the one hand, due to the high resolution of remote sensing images, they cannot be directly input into the SegNet neural network for training. On the other hand, the number of remote sensing cage images is small. Therefore, data augmentation operations are used to crop images and generate more samples. Take the construction of a 512×512 pixel training set as an example. First, the original high-resolution image and the label image are randomly cropped out of 512×512 pixel size at the same time. Secondly, the cropped images undergo other data augmentation operations (Gao et al., 2020; Song et al., 2021), such as random rotation transformation, contrast transformation, blur transformation, flip transformation, and adding Gaussian noise. Finally, the enhanced data is extended to the training set. Rotation transformation and flip transformation simulate the positional differences of the cages. Contrast transformation, blur transformation, and Gaussian noise better simulate various environments. Finally, through the loop operation, the required number of different training samples can be obtained. As Figure 4 As shown, four images generated by data augmentation are randomly selected from the training set.

[0038] Finally, a total of 53 high-resolution remote sensing cage images of marine aquaculture were collected, of which 50 were used as training samples and 3 images of different scenes were used as test samples. Nine training sets were produced by data enhancement and RGB channel separation. For the 256×256 pixel training set, 160,000 images were generated. For the 512×512 pixel training set, 40,000 images were generated. For the 1024×1024 pixel training set, 10,000 images were generated. The test set contains three images, which were taken from Li'an Port, Xincun Port, and Potou Port. The cages in Li'an Port are sparsely distributed, and the image resolution is 10000×8224 pixels. The cages in Xincunkou are densely distributed, and the image resolution is 13000×8000 pixels. The cages in Potou Port are distributed moderately but in complex styles, and the image resolution is 15344×8224 pixels. Through the test samples of three different scene types, the actual segmentation effect and application scenarios of the generated model can be better verified.

[0039] Step 2: Introduce an instance segmentation network (such as SegNet) to train nine training sets and generate nine models;

[0040] The SegNet network is one of the classic networks for segmentation tasks. The network adopts an encoder-decoder structure (Hinton et al., 2006) and channel dimension splicing to achieve multi-scale feature fusion (Dong et al., 2020). Figure 3 As shown in Figure 1, the network uses upsampling in the decoder to enlarge the feature map and keep the integrity of high-frequency details during the segmentation process. It is widely used due to its simple structure, high accuracy, and fast segmentation speed.

[0041] The entire network can be divided into the encoding part on the left and the decoding part on the right. The encoding part contains five blocks, each of which consists of some convolution and pooling operations. The decoding part also contains five blocks, each of which consists of an upsampling and some convolutions. At the same time, for each feature map after upsampling, the encoding part and the corresponding feature map are channel-joined to improve the segmentation accuracy. Since the research task is binary segmentation, a 1×1 convolution layer is added to the last layer of the decoding part, and the sigmoid activation function is used.

[0042] Step 3: Fill the sample images to be detected reasonably, and use the sliding window overlapping cropping method with a sliding step size of half the window size to cut the high-resolution sample images to be detected into image blocks in sequence;

[0043] Since the input size of the training model includes three different sizes: 256×256 pixels, 512×512 pixels, and 1024×1024 pixels, the test set needs to be cropped into the corresponding sizes in turn and then input into the model for segmentation. Subsequently, the segmented image blocks are spliced. To ensure that every part of the remote sensing image can be detected and segmented, the areas on the right and bottom of the remote sensing image that are smaller than the required image block size will be filled with "0". The schematic diagram is as follows Figure 5 As shown, the solid line frame represents the filled remote sensing image, the dotted line frame represents the sliding window, and the arrows represent the windows sliding in sequence.

[0044] In order to further improve the segmentation accuracy, the embodiment of the present invention improves the sliding window non-overlapping cropping scheme and adopts a sliding window overlapping cropping method. When continuously cropping high-resolution images, the sliding step size is half the length of the image block. In order to ensure that each part of the remote sensing image can be segmented four times, the left and top of the remote sensing image are filled with "0", and the filling area is half the size of the image block. For the left and right sides of the remote sensing image, when the sliding window slides to the end of the area less than or equal to half the length of the image block, the remaining area image blocks within the length are filled with "0". Then, sliding window overlapping cropping is performed in sequence, and the cropped image is input into the trained model for segmentation.

[0045] like Figure 6 As shown in the figure, the sliding window cuts the filled image to half the length of the image block. The solid line box represents the filled remote sensing image, the dotted line box represents the sliding window, and the arrows represent the windows sliding in sequence. This operation can ensure that the original remote sensing image area can be segmented four times.

[0046] Step 4: Use nine trained models to segment the image blocks, and splice the segmented image blocks using the mean method to obtain nine segmented images of the same original test sample;

[0047] When the segmented image blocks are spliced ​​into the original image size, the mean method is used for information fusion. The value of each pixel after segmentation is a probability value between 0 and 1. If a point is a target, it will be closer to 1; otherwise it will be closer to 0. Since each pixel of the original remote sensing image has been calculated four times, the average value will be able to better represent the true category of the point.

[0048] Step 5: For each test sample image, vote twice for each pixel through nine segmented images to obtain the final judgment result.

[0049] For a single test sample, three single-channel test samples are cropped according to windows of different sizes. Then, the image patch is input into the trained model to obtain the segmented image patch. The segmented image patches are stitched together in sequence to obtain the segmented image of the original high-resolution image. Finally, the segmented high-resolution image is converted into a binary image with a threshold of 0.5. For each test sample, there will be nine different binary images to perform the voting mechanism. Figure 7 It can be seen that light-colored pixels represent target areas, dark-colored pixels represent misdetected areas, and the images of three different channels, which are spliced ​​from image blocks of the same size, will vote at each point in the same position. If they are all 1 or two values ​​are 1, the value of the final binary image of the corresponding point is 1; otherwise, it is 0. Then, the same voting mechanism is used to vote on the three generated binary images, and then an accurate binary image is generated.

[0050] Next, the method is verified in the embodiment of the present invention. The operating environment is Ubuntu 18.04 operating system, Keras and Python 3. The GPU is RTX 1080Ti 11G. In order to make full use of hardware resources, it is necessary to adjust the training parameters of training sets with different resolutions. For training samples with a resolution of 256×256 pixels, the batch size is 8. For training samples with a resolution of 512×512 pixels, the batch size is 2. For training samples with a resolution of 1024×1024 pixels, the batch size is 1. For other parameters, the learning rate is 5e-5 and the epochs are all 20.

[0051] The first is the verification of sliding window overlapping cropping. In order to better verify the sliding window overlapping cropping method, the embodiment of the present invention compares it with the sliding window non-overlapping cropping method. The experimental results are shown in Table 1. Among them, "NON-OLAP" means that the test sample adopts the sliding window non-overlapping cropping method; "OLAP" means that the test sample adopts the sliding window overlapping cropping method. "256", "512", and "1024" respectively indicate that the resolution of the cropped image block is 256×256 pixels, 512×512 pixels, and 1024×1024 pixels. In the embodiment of the present invention, in order to more accurately verify the effect, the image blocks are all RGB three-channel images.

[0052] Table 1: IoU comparison between sliding window overlapping and non-overlapping crops.

[0053]

[0054] As can be seen from Table 1, the segmentation effect of the sliding window overlapping cropping method is better than the sliding window non-overlapping cropping method. The IoU increases by up to 0.9%. The experimental results verify the effectiveness of the sliding window overlapping cropping method proposed in this study.

[0055] The second step is to verify the voting effect of different channels. In order to better verify the voting effect and ensure the reliability of the results, the present invention adopts sliding window overlapping cropping methods of different sizes for the test sets of four different channel types. The mIoU of each test set (including three samples) before and after voting is shown in Table 2.

[0056] Table 2. Performance comparison of the test set before and after voting for different channels.

[0057]

[0058] As can be seen from Table 2, the segmentation effect is significantly improved by using the voting mechanism. Unlike directly using RGB images, the voting mechanism of different channels can increase by up to 1.9%.

[0059] The next step is to verify the voting effects of different scales. In order to further verify the effects of voting of different scales, voting is conducted again after voting in different channels. The experimental results are shown in Table 3.

[0060] Table 3. Performance comparison of the test set before and after voting of different scales.

[0061]

[0062] From Table 3, we can see that although for one test sample, voting may be slightly worse than the segmentation result of some sizes, from the results of the overall test sample, the voting results of different sizes are better than the voting results of a single size, which can be improved by up to 0.6%. At the same time, by using the sliding window overlapping cropping method and the voting mechanism, the mIoU on the test set can reach 0.890.

[0063] Finally, a comparison of different methods is performed. In order to further verify the segmentation effect of the proposed scheme, the embodiment of the present invention conducts comparative experiments on different methods. As shown in Table 4, "Original FPN", "Original U-Net" and "Original SegNet" indicate that only the corresponding original methods are used for experiments. The input size of the image is 512×512 pixels, and the data format is RGB. "FPN scheme", "U-Net scheme" and "SegNet scheme" indicate that different methods are applied to the proposed scheme for experimental verification. The proposed schemes are superior to the original methods in terms of the final average segmentation accuracy. FPN, U-Net and SegNet have improvements of 0.016 (from 0.862 to 0.878), 0.008 (from 0.858 to 0.866) and 0.025 (from 0.865 to 0.890), respectively. In addition, the final segmentation scheme combined with SegNet achieves the relatively optimal segmentation accuracy.

[0064] Table 4. Comparison of segmentation accuracy between different algorithms.

[0065]

[0066]

[0067] In order to more intuitively demonstrate the segmentation effect of the three test images, the segmentation effects of the three test samples are listed, such as Figure 8-10 As shown in Figure 2. The image patches in each image corner are enlarged images of the corresponding areas.

[0068] from Figure 8-10 It can be seen that the proposed scheme has a good segmentation effect as a whole. Figure 8 The cages in the figure are sparsely distributed. From the enlarged image in the lower left corner, it can be seen that the proposed scheme can avoid the interference of abandoned cages. Fig. 9 The cages in the image are densely distributed. The proposed scheme can accurately detect the boundaries of dense cage areas and circular cages. Fig.10 The cage style in is very special. It can be seen that part of the cage is not segmented. Detailed information is shown in the light-colored area. On the one hand, the style of the cage is too close to that of an abandoned cage. On the other hand, due to the different cage construction methods in different regions, this type of cage does not appear in other collected training samples.

[0069] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic features of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present invention is limited by the attached claims rather than the above description, so it is intended to include all changes within the meaning and scope of the equivalent elements of the claims in the present invention. Although the principle of the present invention is described in detail above in conjunction with the preferred embodiments of the present invention, it should be understood by those skilled in the art that the above embodiments are only explanations of the schematic implementation of the present invention, and are not limitations on the scope of the present invention. The details in the embodiments do not constitute limitations on the scope of the present invention. Without departing from the spirit and scope of the present invention, any obvious changes such as equivalent transformations, simple substitutions, etc. based on the technical solution of the present invention fall within the scope of protection of the present invention.

Claims

1. A remote sensing image target segmentation method based on instance segmentation network and voting mechanism, characterized by: The following steps are involved: Step 1: construct a remote sensing image target training data set, which includes remote sensing cage images, perform data enhancement and three-channel separation operations on the training samples, and generate nine training sets consisting of three image resolutions and three single channels; Step 2: Introduce an instance segmentation network to train nine training sets and generate nine models, wherein the instance segmentation network includes SegNet; Step 3, reasonably fill the sample image to be detected, and use a sliding window overlapping cropping method with a sliding step size of half the window size to cut the high-resolution sample image to be detected into image blocks in sequence, characterized in that: reasonably filling the sample image to be detected includes: first, in the filling process, to ensure that each part of the sample image to be detected can be detected and segmented, for the area smaller than the required image block size on the right and below the sample image to be detected, "0" will be filled; secondly, in order to ensure that each part of the sample image to be detected can be segmented four times, the left and top of the sample image to be detected are filled with "0", and the filling area is half of the image block size; finally, for the left and right sides of the sample image to be detected, when the sliding window slides to the end of the area less than or equal to half the length of the image block, the remaining area image blocks within the length are filled with "0"; Step 4: Use nine trained models to segment the image blocks, and splice the segmented image blocks using the mean method to obtain nine segmented images of the same original test sample; Step 5: For each test sample image, vote twice for each pixel through nine segmented images to obtain the final judgment result.

Citation Information

Patent Citations

  • A remote sensing image ship integrated recognition method based on deep learning

    CN109583425A

  • Remote sensing image segmentation method based on disparity map and multi-scale depth network model

    CN110163213A

  • Remote sensing target segmentation and automatic splicing method based on voting mechanism

    CN113222999A

  • High-resolution remote sensing image vegetation extraction method based on sensitive feature focusing perception

    CN113609889A

  • Semantic segmentation method of remote sensing image and training method of semantic segmentation model

    CN113792742A