A target edge extraction method, image segmentation method and system

Abstract

The present invention belongs to the field of image processing technology, and specifically relates to a target edge extraction method, an image segmentation method and a system. The target edge extraction method simulates the working memory principle of vision by constructing an unsupervised rough segmentation method based on edge search, performs overall cognition of the scene, and obtains the accurate edges of large-area targets. The image segmentation method is based on a target extraction method and a segmentation neural network, simulating visual attention to observe the scene in detail to obtain small-area details; finally, based on the reasoning ability shown by vision, the accurate edges in the rough segmentation are combined to generate a segmentation result. On the basis of simulating visual reasoning to recognize the overall and details of the ground scene respectively, the present invention complements and corrects the overall information and the detailed information, implements noise removal and error repair, and thus effectively improves the accuracy of image segmentation.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a segmentation method based on visual reasoning combined with unsupervised clustering and deep learning. Background Art

[0002] Image segmentation is currently the cornerstone of many applications. It can classify images according to the spectrum, texture, shape and other characteristics of various targets to obtain category information in the scene. However, with the continuous improvement of current image resolution, it not only brings richer and more detailed scene information, but also makes the image information highly detailed. The spectra of the same type of targets may be inconsistent, and the spectra of different targets may overlap with each other, which increases the intra-class variance of the target and reduces the inter-class variance, bringing new challenges to current image segmentation.

[0003] The emergence of deep learning neural networks has provided an excellent new idea for image segmentation. Image segmentation technology based on deep learning, with the help of the powerful cognitive advantages of neural networks, has demonstrated cognitive capabilities that exceed previous methods and can conduct comprehensive and complete analysis and interpretation of various complex information in images. It is currently the most popular and widely acclaimed method.

[0004] However, due to the interference of different imaging lighting conditions and inconsistent target sizes, current segmentation methods find it difficult to reasonably balance cognitive granularity when analyzing scenes. They may not be able to protect the accurate edges of large-area targets, and may easily lose small-area details. Summary of the invention

[0005] In view of the defects or shortcomings of the prior art, the present invention provides a method for extracting the edge of an object.

[0006] To this end, the target edge extraction method provided by the present invention is to extract the edge of the target in the original image, and the method includes:

[0007] Step 1.1, build an image pyramid for the original image to obtain an image pyramid containing n layers of information; any layer is represented by Img i , i=1,2,...,n;n≥2;

[0008] Step 1.2, extract the edge of the image pyramid to obtain the edge pyramid, where any layer Img i The Edge of Ed i , i=1,2,...,n;

[0009] Step 1.3, resize each layer in the edge pyramid to the size of the original image, and any layer after resizing is Ued i , i=1,2,...,n;

[0010] Step 1.4, binarize each layer after Step 1.3, where any layer Ued i The gray level of the pixel is less than the threshold Ed_thd i The value of Ued is 0. i The gray level of the middle pixel is greater than or equal to the threshold Ed_thd i Assign a value of 1 to get any layer of Ued i Binarization layer Bed i , where Ed_thd i =mean(Ued i )-std(Ued i ), mean(Ued i ) is Ued i The mean value of all pixels in, std(Ued i ) is Ued i The standard deviation of all pixels in , i = 1, 2, ..., n;

[0011] Step 1.5, perform morphological dilation on all binary layers obtained in Step 1.4. i Perform i-1 morphological dilations to obtain the dilated edge Ded of the corresponding layer i , i=1,2,...,n;

[0012] Step 1.6, add up the dilated edges of all layers to get the cumulative result Sed;

[0013] Step 1.7, assign the grayscale of pixels in the area Ned_area1 in the accumulation result Sed to 0, and obtain the edge map Wed, wherein the area Ned_area1 is an area composed of pixels whose grayscale in Ded1 is 0;

[0014] Step 1.8, remove the pixels with grayscale values ​​less than n in the edge map to obtain a clear edge map Aed, and complete the extraction of the target edge.

[0015] Optionally, in Step 1.1, a bilinear interpolation resampling method is used to create an image pyramid for the original image.

[0016] Optionally, in Step 1.2, a Gaussian Laplacian filter is used to extract edges from the image pyramid.

[0017] Optionally, in Step 1.3, a nearest neighbor interpolation resampling method is used to adjust the size of each layer in the edge pyramid.

[0018] The present invention also provides an image segmentation method. To this end, the image segmentation method provided by the present invention includes:

[0019] Step 1, performing rough color clustering on the original image Omg to be segmented, including: executing the above method to obtain a clear edge map of the original image Omg; then performing color clustering on the original image Omg according to the edge information in the clear edge map and the different colors of the original image Omg to obtain a rough clustering result Rsg, wherein the rough clustering result Rsg contains the edge information of each target in the clear edge map;

[0020] Use the image segmentation neural network to segment the original image and obtain the segmentation result Nsg;

[0021] Step 2: Select any target Ngc from the segmentation result Nsg, find the corresponding target Rgc in the rough clustering result Rsg according to the position of any target Ngc, and then replace the edge of the target Ngc in the segmentation result Nsg with the clear edge of the target Rgc to complete the edge repair segmentation of the target Ngc.

[0022] Optionally, the image segmentation neural network is obtained after initial network training, and the initial network is a MASK-RCNN network, a shuffle neural network or an improved shuffle neural network. The improved shuffle neural network is obtained by adding residual connections before and after each shuffle structure in the shuffle neural network to obtain a residual shuffle structure, and then connecting a self-attention module to each residual shuffle structure.

[0023] The present invention also provides an image segmentation system, which includes an image edge extraction module, an image segmentation module and an edge replacement module;

[0024] The image edge extraction module is used to perform rough color clustering on the original image Omg to be segmented, including: using the above method to obtain a clear edge map of the original image Omg; then, according to the edge information in the clear edge map and the different colors of the original image Omg, color clustering is performed on the original image Omg to obtain a rough clustering result Rsg, and the rough clustering result Rsg contains the edge information of each target in the clear edge map;

[0025] The image segmentation module uses an image segmentation neural network to segment the original image to obtain a segmentation result Nsg;

[0026] The edge replacement module is used to select any target Ngc from the segmentation result Nsg, find the corresponding target Rgc in the rough clustering result Rsg according to the position of any target Ngc, and then replace the edge of the target Ngc in the segmentation result Nsg with the clear edge of the target Rgc to complete the edge repair segmentation of the target Ngc.

[0027] The target edge extraction method of the present invention is an edge extraction method based on visual working memory. The method is aimed at large-grained targets and can distinguish and protect the accurate edges of large-area targets based on analyzing the edges of the image itself.

[0028] The image segmentation method of the present invention is an image attention shuffle optimization segmentation method based on visual working memory, visual attention and visual reasoning. The method explores the visual reasoning ability by referring to the important visual working memory principle and visual attention mechanism in neural perception theory; on the basis of simulating visual reasoning to recognize the overall and details of the ground scene respectively, the overall information and the detail information are supplemented and corrected with each other, and noise removal and error repair are implemented, thereby effectively improving the accuracy of image segmentation.

[0029] In a further scheme, the attention shuffling neural network structure based on visual attention can find out the details that have an important impact on segmentation with the attention structure, and combine the shuffling structure to provide a more in-depth and comprehensive interpretation of small-grained details. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is the overall flow chart of the image segmentation method of the present invention.

[0031] Figure 2 This is a structural diagram of the improved shuffling network of the present invention.

[0032] Figure 3 1 is an example of the processing data and results of Example 1, where (a) is the original image and (b) is the target edge extraction result in the original image.

[0033] Figure 4 The original images and related segmentation results processed by Examples 2 and 3 of the present invention, (a) the original image, (b) the ground truth of the original image, (c) the segmentation result of the comparison example, (d) the segmentation result of Example 3, and (e) the segmentation result of Example 2.

[0034] Figure 5 Another original image and related segmentation results processed by Examples 2 and 3 of the present invention, (a) the original image, (b) the ground truth of the original image, (c) the comparative segmentation result, (d) the segmentation result of Example 3, and (e) the segmentation result of Example 2. DETAILED DESCRIPTION

[0035] Unless otherwise specified, the terms or methods in this article are understood by ordinary technicians in the relevant field or implemented using existing relevant methods.

[0036] In the target edge extraction method of the present invention, the unified edge pyramid is completely binarized to form a binary pyramid Bed i, i=1,2,...,n; n≥2; the specific value of n in the specific scheme is determined according to the pixel size of the original image; on this basis, for any layer of Bed in the binary pyramid i , with the number of layers i as a parameter, for the binary edge Bed i Perform i-1 morphological dilations to obtain the dilated edge pyramid Ded i ; Then, accumulation is performed layer by layer from the 1st layer to the nth layer to obtain the accumulated result Sed. Under the effect of accumulation, the edges existing in multiple resolutions are repeatedly superimposed, and larger values ​​and weights can be obtained, which are directly expressed in the accumulated result Sed; then, referring to the non-edge 0-value area Ned_area1 of the first layer Ded1 in the expansion pyramid, the non-edge position pixels corresponding to Ned_area1 in the accumulated result Sed are assigned to 0, thereby removing the biased edge positions, protecting the precise edges, and obtaining a precise and weighted edge map Wed; further, rough clustering of edge superpixels simulating visual working memory is performed. Since the higher the weight of the edge, the larger the area of ​​the target is surrounded, the small edges with weights less than n are removed from the weighted edge map Wed, and the clear edge map Aed of the large area target can be retained.

[0037] The image segmentation method of the present invention is based on the above-mentioned target edge extraction method, such as Figure 1 As shown, firstly, the target edge extraction method of the present invention is used for rough segmentation to filter out the interference of tiny features and fine noise, and to repair and protect the edges of various typical targets, that is, to perform rough classification based on edge weights to obtain targets containing accurate edges; at the same time, neural network segmentation is performed to identify targets with complex structures and easy to confuse, that is, to perform accurate classification of neural networks to obtain accurate cognitive results describing details; finally, through the mutual optimization of rough segmentation and neural network segmentation results, that is, edge replacement, a segmentation result with strong consistency of ground objects, accurate edges, clear categories and high credibility of details is obtained.

[0038] It should be noted that since the edge weights of large-area objects and small-area objects are not fully evaluated, when the neural network recognizes the edge of the object, the typical edge is mixed with subtle details, tiny noise and other information and is easily lost. The coarse segmentation result Rsg of the present invention only contains the clear edges of typical objects without the interference of detail noise, and can be used to repair the edges in the neural network segmentation result. Specifically, the present invention first selects a certain type of target Ngc from the neural network segmentation result Nsg, finds the corresponding target Rgc in the coarse segmentation result Rsg according to its position, and then uses the clear edge of the coarse segmentation result target Rgc to replace the edge of the neural network result target Ngc, so as to repair the target edge.

[0039] The image segmentation neural network described in the present invention is obtained after initial network training, wherein the initial network can select an existing neural network, the training method adopts a conventional network training method, and the training data set is data collected by itself according to the actual target or a corresponding data set that has been made public (such as the WHU-RS19 data set). The improved shuffle neural network structure described in this article is an improvement made on the basis of the existing shuffle neural network structure, specifically by adding residual connections before and after each shuffle structure in the shuffle neural network to obtain a residual shuffle structure, and then connecting a self-attention module to each residual shuffle structure to obtain, such as Figure 3 As shown. The improved shuffle neural network of the present invention uses transposed convolution and pooling technology for scale scaling, and through a lightweight residual shuffle feature cognitive structure, deeply analyzes super-resolution small-scale features, low-resolution large-scale features and original resolution original-scale features, and after the analysis is completed, uses attention feature evaluation structure features to adaptively find features with greater segmentation contributions, thereby highlighting important information in multi-channel features, reducing the interference of useless information, and improving the learning performance of the network.

[0040] The following are specific embodiments provided by the inventor to further illustrate the present invention.

[0041] Embodiment 1:

[0042] This embodiment uses the edge extraction method of the present invention to Figure 3 The original image (a) is processed to obtain the result shown in Figure (b). In the target edge extraction method of this embodiment, n=7, a bilinear interpolation resampling method is used in Step 1.1 to establish an image pyramid for the original image, a Gaussian Laplacian filter is used in Step 1.2 to extract the edge of the image pyramid, and a nearest neighbor interpolation resampling method is used in Step 1.3 to adjust the size of each layer in the edge pyramid.

[0043] Embodiment 2:

[0044] The image segmentation method of the present invention is used in this embodiment to respectively Figure 3 and 4 The original image dataset of the example is segmented;

[0045] In the target edge extraction method of this embodiment, n=7, a bilinear interpolation resampling method is used in Step 1.1 to establish an image pyramid for the original image, a Gaussian Laplacian filter is used in Step 1.2 to extract the edge of the image pyramid, and a nearest neighbor interpolation resampling method is used in Step 1.3 to adjust the size of each layer in the edge pyramid;

[0046] The initial network of the image segmentation neural network in this embodiment is an improved shuffle neural network, and the training data set used is self-collected such as Figure 4 (a) and 5(a) are the datasets shown in the original figure. Figure 4 (b) and Figure 5 (b) is the ground truth of the corresponding original image obtained by manual labeling.

[0047] Embodiment 3:

[0048] The difference between this embodiment and embodiment 1 is that the initial network of the image segmentation network is MASK-RCNN, and the network of embodiment 1 is trained using exactly the same samples.

[0049] Comparative Example:

[0050] This comparison uses the MASK-RCNN network trained on the same dataset. Figure 4 and 5 The original image in is segmented.

[0051] Two sets of objective indicators were used to evaluate the segmentation accuracy of the above embodiments and comparative examples:

[0052] (1) Two indicators whose larger the better: overall segmentation accuracy (OA) and kappa coefficient (K), which describe the similarity between the segmentation result and the ground truth [Song K S. Globally convergent algorithms forestimating generalized gamma distributions in fast signal and image processing [J]. IEEE Transactions on Image Processing, 2008, 17 (8): 1233-1250.];

[0053] (2) Two indicators, the smaller the better: quantity disagreement (QD) and allocation disagreement (AD) parameters [Pontius Jr RG, Millones M. Death to Kappa: birth of quantity disagreement and allocation disagreement for accuracy assessment [J]. International Journal of Remote Sensing, 2011, 32 (15): 4407-4429.], which are used to indicate category recognition errors and location recognition errors respectively. The results are shown in Tables 1 and 2.

[0054] Table 1 Figure 4 The categories and accuracy of the original image segmentation results shown

[0055]

[0056] Table 2 Figure 5 The categories and accuracy of the original image segmentation results are shown

[0057]

Claims

1. A method for extracting the edge of a target in an original image, wherein: The method comprises: Step 1.1, build an image pyramid for the original image to obtain an image pyramid containing n layers of information; any layer is represented by Img i , i=1,2,...,n;n≥2; Step 1.2, extract the edge of the image pyramid to obtain the edge pyramid, where any layer Img i The Edge of Ed i , i=1,2,...,n; Step 1.3, resize each layer in the edge pyramid to the size of the original image, and any layer after resizing is Ued i , i=1,2,...,n; Step 1.4, binarize each layer after Step 1.

3. i The gray level of the pixel is less than the threshold Ed_thd i The value of Ued is 0. i The gray level of the middle pixel is greater than or equal to the threshold Ed_thd i Assign a value of 1 to get any layer of Ued i Binarization layer Bed i , where Ed_thd i =mean(Ued i )-std(Ued i ), mean(Ued i ) is Ued i The mean value of all pixels in, std(Ued i ) is Ued i The standard deviation of all pixels in , i = 1, 2, ..., n; Step 1.5, perform morphological dilation on all binary layers obtained in Step 1.4, where any binary layer Bed i Perform i-1 morphological dilations to obtain the dilated edge Ded of the corresponding layer i , i=1,2,...,n; Step 1.6, add up the dilated edges of all layers to get the cumulative result Sed; Step 1.7, assign the grayscale of pixels in the area Ned_area1 in the accumulation result Sed to 0, and obtain the edge map Wed, wherein the area Ned_area1 is an area composed of pixels whose grayscale in Ded1 is 0; Step 1.8, remove the pixels with grayscale values ​​less than n in the edge map to obtain a clear edge map Aed, and complete the extraction of the target edge.

2. The target edge extraction method according to claim 1, characterized in that: In the Step 1.1, a bilinear interpolation resampling method is used to create an image pyramid for the original image.

3. The target edge extraction method according to claim 1, characterized in that: In the Step 1.2, a Laplacian of Gaussian filter is used to extract edges from the image pyramid.

4. The target edge extraction method according to claim 1, characterized in that: In the Step 1.3, the nearest neighbor interpolation resampling method is used to adjust the size of each layer in the edge pyramid.

5. An image segmentation method, characterized in that: The method comprises: Step 1, performing rough color clustering on the original image Omg to be segmented, comprising: executing the method described in any one of claims 1 to 4 to obtain a clear edge map of the original image Omg; then performing color clustering on the original image Omg according to edge information in the clear edge map and different colors of the original image Omg to obtain a rough clustering result Rsg, wherein the rough clustering result Rsg contains edge information of each target in the clear edge map; Use the image segmentation neural network to segment the original image and obtain the segmentation result Nsg; Step 2: Select any target Ngc from the segmentation result Nsg, find the corresponding target Rgc in the rough clustering result Rsg according to the position of any target Ngc, and then replace the edge of the target Ngc in the segmentation result Nsg with the clear edge of the target Rgc to complete the edge repair segmentation of the target Ngc.

6. The image segmentation method according to claim 5, characterized in that: The image segmentation neural network is obtained after initial network training, and the initial network is a MASK-RCNN network, a shuffle neural network or an improved shuffle neural network. The improved shuffle neural network is obtained by adding residual connections before and after each shuffle structure in the shuffle neural network to obtain a residual shuffle structure, and then connecting a self-attention module to each residual shuffle structure.

7. An image segmentation system, characterized in that: It includes an image edge extraction module, an image segmentation module and an edge replacement module; The image edge extraction module is used to perform rough color clustering on the original image Omg to be segmented, including: obtaining a clear edge map of the original image Omg by the method described in any one of claims 1 to 4; then, according to the edge information in the clear edge map and the different colors of the original image Omg, color clustering is performed on the original image Omg to obtain a rough clustering result Rsg, and the rough clustering result Rsg contains edge information of each target in the clear edge map; The image segmentation module uses an image segmentation neural network to segment the original image to obtain a segmentation result Nsg; The edge replacement module is used to select any target Ngc from the segmentation result Nsg, find the corresponding target Rgc in the rough clustering result Rsg according to the position of any target Ngc, and then replace the edge of the target Ngc in the segmentation result Nsg with the clear edge of the target Rgc to complete the edge repair segmentation of the target Ngc.

Images