An Image Segmentation Method Combining Superpixel Blocks and Holistically-Nested Edges

Abstract

The present invention relates to the field of image processing technology, and particularly relates to an image segmentation method that fuses superpixel blocks and global nested edges, including S1: defining the number of seed points; S2: marking the edge probability map E; S3: initializing the seed points to generate a pre-segmentation; S4: generating an initial label L0 by applying the spatial distance of FSLIC and the color energy of SEEDS; S5: defining the superpixel edges as strong boundaries or weak boundaries according to the relationship between E and L0; S6: re-dividing the superpixel edges based on the weak boundaries according to the color energy; S7: outputting the sugarcane aphid image segmentation result. The present invention introduces a probability edge map and a fusion clustering strategy into the superpixel segmentation method, giving full play to the advantages of the superpixel method in terms of fast processing speed and the ability to cope with the variability of natural light; the present invention is an efficient, reliable, and excellent superpixel segmentation method with outstanding segmentation performance.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image segmentation method that combines superpixel blocks and holistic nested edges. Background Art

[0002] The sugarcane aphid was first discovered in Beaumont, Texas in 2013. The sugarcane aphid has a high dispersal ability and reproduction rate, and causes great harm to crops, often resulting in pest outbreaks every year. Therefore, it is crucial for farmers to efficiently and quickly identify pests to quickly understand the degree of pest damage and timely apply pesticides to avoid yield losses.

[0003] The main traditional method for detecting sugarcane aphids still requires grass-roots workers to enter the farmland to observe the distribution of aphids on the leaves, and rely on experience to judge the quantity level of sugarcane aphids on the leaves and diagnose the pest and disease situation in this area. This method has the characteristics of large workload and low efficiency. It cannot quickly and accurately reflect and predict the occurrence of pests and diseases. Compared with traditional manual counting, automatic detection based on images will significantly improve the efficiency of pest early warning and should be widely used in pest management.

[0004] Image-based detection systems can effectively identify pests (such as sugarcane aphids). Traditional image processing methods, such as methods based on multiple features, support vector machines (SVMs), and threshold-based methods, have shown good performance in pest identification. In recent years, deep learning models such as convolutional neural networks (CNNs), Faster Region-based Convolutional Neural Networks (Faster RCNNs), Feature Pyramid Networks (FPNs), and You Only Look Once (YOLO) have also been well applied in pest identification.

[0005] Most of the algorithms in the above research are applied in scenarios with stable illumination or simple backgrounds (such as controlled illumination, greenhouses, and pest traps), without considering the robustness of pest identification methods in the variable natural illumination environment of farmland. Due to the small size of pests, traditional pixel-based methods take a long time to identify pests in images that usually contain millions of pixels. The superpixel method significantly speeds up subsequent processing without significantly sacrificing accuracy, and at the same time, the superpixel method can cope with the variability of natural illumination. Therefore, compared with the above pixel-based methods, the superpixel-based method of the present invention is considered to be a more effective method for segmenting sugarcane aphid images under unstructured farmland backgrounds and randomly variable natural illumination conditions. Summary of the Invention

[0006] In view of the deficiencies of existing algorithms, the present invention provides a superpixel segmentation method based on an unstructured farmland background and randomly variable natural light, integrating a probability edge map and a superpixel clustering strategy, and dedicated to being efficient, reliable, and excellent in segmentation performance in the image processing of millions of pixels.

[0007] The technical solution adopted by the present invention is as follows: An image segmentation method integrating superpixel blocks and globally nested edges includes the following steps:

[0008] S1: Input an image and define the number of seed points according to the target size of sugarcane aphids;

[0009] Furthermore, collect the original images of grain sorghum leaves with sugarcane aphids as a data set; the image data set includes four light intensities: low light, strong light, direct light, and diffused light; separate the collected original images into several sub-images of a certain number of pixels;

[0010] Determine the number of seed points K according to the target size of sugarcane aphids in the image data set;

[0011] Furthermore, K = 400;

[0012] S2: Mark the image based on the Holistically-Nested Edge Detection (HNE) algorithm and mark it as an edge probability map E;

[0013] Furthermore, use the Holistically-Nested Edge Detection algorithm to train and predict the whole image, extract features at multiple scales and levels, mark the image, and mark it as an edge probability map E;

[0014] S3: Initialize the seed points to generate a pre-segmentation;

[0015] Furthermore, initialize K seed points by uniformly distributing the seed points, generate a pre-segmentation, and move the clustering center to the lowest gradient position in the n×n neighborhood;

[0016] S4: Apply the spatial distance of the Fast Simple Linear Iterative Clustering (FSLIC) algorithm and the color energy of the Superpixels Extracted via Energy-Driven Sampling (SEEDS) algorithm to generate an initial label L0, and apply the Predict Candidate Cluster Elimination (PCCE) inequality to optimize the iterative clustering center step to accelerate convergence;

[0017] The specific content is as follows:

[0018] The SEEDS algorithm is an energy optimization method. Its generation method is to iteratively optimize the energy function E(s) on the color energy H(s) and the boundary energy G(s); based on the energy function E(s), pixels are converted between adjacent superpixels, thus quickly changing the boundaries; the number, compactness, and number of iterations of the superpixel blocks can be controlled by the SEEDS algorithm. The formula for the energy function E(s) is:

[0019] E(s) = H(s) + γG(s) (1)

[0020] where H(s) is the color energy, G(s) is the boundary energy, and γ is the balance parameter;

[0021] The formula for the color energy H(s) is:

[0022]

[0023] where, is the quality metric of the color distribution;

[0024]

[0025] where, is the color histogram of the pixel set in the image; is a closed subset of the color space;

[0026]

[0027] where J(i) represents the color of pixel i, Z is the normalization factor of the histogram, δ(•) is the indicator function, is a closed subset of the color space;

[0028] when the color of the pixel falls into bin j, the indicator function returns 1; contains the pixels in superpixel k;

[0029]

[0030] where, is the pixel set in the image, and s(i) represents the superpixel assigned to pixel i;

[0031] The generation method of the FSLIC algorithm is to convert the image into a five-dimensional feature vector in the LAB color space and XY coordinates, evenly divide the seed points (i.e., initialize the clustering centers) within the image to obtain the initial superpixel distribution, and assign the pixel points within the neighborhood of the seed points to the nearest clustering center through the distance metric determined by the color distance and spatial distance, and continuously iterate this process until convergence; finally, reassign the separated pixels to the nearby superpixels, and regular superpixels are obtained after clustering; the color distance matrix D c (i, C k ) and the spatial distance matrix D s (i, C k ) are as follows:

[0032]

[0033]

[0034] Among them, C k is a set of clustering centers with the same feature class, D c (i, C k ) is the color distance matrix between i and C k , D s (i, C k ) is the spatial distance matrix between i and C k , l, a, b are three-dimensional color features, and x and y are two-dimensional features; the center of the i-th superpixel is described as C i = l i + a i + b i ;

[0035] The PCCE inequality is used to optimize the iterative clustering center step and accelerate convergence, and its formula is:

[0036] D″(i, C k ) ≥ d′ min (8)

[0037] In the formula, d′ min represents the current minimum distance, and D″(i, C k ) refers to the next candidate distance;

[0038] S5: Define the superpixel edge as a strong boundary or a weak boundary according to the relationship between E and L0;

[0039] Furthermore, according to the edge probability map E and the generated initial label L0, the fusion of the spatial distance in FSLIC and the color energy in SEEDS is reassigned: L0 is equal to E to define the superpixel edge as a "strong boundary", otherwise it is a "weak boundary", and the strong boundary is drawn as the superpixel boundary;

[0040] S6: Redivide the superpixel edges based on color energy according to weak boundaries;

[0041] Furthermore, if the color energies of the pixel blocks on both sides of E are equal, then remove E; otherwise, draw the edge E as a superpixel boundary.

[0042] S7: Output the segmentation result of the sugarcane aphid image;

[0043] The beneficial effects of the present invention are:

[0044] Aiming at the problem of low robustness of the pest recognition method under the variable natural lighting environment in farmland, the present invention introduces the probability edge map and the fusion clustering strategy into the superpixel segmentation method, giving full play to the advantages of the superpixel method in fast processing speed and coping with the variability of natural lighting. Experiments were carried out with small-sized sugarcane aphids as the research object, verifying that the method proposed by the present invention is an efficient and reliable superpixel segmentation method with excellent segmentation performance, suitable for unstructured farmland backgrounds and randomly variable natural lighting conditions. Description of the Drawings

[0045] Figure 1 is the overall flowchart of the sugarcane aphid image segmentation algorithm of the present invention;

[0046] Figure 2 is the roadmap of the sugarcane aphid image segmentation algorithm of the present invention;

[0047] Figure 3 is an example diagram of the number of targets under different numbers of superpixels K;

[0048] Figure 4 Visual comparison diagrams with eight existing superpixel algorithms under four natural light conditions;

[0049] Figure 5 Performance comparison diagrams with eight existing superpixel algorithms under four natural light conditions. Detailed Embodiments

[0050] The present invention will be further described below in conjunction with the drawings and embodiments. This figure is a simplified schematic diagram, only illustrating the basic structure of the present invention in a schematic manner, so it only shows the components related to the present invention.

[0051] As Figure 1-2 shown, an image segmentation method integrating superpixel blocks and globally nested edges includes the following steps:

[0052] S1: Input an image and define the number of seed points according to the target size of sugarcane aphids; the specific content is:

[0053] The input images are from an image dataset of grain sorghum leaves with sugarcane aphids taken using a camera under natural light in farmland (image resolution is 2448×3264); the vertical distance between the camera and the leaves is approximately 0.2 meters; this image dataset includes four light intensities: low light (light intensity less than 20 kLux), strong light (light intensity greater than 30 kLux), direct light (taken between noon on a sunny day with the sun angle greater than 40° and 1:00 pm), and diffused light (light intensity between 20 kLux and 30 kLux during shooting). To evaluate the performance of the algorithm under various natural light conditions, we separated each original image into 8×8 sub-images of 408×306 pixels.

[0054] The number of seed points set for the image has a great influence on the segmentation accuracy of sugarcane aphids, such as Figure 3 At the appropriate number of seed points K, the sugarcane aphid targets are segmented relatively completely. We determined that when the number of seed points K is 400 according to the sugarcane aphid target size in the image dataset, a good segmentation effect on the sugarcane aphid targets can be achieved.

[0055] S2: Mark the image based on the Holistically-Nested Edge Detection (HED) algorithm and label it as the edge probability map E. The specific content is as follows:

[0056] Use the Holistically-Nested Edge Detection (HED) algorithm to train and predict the entire image, and extract features at multiple scales and levels; it can obtain rich hierarchies conducive to edge and object boundary detection. With the help of the generated edge probability map E, the iterative method of superpixels can capture the edges, thus generating high-quality boundaries; the present invention uses this algorithm to mark the image and label it as the edge probability map E.

[0057] S3: Initialize the seed points to generate a pre-segmentation. The specific content is as follows:

[0058] Furthermore, evenly distribute the seed points to initialize K seed points, generate a pre-segmentation, and move the clustering center to the lowest gradient position in the 2×2 neighborhood.

[0059] S4: Generate the initial label L0 by applying the spatial distance of FSLIC and the color energy of SEEDS, and apply the PCCE inequality to optimize the iterative clustering center step to accelerate convergence. The specific content is as follows:

[0060] The SEEDS algorithm is an energy optimization method. Its generation method is to iteratively optimize the energy function E(s) on the color energy H(s) and the boundary energy G(s) (Formulas 1 - 5); based on the energy function, pixels are transformed between adjacent superpixels, thus quickly changing the boundaries; the number of superpixel blocks, compactness, and the number of iterations can be controlled by the SEEDS algorithm.

[0061] The generation method of the FSLIC algorithm is to convert the image into a five-dimensional feature vector in the LAB color space and XY coordinates, evenly divide seed points (i.e., initialize the clustering centers) within the image to obtain the initial superpixel distribution, and assign the pixel points within the neighborhood of the seed points to the nearest clustering center through the distance metric (Formulas 6-7) determined by color distance and spatial distance. This process is continuously iterated until convergence. Finally, the post-processing step reassigns the separated pixels to nearby superpixels, and regular superpixels are obtained after clustering.

[0062] S5: According to the edge probability map E and the generated initial label L0, reassign the fusion of the spatial distance in FSLIC and the color energy in SEEDS: L0 defines the superpixel edge as a "strong boundary" according to E, otherwise it is a "weak boundary", and draw the strong boundary as the superpixel boundary.

[0063] S6: Redivide the superpixel edge based on the weak boundary according to the color energy: If the color energies of the pixel blocks on both sides of E are equal, remove E; otherwise, draw the E edge as the superpixel boundary.

[0064] S7: Output the segmentation result of the sugarcane aphid image.

[0065] The automatic detection system based on images demonstrates the effective recognition ability of pests (such as sugarcane aphids). However, most methods are developed using images taken under stable lighting conditions, and the factors of natural lighting are rarely considered. The present invention is a superpixel method that fuses the overall nested edge detection (Fuse FSLIC, SEEDS, and Edge map algorithms, FSE), and compares it with eight other state-of-the-art superpixel algorithms, which are implemented on image datasets taken under various natural lighting conditions such as low light, strong light, direct light, and diffused light. Through Figure 4 It can be seen that the method proposed by the present invention can accurately segment sugarcane aphids under different lighting conditions. At the same time, through comparison, it is found that diffused light is an ideal lighting condition for sugarcane aphid recognition. We evaluate the algorithm performance according to visual quality, true value deviation (such as target compactness T-C, boundary recall rate B-R, and under-segmentation error US-E), and segmentation running time Runtime. As Figure 5 shown, compared with other algorithms, the FSE algorithm shows relatively good performance in terms of target compactness (0.61), boundary recall rate (0.76), and running time (0.13 s); the present invention proposes an efficient and reliable superpixel segmentation method with excellent segmentation performance, which is applicable to unstructured farmland backgrounds and random and variable natural lighting conditions.

[0066] The formulas for the target compactness T-C (Equations 9 - 11), boundary recall B-R (Equation 12), and oversegmentation error US-E (Equation 13) are as follows:

[0067]

[0068]

[0069]

[0070] where S represents the set of pixels in a superpixel, I represents the set of pixels in an image, |I| is the size of the image, and |S| is the size of the superpixel; r is the radius of the circle corresponding to the perimeter of the superpixel; L s is the perimeter of the superpixel; Q S is the isoperimetric quotient; A S and A C are the areas of the superpixel and the circle, respectively;

[0071]

[0072] where S and G represent the superpixel segmentation and the corresponding ground truth segmentation, respectively; TP(S, G) is the true positive, representing the number of boundary pixels that belong to both G and S within the deviation range of γ and within the tolerance range of γ; FN(S, G) is the false negative, representing the number of boundary pixels that belong to G but not S within the tolerance range of γ; γ is the allowable deviation from the actual value;

[0073]

[0074] where g i is the ground truth segmentation; s i is the segmented superpixel; represents the percentage of pixels in s i that intersect with g i ; Area(g i ) is the area of the superpixel.

[0075] Based on the above ideal embodiments of the present invention as inspiration, through the above description, relevant workers can completely make various changes and modifications without departing from the technical idea of this invention. The technical scope of this invention is not limited to the content in the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. An image segmentation method that fuses superpixel blocks and holistic nested edges, characterized in that It includes the following steps: S1: Input an image and define the number of seed points according to the target size of sugarcane aphids; S2: Mark the image based on the Holistically-Nested Edge Detection (HED) algorithm and label it as the edge probability map E; S3: Initialize the seed points to generate a pre-segmentation; S4: Generate the initial label L0 by applying the spatial distance of FSLIC and the color energy of SEEDS, and apply the prediction candidate clustering to eliminate the inequality to optimize the iterative clustering center step to accelerate convergence; The calculation of the color energy of the SEEDS includes: Based on the energy function E(s), pixels are transformed between adjacent superpixels, thus quickly changing the boundary; The number of superpixel blocks, compactness, and number of iterations can be controlled by the SEEDS algorithm. The formula for the energy function E(s) is: E(s) = H(s) + γG(s) (1) where H(s) is the color energy, G(s) is the boundary energy, and γ is the balance parameter; The formula for the color energy H(s) is: Among them, is a quality metric of color distribution; Among them, is the color histogram of the pixel set in the image ; is a closed subset of the color space; where J(i) represents the color of pixel i, Z is the normalization factor of the histogram, and δ(·) is the indicator function, is a closed subset of the color space; When the color of a pixel falls into bin j, the indicator function returns 1; including the pixels in superpixel k; Among them, is a pixel set in the image, and s(i) represents the superpixel assigned to pixel i thereof; The spatial distance of the FSLIC is to convert the image into a five-dimensional feature vector in the LAB color space and XY coordinates, evenly divide seed points in the image to obtain the initial superpixel distribution, and assign the pixel points in the neighborhood of the seed points to the nearest clustering center through the distance metric determined by the color distance and the spatial distance. This process is continuously iterated until convergence; finally, the separated pixels are reassigned to the nearby superpixels, and regular superpixels are obtained after clustering; color distance matrix D c (i, C k ) and spatial distance matrix D s (i, C k ) are as follows: Among them, C k is a set of cluster centers with the same feature class, D c (i, C k ) is the color distance matrix between i and C k , D s (i, C k ) is the spatial distance matrix between i and C k , l, a, b are three-dimensional color features, and x and y are two-dimensional features; the center of the i-th superpixel is described as C i = l i + a i + b i ; S5: Define the superpixel edges as strong edges or weak edges according to the relationship between E and L0; S6: Re-partition the superpixel edges based on the weak edges according to the color energy; S7: Output the sugarcane aphid image segmentation result.

2. The image segmentation method that fuses superpixel blocks and overall nested edges according to claim 1, wherein The step S1 includes: Collect the original image of the grain sorghum leaf with sugarcane aphids as the data set; The image data set includes four light intensities: low light, strong light, direct light, and diffused light; Separate the collected original image into several sub-images of a certain number of pixels; Determine the number of seed points K in the image data set according to the target size of sugarcane aphids.

3. The image segmentation method integrating superpixel blocks and overall nested edges according to claim 2, characterized in that: The number of seed points K = 400.

4. The image segmentation method integrating superpixel blocks and global nested edges according to claim 1, characterized in that, The step S2 includes: Use the Holistically-Nested Edge Detection (HED) algorithm to train and predict the whole image, extract features at multiple scales and levels, mark the image, and label it as the edge probability map E.

5. The image segmentation method integrating superpixel blocks and overall nested edges according to claim 1, wherein, The step S3 includes initializing K seed points by uniformly distributing the seed points, generating a pre-segmentation, and moving the clustering center to the lowest gradient position in the n×n neighborhood.

6. The image segmentation method integrating superpixel blocks and global nested edges according to claim 1, characterized in that The step S5 includes: According to the edge probability map E and the generated initial label L0, re-distribute the fusion of the spatial distance in FSLIC and the color energy in SEEDS: L0 is equal to E to define the superpixel edge as a "strong edge", otherwise it is a "weak edge", and draw the strong edge as the superpixel boundary.

7. The image segmentation method integrating superpixel blocks and global nested edges according to claim 1, wherein The step S6 includes: If the color energies of the pixel blocks on both sides of E are equal, remove E; otherwise, draw the E edge as the superpixel boundary.

Images