Image over-segmentation method and device, storage medium and terminal

Abstract

The invention provides an image over-segmentation method and device, a storage medium and a terminal, and belongs to the technical field of image processing, and the method comprises the steps: carrying out the preprocessing of an input image; dividing the preprocessed image into regular initial grids, and sampling on the initial grids to obtain an initial clustering center; identifying a search area of the initial clustering center, searching similar pixel blocks, and determining initial superpixels; according to the initial clustering center and the search area, calculating and updating the membership degree and the clustering center of the initial superpixels, and generating high-quality mixed superpixels by judging superpixels, uncertain pixels and noise / abnormal values; the algorithm efficiency is optimized in a membership filtering mode, and the membership of misclassified pixels is corrected; and a merging strategy is adopted for the current superpixels, the membership degree of the pixels is optimized, and the connectivity of discontinuous superpixels is enhanced, so that the over-segmentation of the image is realized. According to the method, effective over-segmentation of the image is realized, the semantic information resolution is improved, and the image features are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image over-segmentation method, device, storage medium and terminal. Background Art

[0002] Image over-segmentation, as a type of image segmentation, is often used as a preprocessing module for computer vision applications, such as depth estimation, target localization, image analysis, etc. A good over-segmentation algorithm can effectively enhance the performance of computer vision tasks.

[0003] Superpixels are small perceptually meaningful regions generated by over-segmentation of an image, which can be used to replace a pixel grid with a rigid structure. Using a small number of superpixels to replace a large number of pixels (such as hundreds of pixels for tens of thousands) can significantly reduce the computational complexity of subsequent operations and effectively improve system efficiency.

[0004] In current image over-segmentation, superpixels are often considered to be pure superpixels, and each pixel is assigned to a superpixel. At the same time, all pixels in each superpixel are considered to have the same label, which seriously affects the accuracy of subsequent interpretation and processing. In addition, the impact of noise / outliers is often ignored, and the generated superpixels are too sensitive to noise / outliers and have poor quality. Summary of the invention

[0005] The present invention provides an image over-segmentation method, device, storage medium and terminal, which are used to solve the consistency problem of pixel labels in superpixels and the noise / outlier problem in data in the prior art, and realize accurate segmentation of image semantic information.

[0006] In a first aspect, the present invention provides an image over-segmentation method, comprising: preprocessing an input image; dividing the preprocessed image into a regular initial grid, sampling on an initial network to obtain an initial cluster center; identifying a search area for the initial cluster center and searching for similar pixel blocks to determine an initial superpixel; calculating and updating the initial superpixel membership and cluster center based on the initial cluster center and the search area, generating high-quality mixed superpixels by determining superpixels, uncertain pixels, and noise / outliers; optimizing algorithm efficiency by means of membership filtering, and correcting the membership of misclassified pixels; adopting a merging strategy for the current superpixel, optimizing pixel membership, and enhancing the connectivity of discontinuous superpixels to achieve image over-segmentation.

[0007] The present invention provides an image over-segmentation method, wherein the operation of preprocessing an input image comprises one or a combination of the following operations: format conversion operation, resampling operation, grayscale operation, geometric transformation operation, image enhancement operation and standardization operation.

[0008] The present invention provides an image over-segmentation method, wherein the initial grid size is a space width parameter, which is used to determine the initial size of a superpixel and the number of superpixels.

[0009] The present invention provides an image over-segmentation method, which identifies a search area of ​​an initial cluster center and searches for similar pixel blocks to determine an initial superpixel, including: moving the initial cluster center to the lowest gradient position of a local space adjacent to the search area; for each cluster center, calculating the distance from it to all points in the surrounding search area; assigning image pixels to the nearest cluster center in the search area, searching for similar blocks, and forming an initial superpixel.

[0010] The present invention provides an image over-segmentation method, which calculates and updates the initial superpixel membership and cluster center according to the initial cluster center and the search area, and generates high-quality mixed superpixels by determining superpixels, uncertain pixels and noise / outliers, including: using the Lagrange multiplier method to calculate the pixel membership and cluster center according to the initial cluster center and the given search area; wherein clustering is realized by using data reconstruction, sparse constraints are introduced into the deviations of the measured data and the reconstructed data, and a sparse constraint-fuzzy C-means clustering model is constructed to realize noise / outlier suppression and feature preservation in the clustering process; in the sparse constraint-fuzzy Based on the C-means clustering model, neighborhood information constraints are introduced to construct a local information-sparse constraint-fuzzy C-means clustering model to improve the noise / outlier heterogeneity performance and clustering robustness of the algorithm and generate high-quality hybrid superpixels; the deviation matrix of the measured data and its reconstructed data is solved according to the soft threshold operator; the membership matrix and cluster center vector are iteratively updated using an iterative optimization strategy to obtain the optimal conditions until convergence; based on the pixel membership, it is determined whether the pixel belongs to a specific superpixel. If the difference between the maximum and the second maximum membership values ​​is greater than the median membership value, the pixel belongs to the superpixel to which the cluster center belongs, otherwise it is an uncertain pixel.

[0011] The present invention provides an image over-segmentation method, which optimizes algorithm efficiency by means of membership filtering and corrects the membership of wrongly classified pixels, including: using membership filtering to replace the contribution of distance calculation between pixels and cluster centers in a local space neighborhood; using membership filtering to optimize membership division and correct the wrongly classified pixels.

[0012] The present invention provides an image over-segmentation method, which adopts a merging strategy for current superpixels, optimizes pixel membership, and enhances the connectivity of discontinuous superpixels to achieve image over-segmentation, including: firstly arranging all connected areas in ascending order of size and renumbering them, starting from the smallest, and sequentially merging small-sized areas into their adjacent areas; the small-sized area is an area whose size is smaller than a preset limit; the small-sized area is merged into the adjacent connected area, and if the small-sized area is adjacent to multiple connected areas, the one with the closest color is selected for merging.

[0013] In a second aspect, the present invention further provides an image over-segmentation device, comprising:

[0014] An image preprocessing module, used for preprocessing an input image;

[0015] The cluster center initialization module is used to divide the preprocessed image into regular initial grids and sample the initial cluster centers on the initial network;

[0016] A search region determination module is used to identify the search region of the initial cluster center and search for similar pixel blocks to determine the initial superpixel;

[0017] The superpixel generation module is used to calculate and update the initial superpixel membership and cluster center according to the initial cluster center and the search area, and generate high-quality mixed superpixels by determining superpixels, uncertain pixels and noise / outliers;

[0018] The membership correction module is used to optimize the algorithm efficiency by means of membership filtering and correct the membership of the wrongly classified pixels;

[0019] The connectivity enhancement module is used to adopt a merging strategy for the current superpixel, optimize the pixel membership, and enhance the connectivity of discontinuous superpixels to achieve over-segmentation of the image.

[0020] In a third aspect, the present invention provides a terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-described image over-segmentation methods are implemented.

[0021] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described image over-segmentation methods.

[0022] The image over-segmentation method, device, storage medium and terminal provided by the present invention solve the over-segmentation problem by converting it into a clustering problem. In view of the consistency problem of pixel labels in superpixels and the noise / outliers existing in the data, sparse constraints and neighborhood information constraints are introduced into the reconstructed data deviation, and noise / outlier suppression and feature retention are achieved in the over-segmentation process, thereby improving the robustness of high-discrimination features and the accuracy of over-segmentation of complex images. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 It is a flowchart of the image over-segmentation method provided by the present invention;

[0025] Figure 2 It is a schematic diagram of the comparison between superpixels and pixels provided by the present invention;

[0026] Figure 3 It is a structural schematic diagram of the image over-segmentation device provided by the present invention;

[0027] Figure 4 It is a schematic diagram of the image over-segmentation result provided by the present invention;

[0028] Figure 5 It is a schematic diagram of the comparison results of segmentation accuracy of different image over-segmentation algorithms provided by the present invention;

[0029] Figure 6 It is a schematic diagram of the comparison results of the running time of different image over-segmentation algorithms provided by the present invention;

[0030] Figure 7 It is a structural schematic diagram of the terminal provided by the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] It should be noted that, in the description of the embodiments of the present invention, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0033] The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0034] Combine the following Figure 1-Figure 7 The image over-segmentation method and device provided by the embodiments of the present invention are described. The method can be implemented by relying on a computer program and can be run on an image over-segmentation system based on the von Neumann system. The computer program can be integrated into an application or run as an independent tool application. It should be noted that the present invention can achieve robust over-segmentation and is suitable for image feature enhancement.

[0035] Figure 1 is a flow chart of the image over-segmentation method provided by the present invention, such as Figure 1 As shown, including but not limited to the following steps:

[0036] Step 101: pre-process the input image.

[0037] The images to be processed may be images acquired by cameras or other image acquisition devices installed in various places such as roads, communities, airports / train stations, large venues, etc. The image scenes may be in motion or in a static state.

[0038] The image in the present invention may be an ISAR (Inverse Synthetic Aperture Radar) image.

[0039] In the embodiments of the present application, image acquisition and feature information collection are the prerequisites for image over-segmentation, and are an important branch in the field of computer vision and image processing. It purposefully emphasizes the overall or local characteristics of the image, converts the acquired image information into digital signals, and transmits the digital signals to the image processing system and the computer, respectively. Image acquisition mainly involves imaging and digital-to-analog conversion technology. With CCD electrically coupled device technology as the core, current image acquisition equipment includes but is not limited to: black and white cameras, color cameras, scanners, digital cameras, microscopic camera equipment, infrared cameras, high-speed cameras, film scanners, remote sensing satellites, laser radars, robot vision, etc. Commonly used image acquisition methods include but are not limited to: hierarchical MS matching iterative tracking method, incremental histogram method, CCD camera acquisition method, SSACA+EP algorithm, nuclear magnetic resonance imaging, computed tomography, ultrasound imaging, nuclear medicine imaging, optical coherence tomography, reverse time migration imaging, digital terrain model imaging, etc.

[0040] These algorithms can be used alone or in combination to achieve better image acquisition results. The choice of which algorithm usually depends on the specific problem of the image and the acquisition goal.

[0041] In one possible implementation, when performing image over-segmentation, first, the target image is acquired by cameras or other image acquisition devices installed in various places such as roads, communities, airports / train stations, and large venues. After acquisition, the acquisition device sends the target image to the image analysis terminal via wired or wireless means. When the image analysis terminal obtains the target image to be over-segmented, it performs format conversion, grayscale, geometric transformation, resampling, image enhancement, and standardization on the image to ensure the availability and quality of the image data, and generate the final pre-processed image to be over-segmented.

[0042] Optionally, the format conversion, grayscale, geometric transformation, resampling, image enhancement and standardization are in an and / or relationship; and the preprocessing operation is not limited to format conversion, grayscale, geometric transformation, resampling, image enhancement and standardization preprocessing methods, and their order is not limited.

[0043] Step 102: Divide the preprocessed image into regular initial grids, and sample on the initial network to obtain initial cluster centers.

[0044] Wherein, for an image I in the CIELAB color space, the color at a pixel (x, y) is expressed as c(x, y) = (l(x, y), a(x, y), b(x, y)).

[0045] Optionally, when the cluster centers are initialized, the image is divided into k regular initial grids (which may be square grids); wherein the initial grid size is a spatial width parameter, which defines the initial size of the superpixel and also indirectly defines the number of superpixels.

[0046] Assume that there are N pixels on image I and k initial cluster centers C are sampled on a regular grid i =[l i ,a i ,b i ,x i ,y i ] T , the sampling interval is The expected superpixel size is smaller than

[0047] Common initialization methods include:

[0048] (1) Random initialization

[0049] Randomly select k data points from the data set as the initial clustering centers. This method is simple and easy to implement, but it is more dependent on the initial position. Different initial positions may lead to different clustering results. In addition, random initialization may cause the algorithm to converge to a local optimal solution.

[0050] (2) k-means++

[0051] k-means++ is an improved initialization method. It first randomly selects a sample as the first cluster center, then performs probabilistic sampling based on the distance from the existing cluster center, and selects the next cluster center until k cluster centers are selected. This can ensure the initialization quality of the initial cluster center as much as possible and avoid falling into the local optimal solution.

[0052] (3) Uniformly distributed sampling

[0053] K samples are uniformly sampled from the sample space as initial cluster centers.

[0054] (4) Manual designation

[0055] Based on domain knowledge or experience, k sample groups are manually specified as initial clustering centers.

[0056] These algorithms can be used alone or in combination to achieve better cluster center selection results. The choice of which algorithm usually depends on the specific problem and goal.

[0057] Step 103: Identify the search area of ​​the initial cluster center (superpixel cluster center) and search for similar pixel blocks to determine the initial superpixel.

[0058] In order to avoid the cluster center falling on the edge of the superpixel or the noise pixel, the initial cluster center is moved to the lowest gradient position of the adjacent local space of the 3×3 area; the 3×3 area is the search area.

[0059] Optionally, the gradient is calculated using the Sobel method

[0060]

[0061] Among them, G x and G y Represent the gradient values ​​in the horizontal and vertical directions respectively, [-1 0 1] and [-1 -2-1; 0 0 0; 121] represent the convolution kernel matrices in the horizontal and vertical directions respectively.

[0062] Optionally, when calculating the gradient value, the Sobel operator is convolved with the image, and then the gradient value in the convolution result is obtained. Specifically, the following steps can be performed:

[0063] (1) Convert the original image into a grayscale image for easy calculation;

[0064] (2) Fill the grayscale image to avoid incomplete pixel values ​​at the edge of the convolution result;

[0065] (3) Convolve the Sobel operator matrix with the filled grayscale image to obtain the gradient values ​​in the horizontal and vertical directions;

[0066] (4) Based on the obtained gradient value, the gradient size and direction of each pixel can be further calculated.

[0067] Optionally, for each cluster center, calculate the distance from it to all points in the surrounding 3×3 area; where the Euclidean distance is used to calculate the distance between the point x=(x1,...,x n ) and y=(y1,...,y n ):

[0068]

[0069] Assign image pixels to the nearest cluster center in a 3×3 neighborhood, search for similar blocks, and form initial superpixels;

[0070] By using pixel neighborhoods, the scope of the search area is effectively reduced, greatly reducing the number of distance calculations.

[0071] Step 104: Calculate and update the initial superpixel membership and cluster center based on the initial cluster center and the search area, and generate high-quality mixed superpixels by determining superpixels, uncertain pixels, and noise / outliers.

[0072] Optionally, step 104 specifically includes the following steps:

[0073] According to the initial cluster center and the given search area, the Lagrange multiplier method is used to calculate the pixel membership and cluster center;

[0074] Among them, clustering is achieved by using data reconstruction, introducing sparse constraints into the deviations of the measured data and its reconstructed data, and constructing a sparse constraint-fuzzy C-means clustering model to achieve noise / outlier suppression and feature preservation in the clustering process;

[0075] Based on the sparse constraint-fuzzy C-means clustering model, neighborhood information constraints are introduced to build a local information-sparse constraint-fuzzy C-means clustering model to improve the noise / outlier heterogeneity performance and clustering robustness of the algorithm and generate high-quality mixed superpixels.

[0076] Solve the deviation matrix of the measured data and its reconstructed data according to the soft threshold operator;

[0077] Adopt iterative optimization strategy to iteratively update membership matrix and cluster center vector to obtain optimal conditions until convergence;

[0078] Based on the pixel membership, it is determined whether the pixel belongs to a specific superpixel. If the difference between the maximum and the second maximum membership is greater than the median membership, the pixel belongs to the superpixel to which the cluster center belongs, otherwise it is an uncertain pixel.

[0079] The following is a further explanation of the above steps:

[0080] Optionally, clustering is achieved based on improved fuzzy C-means and data reconstruction, sparse constraints are introduced into the deviations of the measured data and their reconstructed data, and a sparse constraint-fuzzy C-means clustering model is constructed to achieve noise / outlier suppression and feature preservation in the clustering process, as follows:

[0081]

[0082] Where p>0, r is the neighboring pixel of j. j Represents a local window centered on j. r∈N j represents the set of adjacent pixels including j. l ) T is the regularization vector. rj is the Euclidean distance between r and j. r is the deviation vector of pixel r. l is the data dimension. U={u ij} represents the membership matrix, i=1,...,c,j=1,...,n, It indicates the membership of the jth measurement data to the i-th cluster center, and for any pixel, its membership satisfies:

[0083]

[0084] V={v1,...,v i ,...v c} represents the cluster center vector, v i represents the i-th cluster center, c represents the number of cluster centers. E is the deviation matrix, E q is the qth row vector of the deviation matrix E. m is the fuzzy factor, which represents the weighted index of the fuzzy membership. Generally, m>1. p>0.

[0085] Optionally, according to the initial cluster center and a given search area, the Lagrange multiplier method is used to calculate the pixel membership and the cluster center:

[0086]

[0087] Optionally, the deviation matrix of the measured data and its reconstructed data is solved according to the soft threshold operator:

[0088]

[0089] in, Indicates the cluster center v when the iteration step is t+1 i The qth element of , soft(x,b)=sign(x)max{|x|-b,0} is a soft threshold operator.

[0090] The method for noise / outlier identification in the embodiment of the present invention is as follows:

[0091] There are often various noises and interferences in the process of data acquisition, transmission and storage, which makes the measured data j Its theoretical value There is a deviation between j Therefore, x j and e j The following model is met:

[0092]

[0093] Among them, e j Often represents noise or outliers in an image.

[0094] Let the data set X = {x1,...,x j ,...,x n} represents the measurement data matrix, where x j , is an l-dimensional vector, denoted by x j =(x j1 ,...,x ji ,...x jl ) T .x ji is x j The observation value of the i-th channel in X. The reconstructed data matrix corresponding to X is expressed as E={e1,...,e j ,...,e n} indicates that X and The deviation matrix between Each element in the measurement matrix X and the deviation matrix E satisfies the formula

[0095] For simplicity, we take the grayscale image (l = 1) as an example for analysis. In this case, x j and e j are all scalars (for grayscale images, q is removed for simplicity). Then, if the measurement data x j is far away from all cluster centers, then x j More likely to be noise or outliers. means e j tends to have a larger value. It also shows that by adding sparsity to E, if x j and cluster center v i If the weighted distance of is less than λ / 2, then e j = 0. Choosing a suitable λ helps to identify noise and outliers.

[0096] Optionally, an iterative optimization strategy is used to iteratively update the membership matrix and the cluster center vector to obtain the optimal condition until convergence;

[0097] Optionally, based on the obtained cluster center and the membership of the pixels in the search area, the pixel membership is used to determine whether the pixel belongs to a specific superpixel. The process is as follows:

[0098] Assume U = [U1, ..., U i ...,U n ],U i =[u i1 ,...,u ij ...,u ic ] is the cluster center [C1,...,C c ]’s membership degree. i The maximum value of U is defined as imax , U imax =[U 1max ,...,U imax ,...,Unmax ]. The cluster center corresponding to the maximum membership is defined as C max =[C 1j ,...,C ij ,...,C nj ],j∈1,...,c。Define U submax =[U 1submax ,...,U isubmax ...,U nsubmax ] is the second maximum value vector of U.

[0099] Udiff=U max -U submax =[Udiff1,...,Udiff i ,...,Udiff n ]

[0100] UdiffMed=median(Udiff)

[0101] Among them, Udiff represents the difference between the maximum and the second maximum membership value, and median() is used to calculate the median of the vector. If Udiff i >UdiffMed, then pixel i belongs to the center pixel C ij Otherwise, pixel i is an uncertain pixel. Where UdiffMed = median(Udiff).

[0102] Step 105: Optimize algorithm efficiency by membership filtering and correct the membership of misclassified pixels.

[0103] Optionally, to reduce computational complexity and improve algorithm efficiency, membership filtering is used to replace the contribution of distance calculation between pixels and cluster centers in the local spatial neighborhood;

[0104] The cluster center and membership calculation are repeated until max{U (t) -U (t+1)}<η, η is the minimum error threshold, so as to obtain a robust membership function.

[0105] Membership filtering can be used to correct misclassified pixels, that is, there is no need to calculate the distance between the pixel's neighbors and the cluster center. The modified membership is divided into:

[0106]

[0107] Among them, d ir Indicates u ij with u ir The Euclidean distance between ir for uij Neighborhood. (1 / (1+d ir )) reflects the spatial structure of membership division.

[0108] Optionally, to further improve the efficiency of the algorithm, membership filtering is applied only once to the final membership matrix, and data reconstruction is used to optimize data distribution.

[0109] Optionally, membership median filtering is used to optimize membership division and correct the membership of pixels that are misclassified in the above process. ij :

[0110] U"=med{U'}

[0111] Among them, med represents median filtering.

[0112] Step 106: The current superpixel is merged using a strategy to optimize pixel membership and enhance connectivity of discontinuous superpixels to achieve over-segmentation of the image.

[0113] Optionally, the small-sized area is merged into the adjacent connected area. If the small-sized area is adjacent to multiple connected areas, the area with the closest color is selected for merging. The small-sized area is an area with a size smaller than a preset limit.

[0114] Optionally, to prevent the appearance of oversized segmented areas, all connected areas are first renumbered in ascending order of size (number of pixels), starting with the smallest, and then the connected areas with sizes smaller than a certain limit v are renumbered. t The region is merged into its adjacent region. t =v x ·v y / 4,v x and v y Represent the length and width of the region respectively.

[0115] It should be noted that before regional fusion, each superpixel strictly appears in its own distribution area. When searching for connected areas, each pixel is assigned a new label according to connectivity. Therefore, after fusion, there is no necessary connection between the superpixel number and its distribution area, but the superpixels are still evenly distributed on the image. Finally, by reallocating non-overlapping pixels to adjacent superpixels, the membership optimization and superpixel connectivity steps are completed.

[0116] Based on the content of the above embodiment, as an optional embodiment, Figure 2 This is a schematic diagram of the comparison between superpixels and pixels provided by the present invention.

[0117] Figure 3 is a schematic diagram of the structure of the image over-segmentation device provided by the present invention, such as Figure 3As shown, the device includes: an image preprocessing module 301, a cluster center initialization module 302, a search area determination module 303, a superpixel generation module 304, a membership correction module 305 and a connectivity enhancement module 306.

[0118] Image preprocessing module 301: preprocessing the input image;

[0119] Cluster center initialization module 302: divide the preprocessed image into regular initial grids, and obtain initial cluster centers by sampling on the initial network;

[0120] Search area determination module 303: identifying the search area of ​​the initial cluster center and searching for similar pixel blocks to determine the initial superpixel;

[0121] Superpixel generation module 304: Calculate and update the initial superpixel membership and cluster center according to the initial cluster center and the search area, and generate high-quality mixed superpixels by determining superpixels, uncertain pixels and noise / outliers;

[0122] Membership correction module 305: optimizes algorithm efficiency by membership filtering and corrects the membership of wrongly classified pixels;

[0123] Connectivity enhancement module 306: adopts a merging strategy for the current superpixel, optimizes the pixel membership, and enhances the connectivity of discontinuous superpixels to achieve over-segmentation of the image.

[0124] It should be noted that the image over-segmentation device provided in the embodiment of the present invention can execute the image over-segmentation method described in any of the above embodiments during specific operation, which will not be described in detail in this embodiment.

[0125] In order to verify the technical advantages of the image over-segmentation method and device provided by the present invention, further analysis and description are given below in conjunction with an experimental simulation embodiment.

[0126] 1. Example conditions

[0127] This embodiment is run in the Ubuntu 18.04 operating system, using Matlab / Python language, PC Intel i7-8750H CPU (2.20GHz) and NVIDIA GeForce RTX 2060 GPU for experiments.

[0128] 2. Example Content

[0129] Two data sets are used for experiments in the embodiment, namely, data set-1 and data set-2. Data set-1 is a noisy image obtained by adding uniform white Gaussian noise of different intensities to a noise-free image, and the noise intensities are 20, 35, 50, 65, 80 and 95 respectively. The noise intensity of 20 means that the standard deviation of the Gaussian noise is 20. Data set-2 is an image obtained by randomly adding salt and pepper / Poisson / Gaussian / multiplicative noise and their mixed noises to a noise-free image, and the noise type and intensity are random.

[0130] Figure 4 This is a schematic diagram of the image over-segmentation result provided by the present invention. The data set used by the present invention is relatively complex, with different viewing angles, scales, postures, different degrees of noise, deformation and low resolution, and unclear and discontinuous geometric features. It can be seen from the figure that the over-segmentation result of the method of the present invention has a good response.

[0131] Figure 5 is a schematic diagram of the comparison results of segmentation accuracy of different image over-segmentation algorithms provided by the present invention, Figure 6 : It is a schematic diagram of the running time comparison results of different image over-segmentation algorithms provided by the present invention. Among them, the control methods used are NC, SL, TP, SLIC and GGM. The larger the segmentation accuracy value, the better the over-segmentation effect of the method, and the shorter the running time, the faster the running speed of the method. It can be seen from the figure that the segmentation accuracy value and running time of the present invention are better than those of other methods. It shows that the present method has better over-segmentation performance.

[0132] Figure 7 is a schematic diagram of the structure of the terminal provided by the present invention, such as Figure 7 As shown, the terminal may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the image over-segmentation method, which includes: pre-processing the input image; dividing the pre-processed image into regular initial grids, sampling on the initial network to obtain the initial cluster center; identifying the search area of ​​the initial cluster center and searching for similar pixel blocks to determine the initial super pixel; calculating and updating the membership and cluster center of the initial super pixel according to the initial cluster center and the search area, and generating high-quality mixed super pixels by determining super pixels, uncertain pixels and noise / outliers; optimizing the algorithm efficiency by membership filtering, and correcting the membership of the wrongly classified pixels; using a merging strategy for the current super pixel, optimizing the pixel membership, and enhancing the connectivity of the discontinuous super pixels to achieve image over-segmentation.

[0133] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the image over-segmentation method provided by the above-mentioned embodiments, and the method includes: preprocessing the input image; dividing the preprocessed image into a regular initial grid, sampling on the initial network to obtain the initial cluster center; identifying the search area of ​​the initial cluster center and searching for similar pixel blocks to determine the initial superpixel; according to the initial cluster center and the search area, calculating and updating the membership and cluster center of the initial superpixel, and generating high-quality mixed superpixels by determining superpixels, uncertain pixels and noise / outliers; optimizing the algorithm efficiency by means of membership filtering, and correcting the membership of misclassified pixels; adopting a merging strategy for the current superpixel, optimizing the pixel membership, and enhancing the connectivity of discontinuous superpixels to achieve image over-segmentation.

[0134] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the image over-segmentation method provided by the above-mentioned embodiments, and the method includes: preprocessing the input image; dividing the preprocessed image into a regular initial grid, sampling on the initial network to obtain the initial cluster center; identifying the search area of ​​the initial cluster center and searching for similar pixel blocks to determine the initial superpixel; according to the initial cluster center and the search area, calculating and updating the membership and cluster center of the initial superpixel, and generating high-quality mixed superpixels by determining superpixels, uncertain pixels and noise / outliers; optimizing the algorithm efficiency by means of membership filtering, and correcting the membership of misclassified pixels; adopting a merging strategy for the current superpixel, optimizing the pixel membership, and enhancing the connectivity of discontinuous superpixels to achieve image over-segmentation.

[0135] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the processor executes the image over-segmentation method provided by the above-mentioned embodiments, the method comprising: preprocessing the input image; dividing the preprocessed image into a regular initial grid, sampling on the initial network to obtain the initial cluster center; identifying the search area of ​​the initial cluster center and searching for similar pixel blocks to determine the initial superpixel; calculating and updating the membership and cluster center of the initial superpixel based on the initial cluster center and the search area, and generating high-quality mixed superpixels by determining superpixels, uncertain pixels and noise / outliers; optimizing the algorithm efficiency by means of membership filtering, and correcting the membership of misclassified pixels; adopting a merging strategy for the current superpixel, optimizing the pixel membership, and enhancing the connectivity of discontinuous superpixels to achieve image over-segmentation.

[0136] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image over-segmentation method, characterized in that: include: Preprocess the input image; Divide the preprocessed image into regular initial grids and sample on the initial network to obtain the initial cluster centers; Identify the search area of ​​the initial cluster center and search for similar pixel blocks to determine the initial superpixel; According to the initial cluster center and search area, the initial superpixel membership and cluster center are calculated and updated, and high-quality mixed superpixels are generated by determining superpixels, uncertain pixels and noise / outliers; Optimize algorithm efficiency through membership filtering and correct the membership of misclassified pixels; The current superpixel is merged, the pixel membership is optimized, and the connectivity of discontinuous superpixels is enhanced to achieve image over-segmentation.

2. The image over-segmentation method according to claim 1, characterized in that: The operation of preprocessing the input image includes one or a combination of the following operations: Format conversion operations, resampling operations, grayscale operations, geometric transformation operations, image enhancement operations, and standardization operations.

3. The image over-segmentation method according to claim 1, characterized in that: The initial grid size is a spatial width parameter used to determine the initial size of superpixels and the number of superpixels.

4. The image over-segmentation method according to claim 3, characterized in that: Identify the search area of ​​the initial cluster center and search for similar pixel blocks to determine the initial superpixel, including: Move the initial cluster center to the lowest gradient position in the local space adjacent to the search area; For each cluster center, calculate the distance to all points in the surrounding search area; The image pixels are assigned to the nearest cluster center in the search area, and similar blocks are searched to form the initial superpixel.

5. The image over-segmentation method according to claim 4, characterized in that: According to the initial cluster center and search area, the initial superpixel membership and cluster center are calculated and updated, and high-quality mixed superpixels are generated by determining superpixels, uncertain pixels and noise / outliers, including: According to the initial cluster center and the given search area, the Lagrange multiplier method is used to calculate the pixel membership and cluster center; Among them, clustering is achieved by using data reconstruction, introducing sparse constraints into the deviations of the measured data and its reconstructed data, and constructing a sparse constraint-fuzzy C-means clustering model to achieve noise / outlier suppression and feature preservation in the clustering process; Based on the sparse constraint-fuzzy C-means clustering model, neighborhood information constraints are introduced to build a local information-sparse constraint-fuzzy C-means clustering model to improve the noise / outlier heterogeneity performance and clustering robustness of the algorithm and generate high-quality mixed superpixels. Solve the deviation matrix of the measured data and its reconstructed data according to the soft threshold operator; Adopt iterative optimization strategy to iteratively update membership matrix and cluster center vector to obtain optimal conditions until convergence; Based on the pixel membership, it is determined whether the pixel belongs to a specific superpixel. If the difference between the maximum and the second maximum membership is greater than the median membership, the pixel belongs to the superpixel to which the cluster center belongs, otherwise it is an uncertain pixel.

6. The image over-segmentation method according to claim 5, characterized in that: The algorithm efficiency is optimized by membership filtering, and the membership of misclassified pixels is corrected, including: Membership filtering is used to replace the contribution of distance calculation between pixels and cluster centers in the local spatial neighborhood; Membership filtering is used to optimize membership division and correct pixels that are misclassified.

7. The image over-segmentation method according to claim 6, characterized in that: The current superpixel is merged, the pixel membership is optimized, and the connectivity of discontinuous superpixels is enhanced to achieve image over-segmentation, including: First, all connected regions are renumbered in ascending order of size, starting from the smallest, and the small-sized regions are merged into their adjacent regions in sequence; the small-sized regions are regions whose size is smaller than the preset limit; The small-sized area is merged into the adjacent connected area. If the small-sized area is adjacent to multiple connected areas, the one with the closest color is selected for merging.

8. An image over-segmentation device, characterized in that: include: An image preprocessing module, used for preprocessing an input image; The cluster center initialization module is used to divide the preprocessed image into regular initial grids and sample the initial cluster centers on the initial network; A search region determination module is used to identify the search region of the initial cluster center and search for similar pixel blocks to determine the initial superpixel; The superpixel generation module is used to calculate and update the initial superpixel membership and cluster center according to the initial cluster center and the search area, and generate high-quality mixed superpixels by determining superpixels, uncertain pixels and noise / outliers; The membership correction module is used to optimize the algorithm efficiency by means of membership filtering and correct the membership of the wrongly classified pixels; The connectivity enhancement module is used to adopt a merging strategy for the current superpixel, optimize the pixel membership, and enhance the connectivity of discontinuous superpixels to achieve over-segmentation of the image.

9. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the image over-segmentation method according to any one of claims 1 to 7 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image over-segmentation method according to any one of claims 1 to 7 are implemented.

Images