Unsupervised image superpixel generation method, device and readable storage medium

The unsupervised superpixel segmentation of unsupervised images is optimized through the CRE feature extraction module and the adaptive loss function, which solves the problem of edge information processing and fixed number of iterations. The generated superpixels are more suitable for the image edges, improving segmentation accuracy and stability.

CN118115365BActive Publication Date: 2024-11-22SHANDONG UNIV OF FINANCE & ECONOMICS
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410377906.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-29
Publication Date
2024-11-22
Estimated Expiration
2044-03-29

AI Technical Summary

Technical Problem

The existing unsupervised image superpixel segmentation methods have shortcomings in edge information processing and reconstruction image constraints, resulting in the generated superpixel edge fitting, the reconstruction information is not accurate enough, and the fixed number of iterations for single-task learning may be Causes the model to be underfitted or overfitted.

Method used

The CRE feature extraction module is adopted, combined with the convolution module and the ECA channel attention module, and the image edge gradient is processed through the side window filter to define the target loss function, including clustering loss, spatial smoothness loss, reconstruction loss and edge perception loss, and An adaptive loss threshold discriminator is introduced to optimize the iteration process.

Benefits of technology

The stability and robustness of image feature extraction are improved, and the generated superpixels are more suitable for the image edges, reducing undersegmentation errors, improving segmentation accuracy and boundary recall, and reducing the risks of overfitting and underfitting.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118115365B_ABST
    Figure CN118115365B_ABST
Patent Text Reader

Abstract

The present invention proposes an unsupervised image superpixel generation method, device and readable storage medium. The method first preprocesses the segmented image, integrates the coordinate information of the image pixels into the pixel features, and obtains the image space features. Next, the preprocessed image features enter the CRE feature extraction module designed by the present invention to extract the image features, and finally output the superpixel confidence estimate through the classification layer, and use argmax to obtain the superpixel segmentation result. Based on the RIM algorithm, the present invention can generate higher quality superpixels and better fit the image edges.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically to an unsupervised image superpixel generation method, device and readable storage medium. Background Art

[0002] Superpixel segmentation is an important technology in the field of digital image processing. It obtains superpixels by clustering pixels into connected regions according to feature similarity. Compared with pixel-based image representation methods, superpixels can reduce the number of elements required to represent the image while retaining image details and structural information, thereby reducing the computational complexity of downstream image processing tasks.

[0003] Existing superpixel segmentation methods can be mainly divided into two categories: traditional methods and deep learning methods. Traditional methods use manually selected features, such as color, gradient, position, etc., and use clustering or graph cut methods to obtain superpixel segmentation. Classic algorithms include Ncut and SLIC (Simple Linear Iterative Clustering Algorithm). The performance of this type of algorithm is closely related to the manually selected features.

[0004] Benefiting from the powerful feature extraction capabilities of neural networks, deep learning methods are also widely used in image superpixel generation. Neural networks have a strong advantage in extracting image features and can better capture high-level features in images to obtain more accurate superpixel segmentation results. Superpixel segmentation methods based on deep learning are divided into supervised methods and unsupervised methods. Supervised methods use labeled training data to guide the model to learn superpixel segmentation in images. These methods can usually produce accurate segmentation results, but require a large amount of labeled data and computing resources to train the model. Due to the high cost of obtaining manually labeled data, unsupervised superpixel segmentation methods have gradually become one of the focuses of attention.

[0005] Unsupervised superpixel segmentation methods do not require labeled data to supervise network training, but rely on the statistical information and features of the image itself to perform superpixel segmentation. A typical method is the RIM method based on deep image prior. The RIM method uses the deep image prior (DIP) process, inputs the image into a simple 4-layer CNN network to extract image features, and then outputs superpixel confidence estimates through the classification layer. In the iterative process, the target loss function is used to constrain the generation of superpixels, and the network model directly outputs superpixel labels. However, due to the lack of labeled data, the accuracy of the unsupervised model is still lower than that of the supervised method. Specifically, the RIM method has the following obvious shortcomings in superpixel segmentation: (1) The edge information of the image is not considered: This method uses a 4-layer convolutional neural network to obtain image features, but does not focus on the edge information of the image, resulting in insufficient attention to detailed edge information and low edge fit of the generated superpixels. (2) Insufficient constraints on reconstructed image information: The RIM algorithm uses RGB reconstructed images in the reconstruction loss, ignores high-frequency information features in the reconstructed image, loses a lot of detail information, and cannot obtain accurate superpixel segmentation results. (3) The RIM method uses an early stopping strategy. As an unsupervised single-task learning method, it uses the same number of iterations when generating superpixels for each image. However, due to the diversity of images, the convergence of each image during superpixel segmentation is different. Fixed convergence times may lead to underfitting or overfitting of the model. Summary of the invention

[0006] In view of the above problems, the purpose of the present invention is to provide an unsupervised image superpixel generation method, device and readable storage medium, which can generate higher quality superpixels based on the RIM algorithm and better fit the image edges.

[0007] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0008] In a first aspect, the present invention discloses an unsupervised method for generating superpixels of an image, comprising the following steps:

[0009] S1: read the natural image to be processed, preprocess the image, add image coordinate information, normalize the image features, and obtain the initial feature vector of the image;

[0010] S2: Processing the natural image to be processed by a side window filter, and calculating the image edge gradient of the processed natural image;

[0011] S3: Construct a CRE feature extraction module, randomly initialize configuration parameters, input the initial feature vector of the image into the CRE feature extraction module for image feature extraction, and output superpixel confidence estimation P, RGB reconstructed image and average superpixel map;

[0012] S4: define a target loss function, and calculate the target loss L using the target loss function based on the image gradient of the processed natural image and the superpixel confidence estimate P, the RGB reconstructed image and the average superpixel map objective , and record the current number of iterations ν;

[0013] S5: Determine whether the current number of iterations ν is greater than a preset lower limit of the number of iterations iteration_m; if so, execute the next step; if not, execute step S3;

[0014] S6: Use the preset discriminator to lose L according to the current target objective Calculate the mean target loss within the range of t;

[0015] S7: Determine whether the target loss mean within the range t meets the iteration stop condition; if not, execute step S3; if yes, execute the next step;

[0016] S8: Based on the current superpixel confidence estimate P, complete the superpixel segmentation of the natural image to be processed.

[0017] Further, step S1 comprises:

[0018] Read a natural image I of size H×W;

[0019] The spatial coordinate information of the natural image I is added to the feature vector of the image, which is defined as an extensibility mapping I→R h×w×5 ;

[0020] The natural image I is embedded into a 5-dimensional feature vector and normalized to obtain the initial feature vector V = (h, w, r, g, b).

[0021] Further, step S2 includes performing the following side window filtering operation on the natural image I:

[0022] In the discrete case, side windows in 8 directions are used, and the side windows in 8 directions correspond to the following formula:

[0023]

[0024] Where θ is the angle between the window and the horizontal line;

[0025] Set ρ∈{0,r} and (x,y) to be the location of the target pixel i and r to be the radius of the window. Change the orientation of the window by varying θ and fixing (x,y) while aligning its edge with i;

[0026] By setting ρ = r, we can get the side windows in four directions: down (D), right (R), up (U), and left (L), which can be expressed as and

[0027] By setting ρ = 0, we get the windows in the southwest (SW), southeast (SE), northeast (NE), and northwest (NW) directions, which are expressed as and

[0028] By applying the filter kernel function F in each side window, 8 outputs are obtained, which are expressed as Where i is the target pixel number and q is the pixel value;

[0029] The result I of the target pixel after processing in different side windows n is calculated by the following formula: n :

[0030]

[0031] Among them, pixel j is the adjacent pixel of pixel i, ω ij is the corresponding weight given for pixel j based on the side window kernel function F, S = {L, R, U, D, NW, NE, SW, SE} is the set of side window indices;

[0032] According to the formula Select the side window with the smallest distance to the original pixel value q, and use its output as the filtered image I s , and finally calculate the processed image gradient

[0033] Further, the CRE feature extraction module includes a convolution module and an ECA channel attention module;

[0034] The function operation of the convolution module is described as:

[0035]

[0036] Among them, l represents the index of the layer, InNorm represents the instantiation normalization operation, and its input is u l , W and b are the weight and bias of the convolutional layer respectively;

[0037] In the last layer of the convolution module, the Relu activation function is used as the final output X of the convolution module.

[0038] X = ReLu(conv l ).

[0039] Further, step S3 includes:

[0040] The initial feature vector of the image is input into the convolution module to generate the final output feature vector X of the convolution module; the final output feature vector X of the convolution module is subjected to global average pooling operation by channel, and the result is recorded as

[0041]

[0042] Among them, H, W and C are the height, width and number of channels of the input feature map respectively;

[0043] Will Through the 1D convolution layer, the result is recorded as Z:

[0044]

[0045] Pass Z through the Sigmoid activation function to get the weight result Y:

[0046] Y = sigmoid(Z);

[0047] Multiply the input feature maps X and Y element by element to get the output feature vector of the ECA channel attention module:

[0048] P′=X×Y;

[0049] Based on the feature vector P′, the final superpixel confidence estimate P is output through a 1×1 convolution classification layer.

[0050] Further, step S4 includes:

[0051] The target loss L is calculated by the following target loss function objective :

[0052] L objective =L clustering +αL smoothness +βL recons +ηL edge ;

[0053] Among them, L clustering , L smoothness , L recons and L edge They represent clustering loss, spatial smoothness loss, reconstruction loss and edge-aware loss respectively; α, β and η are balance parameters;

[0054] The clustering loss calculation formula is as follows:

[0055]

[0056] Where P is the superpixel confidence estimate, represents the average of the superpixel confidence estimates over all pixels:

[0057]

[0058] L clustering The first item inside is P h,w ∈R N The second term is the negative entropy of the average vector over all probability vectors;

[0059] The calculation formula of spatial smoothness loss is as follows:

[0060]

[0061] in, and ||·|| p represents the image gradient and P norm, P h,w and I h,w is the vector of P and I at (h,w); the reconstruction loss calculation formula is as follows:

[0062]

[0063] in, is the RGB reconstructed image, is the average superpixel map, and the calculation formula is:

[0064]

[0065] The edge perception loss calculation formula is as follows:

[0066]

[0067] in, and Represents the gradient of the confidence estimate of the image in the x and y directions, Indicates whether the image pixel is on the image boundary in a certain direction. It is normalized And d∈{x,y}, I s =SWF(I).

[0068] Further, the preset discriminator includes:

[0069]

[0070]

[0071] Where t represents the step size of the discriminator, and r1 and r2 are the means of the loss function within the range of t.

[0072] Further, step S7 includes:

[0073] Determine whether |r1-r2|<θ r ;

[0074] If not, execute step S3; if yes, execute the next step.

[0075] In a second aspect, the present invention further discloses an unsupervised image superpixel generation device, comprising:

[0076] A memory, used for storing an unsupervised image superpixel generation program;

[0077] A processor is used to implement the steps of the unsupervised image superpixel generation method as described in any of the above items when executing the unsupervised image superpixel generation program.

[0078] In a third aspect, the present invention further discloses a readable storage medium, on which an unsupervised image superpixel generation program is stored. When the unsupervised image superpixel generation program is executed by a processor, the steps of the unsupervised image superpixel generation method as described in any one of the above items are implemented.

[0079] Compared with the prior art, the present invention has the following beneficial effects:

[0080] 1. The present invention realizes efficient extraction of image features through the CRE feature extraction module. The module contains 4 layers of CovInReLu (reflection filling, convolution, instantiation normalization, activation function), combined with the instantiation normalization operation, which effectively improves the stability and robustness of the features. In addition, the present invention also introduces the ECA (lightweight channel attention) module, which further optimizes the expressiveness of the features by modeling the correlation between channels. This combination enables the CRE feature extraction module to perform well in image feature extraction tasks and accurately capture the texture, shape and edge information in the image.

[0081] 2. The present invention adopts a clustering method based on maximization of regularized information, which can produce significant results in image processing. This method takes the clustering results as the basis, and combines the constraints of edge perception and average superpixel reconstruction, so as to generate superpixels that are as regular as possible in the flat area of ​​the image, and generate superpixels that are closer to the edge in the area with high image gradient. This method not only shows a high level in terms of achievable segmentation accuracy (ASA), but also maintains a high level of boundary recall rate (BR), and greatly reduces the under-segmentation error (USE).

[0082] 3. The present invention introduces an adaptive loss threshold discriminator operation during the training process, which ensures that the model can adaptively reach the optimal convergence state when processing different images. This adaptive stopping strategy plays a key role in the training process and helps reduce the risk of overfitting and underfitting.

[0083] It can be seen that compared with the prior art, the present invention has outstanding substantive features and significant progress, and the beneficial effects of its implementation are also obvious. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0085] Figure 1 It is a flow chart of the method of the present invention.

[0086] Figure 2 Schematic diagram of edge pixels of the present invention.

[0087] Figure 3 It is the side window filter window direction diagram of the present invention.

[0088] Figure 4 It is a network model design diagram of the CRE feature extraction module of the present invention.

[0089] Figure 5 It is a schematic diagram of the structure of the ECA channel attention module of the present invention.

[0090] Figure 6 This is a superpixel segmentation effect diagram of the present invention. DETAILED DESCRIPTION

[0091] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0092] See also Figure 1 As shown, this embodiment provides an unsupervised method for generating image superpixels, comprising the following steps:

[0093] S1: Read the natural image to be processed, preprocess the image, add image coordinate information, and normalize the image features to obtain the initial feature vector of the image.

[0094] In a specific implementation, a natural image I of size H×W is first read; then, the spatial coordinate information of the natural image I is added to the feature vector of the image, which is defined as an extensible mapping I→R h×w×5; Embed the natural image I into a 5-dimensional feature vector and normalize it to obtain the initial feature vector V = (h, w, r, g, b).

[0095] It should be noted that the purpose of this method is to generate image superpixels through superpixel segmentation. The purpose of superpixel segmentation is to assign pixels to superpixels S = {s1,....,s N In a superpixel in the set, each superpixel represents a set of pixels, and the superparameter N is the upper bound of the number of superpixels. Therefore, the superpixel segmentation task can be classified as an N-classification problem. In order to solve this type of problem, the present invention adopts a method based on regularized information maximization, so that the convolutional neural network directly outputs the confidence estimate of the superpixel. The confidence estimate is expressed as:

[0096]

[0097] Next, the pixel is assigned to the superpixel with the maximum confidence estimate.

[0098] S2: Processing the natural image to be processed through the side window filter, and calculating the image edge gradient of the processed natural image.

[0099] This step involves the operation of side window filtering SWF (Side Window Filtering). When initializing the feature vector, the natural image I is also subjected to a side window filtering operation in order to calculate a more accurate edge gradient. This operation takes into account the spatial domain information and the pixel value domain information in order to better constrain the generated superpixel edge to fit the edge of the original image. When calculating the edge-aware loss function, we hope that the image edge gradient contrast is higher and the non-edge area contrast is lower, that is, retaining the edge and removing the noise. Retaining the edge means that we hope to minimize the distance between the input and output at the edge, that is, the filter output should be the same as or as close as possible to the input at the edge. Traditional filtering algorithms usually use full-window regression, that is, the center of the window is placed at the position of the pixel to be processed. However, even with nonlinear anisotropic weighting, normal diffusion along the edge of the image cannot be avoided. This is the fundamental reason why traditional methods do not preserve edges. The side window filter places the edge of the window directly at the position of the pixel to be processed, thereby cutting off possible normal diffusion. As Figure 2 As shown in the figure, the two adjacent pixels a and b, although they are physically adjacent, the half windows they choose are facing two different directions, so there will be no edge blur. Unlike other edge-preserving filtering methods (such as bilateral filtering), the side window filtering method cuts off the possibility of normal diffusion from the root.

[0100] In a specific implementation, since the side window can be in any direction on the edge pixel point, considering the computational complexity and performance, this step uses side windows in 8 directions in the discrete case, such as Figure 3 These 8 windows correspond to:

[0101]

[0102] where θ is the angle between the window and the horizon, we set ρ∈{0,r} and (x,y) is the location of the target pixel i, and r is the radius of the window. By varying θ and fixing (x,y) we can change the orientation of the window while aligning its edge with i.

[0103] By setting ρ = r, we have side windows in four directions: down (D), right (R), up (U), and left (L), expressed as and By setting ρ = 0, there will be windows in the southwest (SW), southeast (SE), northeast (NE), and northwest (NW) directions, expressed as and By applying the filter kernel F in each side window, we can get 8 outputs, represented as in and ρ∈{0,r}.

[0104]

[0105] ω ij is the weight of pixel j near target pixel i based on kernel function F, q is the target pixel value, and S = {L, R, U, D, NW, NE, SW, SE} is the set of side window indices. So the result I of the target pixel after processing in different side windows n is n for:

[0106]

[0107] As mentioned earlier, retaining the edge means minimizing the distance between the input and output at the edge, that is, the filter output should be the same as or as close as possible to the input at the edge. Therefore, the side window filter selects the side window with the smallest distance from the original pixel value q, and uses its output as the filtered image I s :

[0108]

[0109] Among them, I s Is the final output of SWF, and finally calculate the processed image gradient

[0110] S3: Construct a CRE feature extraction module, randomly initialize configuration parameters, input the initial feature vector of the image into the CRE feature extraction module for image feature extraction, and output superpixel confidence estimate P, RGB reconstructed image and average superpixel map.

[0111] In a specific implementation, the preprocessed 5-dimensional feature vector V = (h, w, r, g, b) is input into the CRE feature extraction module, such as Figure 4 As shown, the same procedure as Deep Image Prior (DIP) is used in the CRE feature extraction module to optimize the randomly initialized model parameters using a single image during unlabeled inference time. The structure in the CRE feature extraction module contains a convolution module and an ECA channel attention module.

[0112] Specifically, the convolution module adopts a repetitive structure: reflection padding layer, convolution layer, normalization layer and activation layer, where A represents the depth of the network layer. The network has a single path, with the input tensor on the left and the output feature tensor on the right. The number of convolution kernels doubles with each layer of depth of the network layer. A reflection padding layer is added between the upper and lower layers to prevent the loss of valid feature data due to downsampling. In addition, in order to improve the overall performance of the network, an instantiated normalization layer is added after the convolution layer of the network to instantiate and normalize the feature values ​​after convolution of the current network layer. This helps to control overfitting, increase the sensitivity of the network to weights during model training, and speed up the convergence of the network. Its function operation can be described as:

[0113] conv l =InNorm(u l )

[0114] u l =W l conv l-1 +b l

[0115] Here, l represents the index of the layer, and InNorm represents the instantiation normalization operation, whose input is u l , W and b are the weight and bias of the convolution layer respectively. Their values ​​are determined by supervised network training. In the network, W can also be called the convolution kernel, and its size is set to 3×3×n l , and after each convolution, the number of kernels in the next network layer is n l Double. Since the shape of the data feature distribution changes during the training and constraint process, the normalization layer will adjust based on the acquired features so that the output data mean is close to 0 and the standard deviation is close to 1. In the last layer of the convolution module, the Relu activation function is used as the final output of the convolution module:

[0116] X = ReLu(conv l )

[0117] like Figure 5 As shown in the figure, compared with the traditional two-dimensional convolution operation, the ECA channel attention module uses a one-dimensional convolution operation to extract the relationship between channels, which not only makes the number of parameters smaller, but also reduces the computational complexity. Therefore, while improving the performance of the model, the ECA channel attention module does not significantly increase the number of parameters and computational burden of the model. Through the one-dimensional convolution operation, the ECA channel attention module can effectively extract the correlation and dependency between channels in the feature map. When calculating the channel attention weight, a local convolution operation is used, that is, only the relationship between channels in the local area of ​​the feature map is considered. This localized operation helps to reduce the confusion of local information, allowing the module to pay more attention to important features in the image. At the same time, the ECA channel attention module can globally affect the entire feature map, ensuring the module's perception of the global information of the image.

[0118] As an example, first input the initial feature vector of the image into the convolution module to generate the final output feature vector X of the convolution module, and then perform a global average pooling operation on the final output feature vector X of the convolution module by channel, and the result is recorded as X:

[0119]

[0120] Among them, H, W and C are the height, width and number of channels of the input feature map respectively. Then, Through the 1D convolution layer, the result is recorded as Z:

[0121]

[0122] Pass Z through the Sigmoid activation function to get the weight result Y:

[0123] Y = sigmoid(Z)

[0124] At this point, the input feature maps X and Y are multiplied element by element to obtain the output feature vector of the ECA module:

[0125] P′=X×Y

[0126] Finally, the feature vector P′ is passed through a 1×1 convolutional classification layer to output the final superpixel confidence estimate P.

[0127] S4: define a target loss function, and calculate a target loss L using the target loss function based on the image gradient of the natural image to be processed and the superpixel confidence estimate P objective , and record the current number of iterations ν.

[0128] In a specific implementation, the target loss function is specifically defined as follows:

[0129] L objective =L clustering +αL smoothness +βL recons +ηL edge

[0130] Among them, L clustering , L smoothness , L recons and L edge denote clustering loss, spatial smoothness loss, reconstruction loss, and edge-aware loss, respectively. α, β, and η are parameters that balance the importance of each term. By minimizing L objective Get superpixel S.

[0131] 1) L clustering is the entropy-based clustering loss, similar to the mutual information term of regularized information maximization, L clustering The losses are as follows:

[0132]

[0133] Where P is the superpixel confidence estimate, represents the average of the superpixel confidence estimates over all pixels:

[0134]

[0135] L clustering The first item inside is P h,w ∈R N The entropy of , minimizing this term will encourage deterministic superpixel assignment, and the second term is the negative entropy of the average vector over all probability vectors, its minimization encourages the size of each superpixel block to be uniform.

[0136] 2) Spatial smoothness loss L smoothness is the main prior for image processing tasks, which quantifies the difference between adjacent pixels. smoothness Defined as:

[0137]

[0138] in, and ||·|| p represents the image gradient and P norm, P h,w and I h,w is the vector of P and I at (h,w).

[0139] 3) Superpixel segmentation is to group pixels with similar features together. If the pixels belonging to the same superpixel are represented by the features of the superpixel to which they belong, the mean square error between the obtained superpixel image and the original image should be minimized. Therefore, the reconstruction loss is redefined as:

[0140]

[0141] in, is the RGB reconstructed image, is the average superpixel map, and the calculation formula is:

[0142]

[0143] 4) In order to generate superpixels with high fit to the image edge, the present invention introduces edge-aware loss. The edge-aware loss is based on the image edge information and constrains the model to generate superpixels that fit the edge better by differentially fitting the image gradient distribution through superpixel confidence estimation. The edge-aware loss function is defined as:

[0144]

[0145] in, and Represents the gradient of the confidence estimate of the image in the x and y directions. Considering the relationship between the image boundary and the image gradient, we introduce Indicates whether the image pixel is on the image boundary in a certain direction. It is normalized And d∈{x,y}, I s =SWF(I). The value of reflects the degree of existence of the edge. When the value is large, it means that the image boundary is more obvious; if If the value is close to 0, it means that there is no obvious edge. Improving the fit between the confidence estimate difference of the superpixel and the gradient distribution of the image can ensure that pixels inside and outside the image boundary belong to different superpixel sets.

[0146] To express the degree of fit, the indicators are defined as follows:

[0147]

[0148] Among them, θ is used as a threshold to determine whether the gradient value is fitted.

[0149] S5: Determine whether the current number of iterations ν is greater than the preset lower limit of the number of iterations iteration_m; if so, execute the next step; if not, execute step S3.

[0150] S6: Use the preset discriminator to lose L according to the current target objective Compute the mean target loss over a range of t.

[0151] In this step, the preset discriminator includes:

[0152]

[0153]

[0154] Where t represents the step size of the discriminator, and r1 and r2 are the means of the loss function within the range of t.

[0155] S7: Determine whether the target loss mean within the range t meets the iteration stop condition; if not, execute step S3; if yes, execute the next step.

[0156] Specifically, determine whether |r1-r2|·<θ r ; If not, proceed to step S3; If yes, proceed to the next step.

[0157] From the above steps S5-S7, it can be seen that the present invention adds an adaptive loss threshold discriminator in the iterative process, which avoids the fixed number of iterations compared with the previous unsupervised single-task learning method. The present invention sets a lower limit on the number of iterations as the starting condition of the loss threshold discriminator:

[0158] ifν>iteration_m

[0159] Where ν represents the current number of iterations, and iteration_m represents the lower limit of the number of iterations. Then, the discriminator can be expressed as:

[0160]

[0161]

[0162] Where t represents the step size of the discriminator, and r1 and r2 are the means of the loss function within a specific range of t.

[0163] Based on this, the stopping condition of the discriminator is:

[0164] |r1-r2|<θ r

[0165] If the above conditions can be met, it means that the network model has reached convergence and the network model stops iterating.

[0166] S8: Based on the current superpixel confidence estimate P, complete the superpixel segmentation of the natural image to be processed.

[0167] The present invention provides an unsupervised image superpixel generation method, which first preprocesses the segmented image, integrates the coordinate information of the image pixels into the pixel features, and obtains the image spatial features. Next, the preprocessed image features enter the CRE feature extraction module designed by the present invention to extract image features, and finally outputs the superpixel confidence estimate through the classification layer, and uses argmax to obtain the superpixel segmentation result.

[0168] In order to make the superpixel segmentation results more accurate, this method designs a loss function that integrates clustering constraints, spatial smoothness constraints, edge-aware constraints, and reconstruction constraints. The clustering loss is a term based on the cross entropy of the superpixel confidence estimate and its average value over all pixels. This loss function aims to encourage pixels to be correctly classified and make the number of pixels in each superpixel set as equal as possible. The spatial smoothness loss takes into account the impact of high-frequency noise in the image on pixel classification. By applying an L1 penalty to the gradient of the superpixel confidence estimate, the smoothness of the superpixel map is maintained locally. The edge-aware loss calculates the image gradient processed by the side window filter and fits the image gradient with the gradient difference of the superpixel confidence estimate to obtain edge-fitting superpixels. In order to minimize the error between the superpixel map and the original image, the reconstruction loss calculates the mean square error between the RGB reconstruction map and the average superpixel map and the original image, respectively, so that the model can learn the details and important features in the data. The target loss function of this method aims to encourage the model to learn the edge information of the image and the detailed features of the pixels to improve the accuracy of superpixel segmentation.

[0169] In terms of stopping strategy, this method adopts an adaptive stopping method based on loss threshold discriminator. On the one hand, the minimum number of iterations is set to prevent insufficient model optimization; on the other hand, the loss function threshold discriminator is set. When the number of iterations is greater than the minimum number of iterations, the rate of change of the loss function is used to determine whether to stop the iteration to avoid redundant calculations and prevent overfitting.

[0170] In actual use, this method can generate Figure 6 The superpixel segmentation effect is shown in the figure. This method is verified with other unsupervised superpixel segmentation methods using the same number of experiments. The experiment is conducted on the test set in the BSDS500 dataset, which includes a pixel resolution of 481 × 321 and 321 × 200 images of 481. As shown in Table 1, the proposed method achieves good performance in superpixel segmentation.

[0171] Table 1 Performance comparison of three methods with 50 superpixels

[0172]

[0173] In Table 1, three common metrics used in superpixel image segmentation are used to measure the estimation:

[0174] (1) Achievable segmentation accuracy (ASA): ASA is used to measure the highest accuracy that can be achieved in accurately identifying objects contained in an image. It is usually measured by the maximum degree of overlap between superpixel blocks and the image ground truth. The larger the value, the better the model performance.

[0175] (2) Boundary recall (BR): BR indicates the degree of fit between the superpixel boundary and the image target boundary. It is measured by calculating the proportion of the ground truth image target boundary that falls within the pixel width range specified by the superpixel segmentation boundary. The higher the value, the more image boundaries are preserved by the superpixel segmentation, and the better the model performance.

[0176] (2) Undersegmentation Error (USE) refers to the error of failing to correctly divide the object boundaries in the image during the superpixel segmentation process. The lower the USE value, the more accurately the algorithm divides the object boundaries in the image during the superpixel segmentation process, that is, the boundaries of the superpixel blocks are closer to the real object boundaries, and the better the algorithm performance.

[0177] The present invention also discloses an unsupervised image superpixel generation device, comprising a processor and a memory; wherein, when the processor executes the unsupervised image superpixel generation program stored in the memory, the steps of the unsupervised image superpixel generation method as described in any one of the above items are implemented.

[0178] Furthermore, the unsupervised image superpixel generation device in this embodiment may also include:

[0179] The input interface is used to obtain an unsupervised image superpixel generation program imported from the outside, and save the obtained unsupervised image superpixel generation program to the memory, and can also be used to obtain various instructions and parameters transmitted by an external terminal device, and transmit them to the processor, so that the processor can use the above various instructions and parameters to carry out corresponding processing. In this embodiment, the input interface can specifically include but is not limited to a USB interface, a serial interface, a voice input interface, a fingerprint input interface, a hard disk reading interface, etc.

[0180] The output interface is used to output various data generated by the processor to the terminal device connected thereto, so that other terminal devices connected to the output interface can obtain various data generated by the processor. In this embodiment, the output interface may specifically include but is not limited to a USB interface, a serial interface, etc.

[0181] A communication unit is used to establish a remote communication connection between the unsupervised image superpixel generation device and the external server so that the unsupervised image superpixel generation device can mount the image file to the external server. In this embodiment, the communication unit may specifically include but is not limited to a remote communication unit based on wireless communication technology or wired communication technology.

[0182] The keyboard is used to obtain various parameter data or instructions input by the user by tapping the keycaps in real time.

[0183] A display for displaying real-time information about the unsupervised image superpixel generation process.

[0184] The mouse can be used to assist users in inputting data and simplify user operations.

[0185] The present invention also discloses a readable storage medium, wherein the readable storage medium includes a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable hard disk, a CD-ROM or any other form of storage medium known in the technical field. The readable storage medium stores an unsupervised image superpixel generation program, and when the unsupervised image superpixel generation program is executed by a processor, the steps of the unsupervised image superpixel generation method described in any one of the above items are implemented.

[0186] In summary, the present invention is based on the RIM algorithm and can generate superpixels with higher quality and better fit the image edge.

[0187] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. As for the method disclosed in the embodiment, since it corresponds to the system disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0188] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0189] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0190] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0191] The above is a detailed introduction to the unsupervised image superpixel generation method, device and readable storage medium provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. An unsupervised method for generating superpixels of an image, characterized in that: The steps include: S1: read the natural image to be processed, preprocess the image, add image coordinate information, normalize the image features, and obtain the initial feature vector of the image; S2: Processing the natural image to be processed by a side window filter, and calculating the image edge gradient of the processed natural image; The step S2 includes performing the following side window filtering operation on the natural image I: In the discrete case, side windows in 8 directions are used, and the side windows in 8 directions correspond to the following formula: Where θ is the angle between the window and the horizontal line; Set ρ∈{0,r} and (x,y) to be the location of the target pixel i and r to be the radius of the window; change the orientation of the window by varying θ and fixing (x,y) while aligning its edge with i; By setting ρ = r, we can get the side windows in four directions: down (D), right (R), up (U), and left (L), which can be expressed as and By setting ρ = 0, we get the windows in the southwest (SW), southeast (SE), northeast (NE), and northwest (NW) directions, which are expressed as and By applying the filter kernel function F in each side window, 8 outputs are obtained, which are expressed as Where i is the target pixel number and q is the pixel value; The result I of the target pixel after processing in different side windows n is calculated by the following formula: n : Among them, pixel j is the adjacent pixel of pixel i, ω ij is the corresponding weight given for pixel j based on the side window kernel function F, S = {L, R, U, D, NW, NE, SW, SE} is the set of side window indices; According to the formula Select the side window with the smallest distance to the original pixel value q, and use its output as the filtered image I s , and finally calculate the processed image gradient S3: Construct a CRE feature extraction module, randomly initialize configuration parameters, input the initial feature vector of the image into the CRE feature extraction module for image feature extraction, and output the superpixel confidence estimate P, RGB reconstructed image and average superpixel map through the classification layer; S4: define a target loss function, calculate the target loss using the target loss function based on the image gradient of the processed natural image and the superpixel confidence estimate P, the RGB reconstructed image and the average superpixel map And record the current number of iterations v; The step S4 comprises: The target loss is calculated by the following target loss function in, and They represent clustering loss, spatial smoothness loss, reconstruction loss and edge-aware loss respectively; α, β and η are balance parameters; The clustering loss calculation formula is as follows: Among them, P is the confidence estimate of the superpixel, represents the average of the superpixel confidence estimates over all pixels: The first item inside is P h,w ∈R N The second term is the negative entropy of the average vector over all probability vectors; The calculation formula of spatial smoothness loss is as follows: Among them, ‖.‖ and ‖·‖ p represents the image gradient and P norm, P h,w and I h,w is the vector of P and I at (h,w); the reconstruction loss calculation formula is as follows: in, is the RGB reconstructed image, is the average superpixel map, and the calculation formula is: The edge perception loss calculation formula is as follows: in, and Represents the gradient of the confidence estimate of the image in the x and y directions, Indicates whether the image pixel is on the image boundary in a certain direction. It is normalized And d∈{x,y}, I s =SWF(I); S5: Determine whether the current number of iterations v is greater than the preset lower limit of the number of iterations iteration_m; if so, execute the next step; if not, execute step S3; S6: Use the preset discriminator to lose the current target Calculate the mean target loss within the range of t; S7: Determine whether the target loss mean within the range of t meets the conditions for determining the iteration stop; if not, execute step S3; if yes, execute the next step; t represents the step size of the discriminator; S8: Based on the current superpixel confidence estimate P, complete the superpixel segmentation of the natural image to be processed.

2. The unsupervised image superpixel generation method according to claim 1, characterized in that: The step S1 comprises: Read a natural image I of size H×W; The spatial coordinate information of the natural image I is added to the feature vector of the image, which is defined as an extensibility mapping I→R h ×w×5 ; The natural image I is embedded into a 5-dimensional feature vector and normalized to obtain the initial feature vector V = (h, w, r, g, b).

3. The unsupervised image superpixel generation method according to claim 2, characterized in that: The CRE feature extraction module includes a convolution module and an ECA channel attention module; The function operation of the convolution module is described as: conv l =InNorm(u l ) u l =W l conv l-1 +b l ; Among them, l represents the index of the layer, InNorm represents the instantiation normalization operation, and its input is u l , W and b are the weight and bias of the convolutional layer respectively; In the last layer of the convolution module, the Relu activation function is used as the final output X of the convolution module, X = ReLu (conv l ).

4. The unsupervised image superpixel generation method according to claim 3, characterized in that: The step S3 comprises: The initial feature vector of the image is input into the convolution module to generate the final output feature vector X of the convolution module; the final output feature vector X of the convolution module is subjected to global average pooling operation by channel, and the result is recorded as Among them, H, W and C are the height, width and number of channels of the input feature map respectively; Will Through the 1D convolution layer, the result is recorded as Z: Pass Z through the Sigmoid activation function to get the weight result Y: Y = sigmoid(Z); Multiply the input feature maps X and Y element by element to get the output feature vector of the ECA channel attention module: P′=X×Y; Based on the feature vector P′, the final superpixel confidence estimate P is output through a 1×1 convolution classification layer.

5. The unsupervised image superpixel generation method according to claim 1, characterized in that: The preset discriminator comprises: Where t represents the step size of the discriminator, and r1 and r2 are the means of the loss function within the range of t.

6. The unsupervised image superpixel generation method according to claim 5, characterized in that: The step S7 comprises: Determine whether |r1-r2|<θ r ; If not, execute step S3; if yes, execute the next step.

7. An unsupervised image superpixel generation device, characterized in that: include: A memory, used for storing an unsupervised image superpixel generation program; A processor is used to implement the steps of the unsupervised image superpixel generation method as described in any one of claims 1 to 6 when executing the unsupervised image superpixel generation program.

8. A readable storage medium, characterized in that: The readable storage medium stores an unsupervised image superpixel generation program, which, when executed by a processor, implements the steps of the unsupervised image superpixel generation method as described in any one of claims 1 to 6.