Spectral image denoising method and device, equipment and storage medium

Abstract

The invention discloses a spectral image denoising method and device, equipment and a storage medium. According to the scheme, the method comprises the following steps: performing super-pixel segmentation processing on a first spectral image to be denoised to obtain a super-pixel tensor containing a super-pixel block; constructing a denoising model of the first spectral image according to the first spectral image and the superpixel tensor; the denoising model is used for solving the second spectral image, so that the sum of the data fidelity term, the low-rank regular term and the contour gradient regular term is minimum; and solving a second spectral image of the denoising model, and determining the finally solved second spectral image as a denoising result of the first spectral image. On one hand, low-rank tensor approximation is carried out on superpixel blocks in each superpixel tensor by setting a low-rank regular term so as to remove various noises in each superpixel block, and on the other hand, gradient amplitudes of contour pixels along the pixel direction are limited by a contour gradient regular term so as to remove noise in each superpixel block. And the phenomena of unsmoothness, artifacts and the like at the edge of the superpixel are prevented.

Description

Technical Field

[0001] The present application relates to the field of hyperspectral images, and in particular to a spectral image denoising method and apparatus, equipment and storage medium. Background Art

[0002] Hyperspectral imaging (HSI) obtains the details and composition information of an object by acquiring the spectral information of the object in different bands. The resulting hyperspectral image usually contains dozens to hundreds of bands, each of which records the intensity of light reflected or transmitted by the object at a specific wavelength. Compared with ordinary imaging technology, hyperspectral imaging can capture the reflection characteristics of the target in dozens or even hundreds of narrow bands, which usually cover the range from visible light to near-infrared, thereby providing rich spectral information and facilitating the accurate identification and analysis of the target material composition.

[0003] When performing feature extraction, segmentation, matching and target recognition of hyperspectral images, the presence of noise will greatly reduce the reliability and effectiveness of image processing technology. These noise interferences will lead to inaccurate extracted features, resulting in errors or uncertainties in target recognition results, which also brings difficulties to the refined application of hyperspectral images. Summary of the invention

[0004] In view of this, the present application is committed to providing a spectral image denoising method and apparatus, equipment and storage medium, which performs low-rank tensor approximation on the superpixel blocks in each superpixel tensor by setting a low-rank regularization term, thereby removing various noises in each superpixel block, and at the same time uses the contour gradient regularization term to limit the gradient amplitude along the pixel direction at the contour pixel to prevent unevenness and artifacts at the superpixel edge.

[0005] According to a first aspect of the present application, a spectral image denoising method is provided, comprising:

[0006] Performing superpixel segmentation processing on the first spectral image to be denoised to obtain a superpixel tensor containing superpixel blocks; the superpixel blocks are obtained by cutting the first spectral image along the spectral dimension; in the superpixel tensor corresponding to each superpixel block, all elements except the superpixel block are zero;

[0007] According to the first spectral image and the superpixel tensor, a denoising model of the first spectral image is constructed; the denoising model is used to solve the second spectral image so that the sum of the data fidelity term, the low-rank regularization term and the contour gradient regularization term obtains a minimum value; wherein the data fidelity term is used to represent the degree of difference between the first spectral image and the second spectral image obtained by denoising; the low-rank regularization term is used to perform low-rank tensor approximation on the superpixel block in each superpixel tensor; the contour gradient regularization term is used to represent the sum of the gradient amplitudes along the pixel direction at the contour pixels of the superpixel block;

[0008] A second spectral image of the denoising model is solved, and the finally obtained second spectral image is determined as the denoising result of the first spectral image.

[0009] Optionally, the contour gradient regularization term is the sum of total variation norms of contours of each superpixel block; the superpixel block contour includes contour pixels of its corresponding superpixel block.

[0010] Optionally, the low-rank regularization term is used to perform low-rank tensor approximation based on the ε norm on the superpixel block in the superpixel tensor; any tensor The ε norm of is:

[0011]

[0012] In the formula, I is a tensor The order of m i is the weight corresponding to different dimensions i, m i ≥0, For tensors The mode i of expansion; for The ath singular value of i ,∏ r≠i n r},n i For tensors The dimension size in dimension i;

[0013] Optionally, the denoising model is expressed as:

[0014]

[0015] In the formula, is the first spectral image, is the second spectral image; is the Gaussian noise in the first spectral image; ε is the sparse noise in the first spectral image; Used to represent the kth denoised superpixel block; Including edge pixels of the kth superpixel block, k∈K, K is the number of the superpixel blocks in the first spectral image; b∈B;

[0016] λ1, λ2 are regularization coefficients; ∥∥ F is the F norm of the tensor; ∥∥ ε is the ε norm of the tensor.

[0017] Optionally, solving the second spectral image of the denoising model includes:

[0018] The denoising model is solved by using an alternating direction multiplier method to obtain the second spectral image.

[0019] Optionally, the performing superpixel segmentation processing on the first spectral image to be denoised to obtain a superpixel tensor containing superpixel blocks includes:

[0020] Performing superpixel segmentation processing on the first spectral image to obtain superpixel blocks in the first spectral image;

[0021] Determining a regular tensor containing the superpixel block;

[0022] For each superpixel block, the elements in the regular tensor other than the superpixel block are set to zero to obtain the superpixel tensor corresponding to the superpixel block.

[0023] Optionally, setting elements in the regular tensor other than the superpixel block to zero to obtain the superpixel tensor corresponding to the superpixel block includes:

[0024] Generate a Boolean tensor for representing the position of the superpixel block in the regular tensor, and obtain a weight tensor corresponding to the superpixel block; each element in the weight tensor is used to indicate whether the element at the corresponding position in the regular tensor belongs to the superpixel block;

[0025] Calculate the element-by-element product of the rule tensor and the weight tensor to obtain the superpixel tensor.

[0026] Optionally, performing superpixel segmentation processing on the first spectral image to be denoised includes:

[0027] Based on a preset superpixel segmentation method, superpixel segmentation processing is performed on the first spectral image; the preset superpixel segmentation method includes: a superpixel segmentation method based on graph theory and a superpixel segmentation method based on gradient descent.

[0028] According to a second aspect of the present application, a spectral image denoising device is provided, comprising:

[0029] A superpixel segmentation module is used to perform superpixel segmentation on the first spectral image to be denoised to obtain a superpixel tensor containing superpixel blocks; the superpixel blocks are obtained by cutting the first spectral image along the spectral dimension; in the superpixel tensor corresponding to each superpixel block, all elements except the superpixel block are zero;

[0030] A model building module, used to build a denoising model of the first spectral image according to the first spectral image and the superpixel tensor; the denoising model is used to solve the second spectral image so that the sum of the data fidelity term, the low-rank regularization term and the contour gradient regularization term obtains a minimum value; wherein the data fidelity term is used to represent the degree of difference between the first spectral image and the second spectral image obtained by denoising; the low-rank regularization term is used to perform low-rank tensor approximation on the superpixel block in each superpixel tensor; the contour gradient regularization term is used to represent the sum of the gradient amplitudes along the pixel direction at the contour pixels of the superpixel block;

[0031] A solving module is used to solve the second spectral image of the denoising model, and determine the finally obtained second spectral image as the denoising result of the first spectral image.

[0032] According to a third aspect of the present application, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; and the processor is used to execute the method described in any one of the above embodiments.

[0033] According to a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the method described in any one of the above embodiments.

[0034] The present application provides a spectral image denoising method and device, equipment and storage medium, the scheme includes: performing superpixel segmentation processing on the first spectral image to be denoised to obtain a superpixel tensor containing superpixel blocks; then constructing a denoising model of the first spectral image based on the first spectral image and the superpixel tensor; the denoising model is used to solve the second spectral image so that the sum of the data fidelity term, the low-rank regularization term and the contour gradient regularization term obtains a minimum value; solve the second spectral image of the denoising model, and determine the second spectral image finally obtained as the denoising result of the first spectral image. On the one hand, by setting a low-rank regularization term, a low-rank tensor approximation is performed on the superpixel block in each superpixel tensor, thereby removing various noises in each superpixel block. On the other hand, the contour gradient regularization term is used to limit the gradient amplitude along the pixel direction at the contour pixel to prevent the occurrence of non-smoothing and artifacts at the edge of the superpixel. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 The figure is a schematic flow chart of a spectral image denoising method provided in one embodiment of the present application.

[0036] Figure 2 The figure shows a block diagram of a spectral image denoising device provided by one embodiment of the present application.

[0037] Figure 3 Shown is a structural block diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] This article uses lowercase or uppercase letters to represent scalars (x or X), lowercase bold font to represent vectors (x), uppercase bold font to represent matrices (X), and cursive letters to represent tensors.

[0040] Application Overview

[0041] In hyperspectral images, there is often a high correlation between adjacent bands and neighboring pixels, which makes hyperspectral images have low-rank characteristics. Specifically, since hyperspectral imaging technology can sample different parts of the electromagnetic spectrum with very fine resolution, and the reflectivity of some materials in the visible to near-infrared band may change slowly with wavelength, hyperspectral images have low-rank characteristics in the spectral dimension. Since the spatial distribution of objects is usually continuous, the spectral characteristics of adjacent pixels are often very similar. This spatial continuity and similarity leads to a high correlation of spectral characteristics between pixels. This means that within a certain area, the spectral characteristics of the pixels will be very close, forming a low-rank structure in the spatial dimension.

[0042] Therefore, the entire hyperspectral image can be regarded as a combination of a small number of basic components (i.e., different object types or materials). Although the hyperspectral image is a three-dimensional tensor (pixel length × pixel width × number of spectral channels), the actual effective information is often concentrated in a lower-dimensional space. In the absence of noise, the hyperspectral image can be regarded as a low-rank tensor; while the noise in the hyperspectral image (such as Gaussian noise, impulse noise, dead line and stripe noise, etc.) is usually randomly distributed and unstructured. The noise introduces new and irrelevant information in the spectral and spatial dimensions of the image, which requires a higher-rank tensor to represent, thereby destroying the original low-rank characteristics of the hyperspectral image. Therefore, the low-rank nature of the hyperspectral image is used as a priori knowledge in image restoration to remove various types of noise in the contaminated image. Specifically, for a given noisy hyperspectral image, we try to find a low-rank tensor to be as close to the hyperspectral image as possible while maintaining a low rank. In this way, we can effectively separate the low-rank components representing the real scene and the high-rank components representing the noise.

[0043] Due to the low-rank characteristics of superpixel images, superpixel images are divided into multiple three-dimensional superpixel blocks. All pixels in a superpixel block tend to share common spectral information and structure, so that the pixels in the superpixel block have strong correlation in both spatial and spectral dimensions. Based on the above properties, denoising is performed on the superpixel blocks separately, which can not only effectively maintain edge information and details, but also effectively eliminate most of the mixed noise. However, the above scheme easily leads to a sudden change in the spectral characteristics at the junction between superpixel blocks, resulting in an unsmooth junction and artifacts between superpixel blocks.

[0044] In order to solve the above problems, in an embodiment of the present application, a contour gradient regularization term (such as a TV regularization term) is set in the denoising model of the hyperspectral image. The contour gradient regularization term can be used to represent the sum of the gradient amplitudes along the pixel direction at the contour pixels of the superpixel block, so that the boundaries between each three-dimensional superpixel block remain smooth and remove artifacts generated between irregular superpixel blocks.

[0045] After introducing the basic principles of the present application, various non-limiting embodiments of the present application will be described in detail with reference to the accompanying drawings.

[0046] Exemplary Methods

[0047] Figure 1 It is a flowchart of a spectral image denoising method provided by an embodiment of the present application. Figure 1 The method is executed by a computing device (eg, a server), which is not limited in this embodiment of the present application.

[0048] like Figure 1As shown, the method includes the following contents:

[0049] Step S110: Perform superpixel segmentation processing on the first spectral image to be denoised to obtain a superpixel tensor containing superpixel blocks; the superpixel blocks are obtained by cutting the first spectral image along the spectral dimension; in the superpixel tensor corresponding to each superpixel block, all elements except the superpixel block are zero.

[0050] Generally, there are many different types of noise in hyperspectral images, such as Gaussian noise, fringe noise, impulse noise and dead lines. Among them, impulse noise, fringe noise and dead lines are sparse, so the mixed noise of impulse noise, fringe noise and dead lines is regarded as sparse noise.

[0051] In the embodiment of the present application, the first spectral image is a spectral image to be denoised, which may include a multispectral image, a hyperspectral image, and a hyperspectral image. The main difference between the three is the spectral resolution and the number of bands, that is, the number of bands in the first spectral image is multiple, and there is no specific limitation on this.

[0052] In the embodiment of the present application, the first spectral image can be expressed as a three-dimensional tensor (also called a third-order tensor): M, N, and B are respectively the pixel length, pixel width, and number of spectral channels (ie, number of bands) of the first spectral image. is the second spectral image; is the Gaussian noise in the first spectral image; is the sparse noise in the first spectral image. It should be noted that ε is the cursive of the English letter E, not the Greek letter ε.

[0053] In an embodiment of the present application, the superpixel segmentation process cuts the first spectral image along the spectral dimension to obtain multiple irregular three-dimensional superpixel blocks. A superpixel block can be regarded as a non-overlapping homogeneous area composed of multiple highly correlated pixels in the first spectral image. In short, a superpixel block can be regarded as spectral information of regions of similar materials (or even the same material) in different bands. The pixels in the superpixel block have strong correlation in both spatial and spectral dimensions and have the same semantic information (category information). Therefore, relative to the first spectral image, the superpixel block has a stronger tensor low-rank property.

[0054] In the embodiment of the present application, since the super pixel block is irregular in size and inconvenient to calculate and process, a super pixel tensor is used to represent it. In the super pixel tensor corresponding to each super pixel block, all elements except the super pixel block are zero.

[0055] Step S120: Construct a denoising model of the first spectral image according to the first spectral image and the superpixel tensor; the denoising model is used to solve the second spectral image so that the sum of the data fidelity term, the low-rank regularization term and the contour gradient regularization term obtains a minimum value; wherein the data fidelity term is used to represent the degree of difference between the first spectral image and the second spectral image obtained by denoising; the low-rank regularization term is used to perform low-rank tensor approximation on the superpixel block in each superpixel tensor; the contour gradient regularization term is used to represent the sum of the gradient amplitudes along the pixel direction at the contour pixels of the superpixel block.

[0056] In the embodiment of the present application, the data fidelity term can be used to maintain the similarity between the first spectral image and the second spectral image obtained by denoising, that is, the data fidelity of the second spectral image. The data fidelity term can be used to represent the noise of the first spectral image; in some cases, the data fidelity term can be used to represent the Gaussian noise in the first spectral image, that is, the data fidelity term is Among them, ∥∥ F is the F-norm (Frobenius norm) of the tensor, which is defined as the square root of the sum of the squares of all elements in the tensor.

[0057] In an embodiment of the present application, the low-rank regularization term is used to perform low-rank tensor approximation on the superpixel block in each of the superpixel tensors. As mentioned above, the superpixel block has a stronger tensor low-rank characteristic than the first spectral image. Performing low-rank tensor approximation on each superpixel block can effectively remove the high-rank parts that are considered to be noise. Compared with directly performing low-rank tensor approximation on the first spectral image, it can not only more effectively eliminate most of the mixed noise, but also help to maintain edge information and details.

[0058] In the embodiment of the present application, the contour pixel is a pixel on the contour of the superpixel block. Specifically, for a pixel belonging to a superpixel block, if its neighborhood includes pixels of other pixel blocks, then the pixel belongs to the contour pixel of the superpixel block.

[0059] In the embodiment of the present application, the contour gradient regularization term is used to represent the sum of the gradient amplitudes along the pixel direction at the contour pixels of the superpixel block. This is because the reason for the unevenness of the superpixel edge and the generation of artifacts is that the gradient amplitude at the contour pixels of the superpixel is large. Therefore, setting the contour gradient regularization term in the denoising model helps to suppress the unevenness and artifacts at the superpixel edge. The pixel can refer to a tensor A fiber y along the spectral direction mn:The pixel direction may include the pixel length direction and the pixel width direction.

[0060] Step S130: solving the second spectral image of the denoising model, and determining the finally obtained second spectral image as the denoising result of the first spectral image.

[0061] In the embodiment of the present application, the sum of the data fidelity term, the low-rank regularization term and the contour gradient regularization term can be regarded as the objective function of the second spectral image; the process of solving the denoising model is to find the second spectral image that makes the objective function take the minimum value. The second spectral image that makes the objective function take the minimum value is the optimal solution of the denoising model. The optimal solution of the denoising model is the second spectral image obtained by denoising the first spectral image.

[0062] In an embodiment of the present application, the first spectral image to be denoised is subjected to superpixel segmentation processing to obtain a superpixel tensor containing superpixel blocks; then, a denoising model of the first spectral image is constructed based on the first spectral image and the superpixel tensor; the denoising model is used to solve the second spectral image so that the sum of the data fidelity term, the low-rank regularization term and the contour gradient regularization term obtains a minimum value; finally, the second spectral image of the denoising model is solved, and the second spectral image finally obtained is determined as the denoising result of the first spectral image. On the one hand, by setting a low-rank regularization term, a low-rank tensor approximation is performed on the superpixel block in each superpixel tensor, thereby removing various noises in each superpixel block. On the other hand, the contour gradient regularization term is used to limit the gradient amplitude along the pixel direction at the contour pixel to prevent the occurrence of non-smoothing and artifacts at the edge of the superpixel.

[0063] based on Figure 1 The method in this specification also provides some specific implementation plans of the method, which are described below.

[0064] Optionally, the contour gradient regularization term is the sum of total variation norms of contours of each superpixel block; the superpixel block contour includes contour pixels of its corresponding superpixel block.

[0065] In the embodiment of the present application, the superpixel segmentation result may be The segmentation result matrix has the same pixel size as The elements in the matrix are used to represent the superpixel flags of the pixels at the corresponding positions of the first spectral image.

[0066] In an embodiment of the present application, for each superpixel block, if a pixel in a superpixel block has a neighborhood (four neighborhoods or eight neighborhoods) that includes pixels with different superpixel marks (i.e., pixels of other superpixel blocks), then the pixel belongs to the contour pixel of the superpixel block.

[0067] In an embodiment of the present application, the total variation norm (Total Variation Norm) can be used to measure the total variation of an image in image processing, that is, the sum of the intensity changes of all pixels in the image. The total variation norm can be used to calculate the total variation norm of each superpixel block contour in each spectral dimension in the sub-image corresponding to each spectral dimension; the sum of the total variation norms of each superpixel block contour in each spectral dimension is calculated to obtain the total variation norm of the superpixel block contour. The sub-image corresponding to each spectral dimension is the first spectral image Each slice along the spectral direction

[0068] In an embodiment of the present application, the total variation norm of the contour of any superpixel block is calculated as follows: for each contour pixel of the superpixel block, the first gradient of the contour pixel along the pixel length direction and the second gradient along the pixel width direction are calculated in each spectral dimension; the first square sum of the first gradient and the second square sum of the second gradient of each contour pixel are calculated; the arithmetic square root of the first square sum and the second square sum are calculated to obtain the total variation norm of the contour of the superpixel block.

[0069] In the embodiment of the present application, in order to simplify the calculation, the first spectral image can be first Expand along the spectral direction (modal b expansion) to obtain an expansion matrix. Based on the expansion matrix, the sum of the total variation norms of the contours of each superpixel block is calculated using the existing total variation formula.

[0070] The calculation formula of the total variation norm is as follows:

[0071]

[0072] In the formula, S (k) is the superpixel block outline of the kth superpixel block in the unfolding matrix (modal b unfolding); S (k)p,q is the contour element corresponding to the contour pixel in the expansion matrix, S (k)p+1,q , S (k)p,q+1 are the next pixel of the contour element along the pixel length direction and the pixel width direction respectively. It should be noted that the number of contour elements corresponding to a pixel in the first spectral image is equal to the number of spectral channels (ie, the number of bands).

[0073] Similarly, by calculating the Kronecker Product of the M×N all-one matrix and the pixel vector corresponding to each pixel, another unfolded matrix can be obtained. Based on the unfolded matrix, the sum of the total variation norms of the contours of each superpixel block can also be calculated; wherein, the length and width of the all-one matrix are the pixel length and pixel width respectively; the elements in the all-one matrix are all 1 (also called the unit element matrix); the pixel vector corresponding to any pixel is a tensor Fibers along the spectrum

[0074] In an embodiment of the present application, the contour gradient regularization term is set to the sum of the total variation norms of the contours of each superpixel block, thereby removing noise at the superpixel edge and maintaining or enhancing the main edge structure, while preventing unevenness and artifacts at the superpixel edge.

[0075] Optionally, the low-rank regularization term is used to perform low-rank tensor approximation based on the ε norm on the superpixel block in the superpixel tensor; any tensor The ε norm of is:

[0076]

[0077] In the formula, I is a tensor The order of m i is the weight corresponding to different dimensions i, m i ≥0, For tensors The mode i of expansion; for The ath singular value of i ,∏ r≠i n r},n i For tensors The dimension size in dimension i;

[0078] In the embodiment of the present application, the first spectral image is generally represented as a third-order tensor, i=1,2,3; r=1,2,3; n i , n2, and n3 are pixel length, pixel width, and number of spectral channels, respectively.

[0079] In the prior art, the sum of nuclear norms is usually used for low-rank tensor approximation, but there are still two disadvantages. First, it measures the singular values ​​by the L1 norm, which is not a good substitute for Tucker rank. Second, the norm treats each singular value equally, which may lead to the loss of main information, because larger singular values ​​usually correspond to main information, which means that shrinkage should be reduced to retain more information.

[0080] In the embodiment of the present application, the ε norm is used as a non-convex alternative to the Tucker rank. It approximates the rank of the tensor better than the sum of nuclear norms, which helps promote the low-rank nature of the solution. In addition, the singular values ​​are adjusted based on the weighted singular value threshold of the ε norm, and the singular values ​​are treated differently based on the weighted singular value threshold of the ε norm. This norm enables us to retain the main details by shrinking the larger singular values ​​less, while effectively suppressing the noise by shrinking the smaller singular values ​​more for the smaller singular values.

[0081] Optionally, the denoising model is expressed as:

[0082]

[0083] In the formula, is the first spectral image, is the second spectral image; is the Gaussian noise in the first spectral image; ε is the sparse noise in the first spectral image; Used to represent the kth denoised superpixel block; Including edge pixels of the kth superpixel block, k∈K, K is the number of superpixel blocks in the first spectral image; b∈B; λ1, λ2 are regularization coefficients; ∥∥ F is the F norm of the tensor; ∥∥ ε is the ε norm of the tensor.

[0084] In the embodiments of the present application, is the data fidelity item; is the low-rank regularization term; is the contour gradient regularization term. λ1 and λ2 are regularization coefficients of the low-rank regularization term and the contour gradient regularization term, respectively.

[0085] In the embodiments of the present application, is a regular tensor containing the kth superpixel block; is a part of the second spectral image after denoising. Superpixel block A corresponding weight tensor, each element in the weight tensor is used to indicate whether the element at the corresponding position in the regular tensor belongs to the superpixel block. is a regular tensor and the weight tensor The element-wise product of All elements except the superpixel block are zero. Used to represent irregular superpixel blocks, is the weight tensor; Used to represent the regular tensor after denoising

[0086] In an embodiment of the present application, when the first spectral image is processed based on the denoising model, on the one hand, the denoised second spectral image is made as close as possible to the original first spectral image through the data fidelity term, and the noise in the irregular super-pixel blocks is removed through the low-rank regularization term. At the same time, the gradient amplitude of the super-pixel edge is reduced through the contour gradient regularization term to prevent unevenness and artifacts at the super-pixel edge.

[0087] Optionally, solving the second spectral image of the denoising model includes:

[0088] The denoising model is solved by using an alternating direction multiplier method to obtain the second spectral image.

[0089] In the embodiment of the present application, the Alternating Direction Method of Multipliers (ADMM) is an optimization algorithm for solving convex optimization problems with separable structures. The core idea of ​​the ADMM algorithm is to decompose a complex problem into several simpler and easier to solve sub-problems, and gradually approach the solution of the original problem in an iterative manner.

[0090] In an embodiment of the present application, the alternating direction multiplier method includes: introducing auxiliary variables to transform the denoising model into an unconstrained optimization problem, performing alternating iterative solutions to the unknown variables therein, and obtaining the second spectral image when the number of iteration steps reaches the maximum number or the original residual and the dual residual are both less than a preset tolerance.

[0091] In the embodiment of the present application, an auxiliary variable is introduced The denoising model is reformulated as:

[0092]

[0093] wxya k ⊙X k =P ik ,i=1,2,3,k=1,…,K

[0094] Introduce auxiliary variable V (k) =S (k) , Lagrange multipliers and η k , and represents the penalty parameter α ik and μ k , transform the above denoising model into an unconstrained augmented Lagrangian function: L

[0095]

[0096] The variables in the above formula are updated alternately and iteratively in the loop to obtain the second spectral image. The specific process is as follows:

[0097] Update P ik ;P ik The corresponding sub-questions are as follows:

[0098]

[0099] renew The corresponding sub-questions are as follows:

[0100]

[0101] Where t is the number of iterations of iterative update.

[0102] The sub-problem corresponding to the k-th superpixel block is:

[0103]

[0104] Introducing variables The problem corresponding to the k-th superpixel block is transformed into:

[0105]

[0106] After denoising each superpixel block, the updated

[0107] Update S (k) , S (k) The corresponding sub-questions are as follows:

[0108]

[0109] Updated for:

[0110]

[0111] Update V k , V kThe corresponding sub-questions are as follows:

[0112]

[0113] To solve the TV minimization problem, a fast gradient projection algorithm is used. where P is a linear operator, is the orthogonal projection operator, and (a,b) are the variables it acts on.

[0114] In the embodiment of the present application, the denoising model is solved by using the alternating direction multiplier method, the complex problem of the denoising model is decomposed into a number of simpler and easier to solve sub-problems, and the solution of the original problem is gradually approached by iteration. The variables in the denoising model and the associated Lagrange multipliers are updated in sequence through the iterative process, and the denoised second spectral image can be obtained after reaching the convergence standard, thereby significantly shortening the solution time of the denoising model without reducing the quality of the second spectral image.

[0115] Optionally, the performing superpixel segmentation processing on the first spectral image to be denoised to obtain a superpixel tensor containing superpixel blocks includes:

[0116] Performing superpixel segmentation processing on the first spectral image to obtain superpixel blocks in the first spectral image;

[0117] Determining a regular tensor containing the superpixel block;

[0118] For each superpixel block, the elements in the regular tensor other than the superpixel block are set to zero to obtain the superpixel tensor corresponding to the superpixel block.

[0119] In an embodiment of the present application, the superpixel segmentation processing is performed on the first spectral image to obtain each superpixel block in the first spectral image, including: for each pixel in the first spectral image, calculating the average value of the intensity of the pixel in each spectral channel to obtain a smooth band image of the first spectral image; performing superpixel segmentation processing on the smooth band image to obtain a segmentation result of the smooth band image; based on the segmentation result, dividing the first spectral image along the spectral direction to obtain each superpixel block in the first spectral image.

[0120] In the embodiment of the present application, the regular tensor may be a three-dimensional tensor including the superpixel block. The regular tensor may be the first spectral image, or may be the minimum regular tensor including the superpixel block in the first spectral image, or may be the minimum regular tensor including the superpixel block and its adjacent pixels.

[0121] In the embodiment of the present application, the determining of the regular tensor containing the superpixel block may include: determining the superpixel area of ​​each superpixel block in the smooth band image according to the segmentation result of the smooth band image; the segmentation result may be the same as the first spectral image The segmentation result matrix has the same pixel size as The elements in the segmentation result matrix are used to represent the superpixel flags of the pixels at the corresponding positions of the first spectral image.

[0122] In an embodiment of the present application, the first spectral image is subjected to superpixel segmentation processing to obtain each superpixel block in the first spectral image, and a regular tensor containing the superpixel block; the elements in the regular tensor except the superpixel block are set to zero to obtain the superpixel tensor corresponding to the superpixel block. Thus, the superpixel tensor is used to represent the superpixel block. Compared with expanding the superpixel block into a two-dimensional matrix, the superpixel tensor can retain the spatial-spectral information obtained by the superpixel segmentation technology, and retain the inherent structure and information of the tensor. At the same time, the matrix optimization problem is converted into a tensor optimization problem, so that the low-rank tensor approximation method can be used to better explore the correlation between the spatial dimension and the spectral dimension and remove the noise in the superpixel block.

[0123] Optionally, setting elements in the regular tensor other than the superpixel block to zero to obtain the superpixel tensor corresponding to the superpixel block includes:

[0124] Generate a Boolean tensor for representing the position of the superpixel block in the regular tensor, and obtain a weight tensor corresponding to the superpixel block; each element in the weight tensor is used to indicate whether the element at the corresponding position in the regular tensor belongs to the superpixel block;

[0125] Calculate the element-by-element product of the rule tensor and the weight tensor to obtain the superpixel tensor.

[0126] In an embodiment of the present application, the weight tensor can be a two-dimensional weight matrix with the size of the pixel length and pixel width of the regular tensor, or a three-dimensional weight tensor with the same size as the regular tensor. The slices of the three-dimensional weight tensor along the spectral direction are all two-dimensional weight matrices.

[0127] In an embodiment of the present application, the weight tensor is a Boolean tensor (0-1 tensor). The value of each element in the weight tensor is 1, indicating that the element at the corresponding position in the regular tensor belongs to the superpixel block; the value of each element in the weight tensor is 0, indicating that the element at the corresponding position in the regular tensor does not belong to the superpixel block.

[0128] In an embodiment of the present application, when the weight tensor is a three-dimensional weight tensor, the superpixel tensor is the element-wise product (also called Hadamard product) of the regular tensor and the weight tensor. It can be expressed as: In the formula, is a three-dimensional weight tensor, is a regular tensor containing the kth superpixel block.

[0129] In the embodiment of the present application, when the weight tensor is a two-dimensional weight matrix, each pixel vector in the regular tensor (the fiber of the regular tensor along the spectral direction) can be multiplied by the corresponding weight value (the element value at the corresponding position in the two-dimensional weight matrix) to obtain the superpixel tensor. Alternatively, for each slice of the superpixel tensor along the spectral direction, the element-by-element product of the slice and the two-dimensional weight matrix can be calculated to obtain the superpixel tensor.

[0130] Optionally, performing superpixel segmentation processing on the first spectral image to be denoised includes:

[0131] Based on a preset superpixel segmentation method, superpixel segmentation processing is performed on the first spectral image; the preset superpixel segmentation method includes: a superpixel segmentation method based on graph theory and a superpixel segmentation method based on gradient descent.

[0132] In an embodiment of the present application, the superpixel segmentation processing is performed on the first spectral image to obtain each superpixel block in the first spectral image, including: for each pixel in the first spectral image, calculating the average value of the intensity of the pixel in each spectral channel to obtain a smooth band image of the first spectral image; based on a preset superpixel segmentation method, the superpixel segmentation processing is performed on the smooth band image to obtain a segmentation result of the smooth band image; based on the segmentation result, the first spectral image is divided along the spectral direction to obtain each superpixel block in the first spectral image.

[0133] In an embodiment of the present application, the superpixel segmentation processing of the first spectral image to obtain each superpixel block in the first spectral image includes: based on a preset conversion algorithm, converting the first spectral image into a three-channel color image (i.e., RGB image); based on a preset superpixel segmentation method, performing superpixel segmentation processing on the three-channel color image to obtain a segmentation result of the three-channel color image; based on the segmentation result, segmenting the first spectral image along the spectral direction to obtain each superpixel block in the first spectral image. The preset conversion algorithm may include direct band selection, principal component analysis (PCA), linear transformation (such as tasseled cap transformation), spectral reconstruction and machine learning methods, etc.

[0134] In the embodiment of the present application, the graph-theory-based superpixel segmentation method uses a mesh graph in a data structure to describe the image data to be superpixel segmented, wherein each node in the graph corresponds to each pixel in the image. The weight of the edge is proportional to the similarity between adjacent pixels, and the size of the weight value represents the similarity and difference of the features between adjacent pixels, defines the standard of superpixel segmentation, and achieves the superpixel segmentation result of the target segmentation number by effectively minimizing the penalty function.

[0135] In the embodiment of the present application, the graph-based superpixel segmentation method may include graph-based, entropy rate-based and normalized cuts algorithms, etc. The gradient descent-based superpixel segmentation method mainly includes Turbopixels, SLIC and other methods.

[0136] In an embodiment of the present application, a new objective function for superpixel segmentation is proposed in an algorithm based on entropy rate. This objective function consists of two parts: the entropy rate of random walks on the graph and a balance term. The entropy rate tends to form compact and uniform clusters, encourages the partitioning of images on perceptual boundaries, and tends to favor superpixels that overlap with a single object; while the balance term encourages clusters of similar size and dimensions, thereby reducing the number of unbalanced superpixels. This method proposes a novel image graph construction method and shows that this construction method leads to a quasi-array structure, a generalization of the concept of linear independence in vector space. Then, according to the constraints of the quasi-array, segmentation is performed by optimizing the objective function to maximize the topological structure of the graph. By exploiting the submodularity and monotonicity of the objective function, an efficient greedy algorithm based on the entropy rate is developed.

[0137] Exemplary Devices

[0138] The device embodiments of the present application can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0139] Figure 2 FIG. 1 is a block diagram of a spectral image denoising device provided by an embodiment of the present application. Figure 2 As shown, the device 200 includes:

[0140] The superpixel segmentation module 210 is used to perform superpixel segmentation on the first spectral image to be denoised to obtain a superpixel tensor including superpixel blocks; the superpixel blocks are obtained by cutting the first spectral image along the spectral dimension; in the superpixel tensor corresponding to each superpixel block, all elements except the superpixel block are zero;

[0141] The model building module 220 is used to build a denoising model of the first spectral image according to the first spectral image and the superpixel tensor; the denoising model is used to solve the second spectral image so that the sum of the data fidelity term, the low-rank regularization term and the contour gradient regularization term obtains a minimum value; wherein the data fidelity term is used to represent the degree of difference between the first spectral image and the second spectral image obtained by denoising; the low-rank regularization term is used to perform low-rank tensor approximation on the superpixel block in each superpixel tensor; the contour gradient regularization term is used to represent the sum of the gradient amplitudes along the pixel direction at the contour pixels of the superpixel block;

[0142] The solving module 230 is used to solve the second spectral image of the denoising model, and determine the finally obtained second spectral image as the denoising result of the first spectral image.

[0143] Optionally, the contour gradient regularization term is the sum of total variation norms of contours of each superpixel block; the superpixel block contour includes contour pixels of its corresponding superpixel block.

[0144] Optionally, the low-rank regularization term is used to perform low-rank tensor approximation based on the ε norm on the superpixel block in the superpixel tensor; any tensor The ε norm of is:

[0145]

[0146] In the formula, I is a tensor The order of m i is the weight corresponding to different dimensions i, m i ≥0, For tensors The mode i of expansion; for The ath singular value of i ,∏ r≠i n r},n i For tensors The dimension size in dimension i;

[0147] Optionally, the denoising model is expressed as:

[0148]

[0149] In the formula, is the first spectral image, is the second spectral image; is the Gaussian noise in the first spectral image; ε is the sparse noise in the first spectral image; Used to represent the kth denoised superpixel block; Including edge pixels of the kth superpixel block, k∈K, K is the number of the superpixel blocks in the first spectral image; b∈B;

[0150] λ1, λ2 are regularization coefficients; ∥∥ F is the F norm of the tensor; ∥∥ ε is the ε norm of the tensor.

[0151] Optionally, the solution module 230 is used to:

[0152] The denoising model is solved by using an alternating direction multiplier method to obtain the second spectral image.

[0153] Optionally, the superpixel segmentation module 210 includes:

[0154] A superpixel segmentation unit, configured to perform superpixel segmentation processing on the first spectral image to obtain superpixel blocks in the first spectral image;

[0155] A regular tensor determination unit, used to determine a regular tensor containing the superpixel block;

[0156] The superpixel tensor unit is used to set the elements of the regular tensor other than the superpixel block to zero for each superpixel block, so as to obtain the superpixel tensor corresponding to the superpixel block.

[0157] Optionally, the superpixel tensor unit is used to:

[0158] Generate a Boolean tensor for representing the position of the superpixel block in the regular tensor, and obtain a weight tensor corresponding to the superpixel block; each element in the weight tensor is used to indicate whether the element at the corresponding position in the regular tensor belongs to the superpixel block;

[0159] Calculate the element-by-element product of the rule tensor and the weight tensor to obtain the superpixel tensor.

[0160] Optionally, the superpixel segmentation module 210 is used to:

[0161] Based on a preset superpixel segmentation method, superpixel segmentation processing is performed on the first spectral image; the preset superpixel segmentation method includes: a superpixel segmentation method based on graph theory and a superpixel segmentation method based on gradient descent.

[0162] Exemplary Electronic Devices

[0163] Below, reference Figure 3 To describe an electronic device according to an embodiment of the present application. Figure 3 A block diagram of an electronic device according to an embodiment of the present application is illustrated.

[0164] like Figure 3 As shown, the electronic device 300 includes one or more processors 310 and a memory 320 .

[0165] The processor 310 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 300 to perform desired functions.

[0166] The memory 320 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 310 may run the program instructions to implement the spectral image denoising method of each embodiment of the present application described above and / or other desired functions. Various contents such as category correspondences may also be stored in the computer-readable storage medium.

[0167] In one example, the electronic device 300 may further include: an input device 330 and an output device 340 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0168] In addition, the input device 330 may also include, for example, a keyboard, a mouse, etc. The output device 340 may output various information to the outside. The output device 340 may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0169] Of course, to simplify, Figure 3 Only some of the components related to the present application in the electronic device 300 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 300 may also include any other appropriate components.

[0170] Exemplary computer program products and computer-readable storage media

[0171] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the spectral image denoising method according to various embodiments of the present application described in the above-mentioned "Exemplary Method" section of this specification.

[0172] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0173] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the spectral image denoising method according to various embodiments of the present application described in the above "Exemplary Method" section of this specification.

[0174] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0175] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0176] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0177] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0178] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0179] It should be understood that the qualifiers "first", "second", "third", "fourth", "fifth" and "sixth" used in the description of the embodiments of the present application are only used to more clearly explain the technical solutions and cannot be used to limit the scope of protection of the present application.

[0180] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A spectral image denoising method, characterized in that: include: Performing superpixel segmentation processing on the first spectral image to be denoised to obtain a superpixel tensor containing superpixel blocks; The superpixel block is obtained by cutting the first spectral image along the spectral dimension; in the superpixel tensor corresponding to each superpixel block, all elements except the superpixel block are zero; According to the first spectral image and the superpixel tensor, a denoising model of the first spectral image is constructed; the denoising model is used to solve the second spectral image so that the sum of the data fidelity term, the low-rank regularization term and the contour gradient regularization term obtains a minimum value; wherein the data fidelity term is used to represent the degree of difference between the first spectral image and the second spectral image; the low-rank regularization term is used to perform low-rank tensor approximation on the superpixel block in each superpixel tensor; the contour gradient regularization term is used to represent the sum of the gradient amplitudes along the pixel direction at the contour pixels of the superpixel block; The second spectral image in the denoising model is solved, and the finally obtained second spectral image is determined as the denoising result of the first spectral image.

2. The method according to claim 1, characterized in that The contour gradient regularization term is the sum of the total variation norms of the contours of each superpixel block; the superpixel block contour includes contour pixels of its corresponding superpixel block.

3. The method according to claim 1, characterized in that The low-rank regularization term is used to perform low-rank tensor approximation based on the ε norm on the superpixel block in the superpixel tensor; any tensor The ε norm of is: In the formula, I is a tensor The order of m i is the weight corresponding to different dimensions i, m i ≥0, For tensors The mode i of expansion; for The ath singular value of i , Π r≠i n r },n i For tensors The dimension size in dimension i; i∈I, r∈I.

4. The method according to claim 2, characterized in that: The denoising model is expressed as: In the formula, is the first spectral image, is the second spectral image; is the Gaussian noise in the first spectral image; ε is the sparse noise in the first spectral image; Used to represent the kth denoised superpixel block; Including edge pixels of the kth superpixel block, k∈K, K is the number of the superpixel blocks in the first spectral image; b∈B; λ1, λ2 are regularization coefficients; ∥∥ F is the F norm of the tensor; ∥∥ ε is the ε norm of the tensor.

5. The method according to claim 1, characterized in that The second spectral image of solving the denoising model comprises: The denoising model is solved by using an alternating direction multiplier method to obtain the second spectral image.

6. The method according to claim 1, characterized in that The first spectral image to be denoised is subjected to superpixel segmentation processing to obtain a superpixel tensor containing superpixel blocks, including: Performing superpixel segmentation processing on the first spectral image to obtain superpixel blocks in the first spectral image; Determining a regular tensor containing the superpixel block; For each superpixel block, the elements in the regular tensor other than the superpixel block are set to zero to obtain the superpixel tensor corresponding to the superpixel block.

7. The method according to claim 6, characterized in that The step of setting the elements in the regular tensor other than the superpixel block to zero to obtain the superpixel tensor corresponding to the superpixel block includes: Generate a Boolean tensor for representing the position of the superpixel block in the regular tensor, and obtain a weight tensor corresponding to the superpixel block; each element in the weight tensor is used to indicate whether the element at the corresponding position in the regular tensor belongs to the superpixel block; Calculate the element-by-element product of the rule tensor and the weight tensor to obtain the superpixel tensor.

8. A spectral image denoising device, characterized in that: include: A superpixel segmentation module, used for performing superpixel segmentation on the first spectral image to be denoised to obtain a superpixel tensor containing superpixel blocks; The superpixel block is obtained by cutting the first spectral image along the spectral dimension; in the superpixel tensor corresponding to each superpixel block, all elements except the superpixel block are zero; A model building module, used to build a denoising model of the first spectral image according to the first spectral image and the superpixel tensor; the denoising model is used to solve the second spectral image so that the sum of the data fidelity term, the low-rank regularization term and the contour gradient regularization term obtains a minimum value; wherein the data fidelity term is used to represent the degree of difference between the first spectral image and the second spectral image obtained by denoising; the low-rank regularization term is used to perform low-rank tensor approximation on the superpixel block in each superpixel tensor; the contour gradient regularization term is used to represent the sum of the gradient amplitudes along the pixel direction at the contour pixels of the superpixel block; A solving module is used to solve the second spectral image of the denoising model, and determine the finally obtained second spectral image as the denoising result of the first spectral image.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is used to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 7.

Images