Signal processing device, signal processing method, and signal processing program

The signal processing device and method use a cusp star norm and ADMM to independently adjust gradation smoothness and edge steepness in edge-preserving smoothing, addressing the limitations of existing methods and reducing calculation time.

JP7590673B2Active Publication Date: 2024-11-27NIPPON TELEGRAPH & TELEPHONE CORP
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2023504954
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-03-10
Publication Date
2024-11-27
Estimated Expiration
2041-03-10

AI Technical Summary

Technical Problem

Existing global optimization methods for edge-preserving smoothing fail to independently adjust both the smoothness of gradation and the steepness of edges, leading to issues such as blurry outputs or pseudo-contours.

Method used

A signal processing device and method using a cusp star norm as a non-convex function to adjust gradation smoothness and edge steepness through a sparse regularization function, combined with the Alternating Direction Method of Multipliers (ADMM) to solve the optimization problem efficiently.

Benefits of technology

Enables independent adjustment of gradation smoothness and edge steepness in edge-preserving smoothing, reducing calculation time while maintaining robustness and effectiveness.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007590673000057
    Figure 0007590673000057
  • Figure 0007590673000058
    Figure 0007590673000058
  • Figure 0007590673000059
    Figure 0007590673000059
Patent Text Reader

Abstract

This signal processing device is provided with a control unit for generating, from an input signal, an output signal that has undergone edge-preserving smoothing by: applying, to the input signal, a sparse regularization function which is a nonconvex function and which changes the smoothness of gradation by adjusting a first control parameter included in weights of the nonconvex function and changes the steepness of an edge by adjusting a second control parameter included in the weights of the nonconvex function; and calculating a solution to an optimization problem using global optimization.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a signal processing device, a signal processing method, and a signal processing program. [Background technology]

[0002] Edge-preserving smoothing is one of the important processes in image processing. Edge-preserving smoothing captures edges and contours to clarify the structure of an image and smooth out unimportant details. Edge-preserving smoothing not only improves image quality, but is also used as a preprocessing step for various image processing such as contour extraction, segmentation, and style transfer. Therefore, various edge-preserving smoothing methods have been proposed so far, as shown in Figure 9.

[0003] As shown in Fig. 9, edge-preserving smoothing methods are classified into local filter methods and global optimization methods. Global optimization methods perform edge-preserving smoothing by formulating an optimization problem using the statistical properties of the entire image, finding a solution, and calculating the output. Although global optimization methods require a long execution time, they have high edge preservation performance, are less likely to produce artifacts, and have better smoothing performance than local filter methods.

[0004] In global optimization techniques, several control parameters are used. In many global optimization techniques, the smoothness of an image is adjusted, for example, by changing the value of a control parameter. For example, the control parameter λ of Total Variation (hereinafter also referred to as "TV") adjusts the smoothness of gradation by balancing the fidelity term and the TV term (see, for example, Non-Patent Documents 1 and 2). In contrast, in L0 gradient projection, the control parameter α is changed to change the total number of pixels where an edge exists, thereby adjusting a steep edge (see, for example, Non-Patent Document 3). [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] LI Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Physica D: nonlinear phenomena, vol.60, no.1-4, pp.259-268, 1992 [Non-Patent Document 2] A. Chambolle, “An Algorithm for Total Variation Minimization and Applications,” Journal of Mathematical Imaging and Vision, vol.20, no.1-2, pp.89-97, 2004 [Non-Patent Document 3] S. Ono, “L0 Gradient Projection,” IEEE Transactions on Image Processing, vol.26, no.4, pp.1554-1564, 2017 Summary of the Invention [Problem to be solved by the invention]

[0006] However, there is no global optimization method that adjusts both the smoothness of the gradation and the steepness of the edge. Therefore, if the smoothness of the gradation is increased using a global optimization method that adjusts the smoothness of the gradation, the edges become dull and the output becomes blurred. On the other hand, if the steepness of the edge is increased using a global optimization method that adjusts the steepness of the edge, the gradation deteriorates into a false contour.

[0007] In view of the above circumstances, an object of the present invention is to provide a technique capable of independently adjusting both the smoothness of gradation and the sharpness of edges in edge-preserving smoothing through global optimization. [Means for solving the problem]

[0008] One aspect of the present invention is a signal processing device that includes a control unit that generates an edge-preserving, smoothed output signal from an input signal by applying a sparse regularization function of a non-convex function to an input signal, the sparse regularization function changing smoothness of a gradation by adjusting a first control parameter included in a weight of the non-convex function, and changing steepness of an edge by adjusting a second control parameter included in a weight of the non-convex function, thereby calculating a solution to an optimization problem by global optimization.

[0009] One aspect of the present invention is a signal processing method performed by a signal processing device, which generates an edge-preserving, smoothed output signal from an input signal by applying a sparse regularization function of a non-convex function to the input signal, the sparse regularization function changing the smoothness of a gradation by adjusting a first control parameter included in a weight of the non-convex function, and by adjusting a second control parameter included in a weight of the non-convex function, the sparse regularization function changing the steepness of an edge by calculating a solution to an optimization problem by global optimization.

[0010] One aspect of the present invention is a signal processing program for causing a computer to operate as the above-mentioned signal processing device. Effect of the Invention

[0011] The present invention enables independent adjustment of both gradation smoothness and edge sharpness in edge-preserving smoothing by global optimization. [Brief description of the drawings]

[0012] [Figure 1] FIG. 11 is a diagram comparing the contour lines of the cusp norm according to the present embodiment with the contour lines of the L1 norm and the Lp pseudo norm, which are examples of other norms. [Diagram 2] 1 is a block diagram showing an internal configuration of a signal processing device according to an embodiment of the present invention; [Diagram 3]3 is a block diagram showing an internal configuration of a first variable update unit according to the present embodiment. FIG. [Figure 4] 4 is a block diagram showing an internal configuration of a second variable update unit according to the present embodiment. FIG. [Diagram 5] 4 is a block diagram showing an internal configuration of a third variable update unit according to the present embodiment. FIG. [Figure 6] 4 is a flowchart showing a flow of processing by the signal processing device according to the present embodiment. [Figure 7] FIG. 1 is a diagram (part 1) showing an implementation result using the signal processing device of the present embodiment. [Figure 8] FIG. 2 is a diagram (part 2) showing an implementation result using the signal processing device of the present embodiment. [Figure 9] FIG. 1 illustrates types of hedge-preserving smoothing methods. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. As described above, the present invention aims to independently adjust both the smoothness of gradation and the steepness of edges in edge-preserving smoothing by global optimization. To achieve this aim, the use of a cusp norm, which will be described later, is proposed to enable the incorporation of two parameters, a parameter for adjusting the smoothness of gradation and a parameter for adjusting the steepness of edges. In addition, since the cusp norm is a non-convex function, it is known that a very long calculation time is generally required to solve the optimization problem formulated in edge-preserving smoothing by global optimization, and a method for solving this problem of calculation time is proposed below. First, the optimization problem formulated in the method of edge-preserving smoothing by global optimization used in this embodiment and the solution method thereof will be described. An input color image signal having three channels, RGB, as a target for hedge-preserving smoothing is defined by the following equation (1).

[0014]

number

[0015] In formula (1), N = N v ×N h and N v is the number of pixels in the vertical direction, and N h is the number of pixels in the horizontal direction. Since there are three channels, red, green, and blue, it is "3N" in formula (1). In formula (1), vector f is a column vector obtained by vertically scanning and stacking the input color image signal for each channel, and is hereinafter referred to as input image signal vector f. The output image signal vector x obtained by performing the hedge-preserving smoothing of this embodiment on the input image signal vector f is defined by the following formula (2).

[0016]

number

[0017] The following equation (3) represents the optimization problem of edge-preserving smoothing of this embodiment expressed using the above-mentioned input image signal vector f and output image signal vector x.

[0018]

number

[0019] The function in the second term of equation (3) is the pointed star gradient (hereinafter also referred to as "PSG" (Pointed Star Gradient)). The name "pointed star" comes from the pointed star norm, which will be described later. The subscript vector w of this function is a vector containing N weight parameters defined by the following equation (4). Hereinafter, vector w will be referred to as the weight vector w.

[0020]

number

[0021] Here, the elements included in the output image signal vector x, that is, the pixel values, can be expressed as the following equation (5).

[0022]

number

[0023] In equation (5), the subscript "i" refers to the vertical pixel position, the subscript "j" refers to the horizontal pixel position, and the subscript "c" refers to the color, i.e., red, blue, or green.

[0024] The function of the second term in equation (3) is defined as the following equation (6).

[0025]

number

[0026] In equation (6), the function Grad(·) is a function that calculates the gradient at each pixel of the image signal vector given as an argument. The vec operator is an operator that vertically aligns each column of the matrix given as an argument. When the output image signal vector x of equation (5) is given as an argument to the function Grad(·), the output of the function Grad(·) at pixel positions "i" and "j" is given by the following equation (7).

[0027]

number

[0028] However, in equation (7), i+1>N v If x i+1,j,c -x i,j,c = 0, and j+1>N h If x i,j+1,c -x i,j,c = 0. In addition, in formula (7), c is one of the values ​​1, 2, or 3, and indicates one of red, green, or blue.

[0029] In equation (6), the function Ω w(·) (where the subscript w is the weight vector w) is the ordered weighted L1-norm (hereinafter referred to as “OWL” (Ordered Weighted L1-norm)) shown in Reference 1 below.

[0030] “Reference 1: X. Zeng and MA Figueiredo, “The ordered weighted L1norm: Atomic formulation, projections, and algorithms”, arXiv:1409.4271, pp.1-13, 2014”

[0031] Function Ω w The argument of (·) is a vector u = (u1, u2, …, u N ) T Given the function Ω w (·) is expressed as the following equation (8).

[0032]

number

[0033] In equation (8), |·| [n] The operation is performed by giving a vector u as an argument, |u1|,|u2|,…,|u N This is an operation to select the n-th largest value of |. In formula (8), the function |·| ↓ Given a vector u as an argument, returns a vector whose elements are the absolute values ​​of the elements of vector u, that is, (|u1|,|u2|,…,|u N |) T This is an operation that generates a vector by sorting the elements of in descending order.

[0034] Function Ω w Let the elements of the weight vector w in (·) be non-negative and non-decreasing, i.e. (0≦w1≦w2≦…≦w N), the contours of the OWL graph will have the shape shown in Figure 1(a). In Figure 1(a), the contour line indicated by reference numeral 51 indicates the lowest position. The height of the contour line indicated by reference numeral 52, which is one position outside the contour line indicated by reference numeral 51, is higher than the height of the contour line indicated by reference numeral 51, and so on, with the height of the outer contour line being higher than the height of the inner contour line. Hereinafter, OWL with a weight vector w that is non-negative and non-decreasing will be referred to as pointed star norm (hereinafter also referred to as PSN (Pointed Star Norm)) based on the shape shown in Figure 1(a).

[0035] The cusp norm is a non-convex function, as can be seen from the shape shown in Figure 1(a). The cusp norm is a sparse regularization function that has stronger sparse regularization properties than the L1 norm, which has the shape shown in Figure 1(b) and is most commonly used in sparse regularization. The L p There is a norm called the pseudo norm, but the cusp norm is L p It can be said to be an approximation function of the pseudo-norm. Note that in Figures 1(b) and (c), the height of the outer contour line is higher than the height of the inner contour line, just like in Figure 1(a).

[0036]

number

[0037] By using the cusp norm, the cusp gradient imposes a small weight regularization on pixels with large gradients. Therefore, the cusp norm can use the weight vector w expressed by the following equation (10) that introduces two control parameters, λ and α.

[0038]

number

[0039] In equation (10), vector 0 Nis a vector of length N whose elements are all "0", and vector 1 N is a vector of length N in which all elements are "1". The first α elements are 0, and the following N-α elements are λ vectors. Parts with large adjacent differences have a multiplier of 0 and are not included in the optimization target, while parts with small differences have a multiplier of λ, making them smoother. This means that the smoothness of the gradation can be adjusted by the control parameter λ of the weight vector w, and the steepness of the edges can be adjusted by the control parameter α. The control parameter λ is a real number equal to or greater than 0, and the larger the parameter, the smoother the gradation can be adjusted. The control parameter α is an integer equal to or greater than 0, and it is possible to control the number of pixels with steep edges.

[0040] In addition, in the weight vector of equation (10), if α = 0, the weight vector w becomes vector 1 N multiplied by λ (w1=w2=…=w N = λ), in which case the optimization problem of equation (3) using the cusp norm is equivalent to the L1 regularization problem formulated in the case of TV.

[0041] In contrast, (w1=w2=…=w α =0,w α+1 =w α+2 =…=w N = λ), as λ → ∞, the optimization problem of equation (3) using the cusp norm approaches the L0 pseudo-norm constraint problem formulated in the case of L0 gradient projection. Thus, the cusp norm has the properties of a sparse regularization function that is stronger than the L1 norm and milder than the L0 pseudo-norm.

[0042] The control parameter λ, which adjusts the smoothness of the gradation, must be adjusted while observing the output image of the output image signal vector x displayed on a screen. In contrast, the control parameter α, which adjusts the steepness of the edges, is set to an integer that is approximately a few percent of the number of pixels N.

[0043] As described above, since the cusp norm is a non-convex function, it is known that finding a solution to the optimization problem of formula (3) generally requires a very long calculation time. In order to solve this problem of calculation time, the present embodiment uses the Alternating Direction Method of Multipliers (hereinafter referred to as "ADMM" (Alternating Direction Method of Multipliers)) shown in the following Reference 2, which is one of the algorithms included in the proximity separation method.

[0044] “Reference 2: D. Gabay and B. Mercier, “A dual algorithm for the solution of nonlinear variational problems via finite element approximation”, Computers and Mathematics with Applications, Vol.2 pp17-40, Pergamon Press, 1976”

[0045] To use ADMM, it is necessary to transform equation (3) into a form that can be applied to ADMM. For this purpose, we use GPSL (Group Pointed Star L 1,2 -norm) function is defined as the following equation (11).

[0046]

number

[0047] In equation (11), vector y is a vector defined by the following equation (12).

[0048]

number

[0049] In equation (11), the symbol shown in the following equation (13) is a symbol indicating a group for each pixel of n=1,...,N. Therefore, the function Ωw The (·) argument means that the vector y is divided into pixels.

[0050]

number

[0051] By using the GPSL function in equation (11), the function PSG w (·) (where the subscript w is the weight vector w) can be transformed as shown in the following equation (14).

[0052]

number

[0053] In equation (14), matrix D is a matrix defined by the following equation (15), which is a discrete difference operator matrix having a cyclic boundary condition. In equation (14), matrix B is a matrix defined by the following equation (16), which is a 0-1 binary diagonal matrix that sets the cyclic boundary to 0 (for example, see Non-Patent Document 3).

[0054]

number

[0055]

number

[0056] By the transformation shown in equation (14), the optimization problem expressed by equation (3) can be transformed into the following equation (17) which can utilize the ADMM.

[0057]

number

[0058] By using the ADMM, a local optimum solution to the optimization problem of equation (17) can be obtained based on the update equations shown in the following equations (18) to (20). Note that in equations (18) and (19), γ is a step size, and γ>0.

[0059]

number

[0060]

number

[0061]

number

[0062] In the above equations (18) to (20), the vector x (k+1) The update equation (18) and the vector y (k+1) Each of the update equations (19) is an optimization problem in itself. Therefore, the solution method for each equation is explained below.

[0063] Vector x (k+1) As shown in Non-Patent Document 3, the solution for is the following equation (21):

[0064]

number

[0065] In equation (21), the function F(·) is the two-dimensional discrete Fourier transform, and the function F * (·) is the two-dimensional discrete inverse Fourier transform. The matrix Λ is a diagonal matrix whose diagonal elements are the Fourier transform results of the Laplacian filter kernel. Therefore, F * The matrix in the following equation (22) included in the argument of (·) is a diagonal matrix, and the inverse matrix of equation (22) can be calculated by element-by-element division.

[0066]

number

[0067] Next, the vector y (k+1) We will explain the solution of the above. First, define the following equation (23).

[0068]

number

[0069] Using equation (23), vector y (k+1) The solution to this equation is given by the following equation (24).

[0070]

number

[0071] The calculation of the following equation (25), which is the first term on the right side of equation (24), can be expressed as the following equation (27) after substituting the following equation (26).

[0072]

number

[0073]

number

[0074]

number

[0075] In equation (27), n = 1, ..., N. In equation (27), (p1 (k) ,…,p N (k) ) is expressed as the following equation (28).

[0076]

number

[0077] The argument of the function on the right side of equation (28) is expressed as the vector v (k) Let us assume that.

[0078]

number

[0079] The right-hand side of equation (28) is defined as the following equation (30), and by calculating the right-hand side of equation (30), (p1 (k) ,…,p N (k) ) can be calculated.

[0080]

number

[0081] Here, since the relationship shown in the following equation (31) exists, the operation using the sign function sgn(·) on the right-hand side of equation (30) can be omitted.

[0082]

number

[0083] The function shown in the following equation (32) included on the right side of equation (30) indicates that a projection onto a non-increasing monotonic cone expressed by the following equation (33) is performed.

[0084]

number

[0085]

number

[0086] The calculation of equation (32) is known as a problem called Isotonic Regression. However, because of the relationship shown in the following equation (34), the projection by equation (33) becomes an identity mapping and can be omitted.

[0087]

number

[0088] As a result, equation (30) can be transformed into the following equation (35).

[0089]

number

[0090] In equation (35), the function (·) + is a ramp function that clips its arguments to non-negative values. (k) is the vector v (k) is a permutation matrix of vectors generated by sorting the elements of a vector whose elements are the absolute values ​​of the elements of in descending order, and has the relationship of the following equation (36).

[0091]

number

[0092] As a result, the optimization problem shown by equation (17) can be solved by repeatedly executing equation (21), equation (24) to which equations (27) and (35) are applied, and the update equation of equation (20). Note that the step size γ is updated as γ←ηγ (where 0<η<1) for each iteration. That is, the new step size γ is updated to a value obtained by multiplying the previous step size γ by the attenuation rate η.

[0093] (Configuration of signal processing device) 2 is a block diagram showing the internal configuration of a signal processing device 1 that calculates a solution to the optimization problem of equation (17). The signal processing device 1 includes a control unit 2, which includes a first variable updating unit 10, a second variable updating unit 20, a third variable updating unit 30, an update determination unit 40, and a step size updating unit 50.

[0094] The update determination unit 40 stores the value of "k" (hereinafter referred to as "step k") indicating the number of times the update process is performed in an internal storage area. The update determination unit 40 outputs an instruction signal to update the step size to the step size update unit 50 at the timing when the value of step k is increased. The update determination unit 40 determines whether or not to continue the process of updating the first variable, vector x, the second variable, vector y, and the third variable, vector d, in accordance with a predetermined termination condition. Here, the termination condition is, for example, that the value of step k at that time coincides with a predetermined upper limit value for the number of updates.

[0095] When it is determined that the update process is to be continued, the update determination unit 40 updates the vector y (k) to the first variable update unit 10, and the third variable update unit 30 outputs the vector d (k) to the first variable update unit 10, the second variable update unit 20, and the third variable update unit 30. On the other hand, when the update determination unit 40 determines not to continue the update process, it outputs the vector x (k+1) is output to the outside as an output image signal vector x, and the process ends.

[0096] The step size update unit 50 takes in a step size γ and a decay rate η that are given from the outside. The step size update unit 50 writes and stores the taken-in step size γ and the decay rate η in an internal storage area. After writing the step size γ in the internal storage area, the step size update unit 50 outputs the written step size γ to the first variable update unit 10 and the second variable update unit 20. When the step size update unit 50 receives an instruction signal to update the step size from the update determination unit 40, it performs a process of updating the step size γ stored in the internal storage area based on the decay rate η, and outputs the updated step size γ to the first variable update unit 10 and the second variable update unit 20.

[0097] The first variable update unit 10 updates the input image signal vector f given from the outside, the step size γ, and the vector y (k) and vector d (k) The first variable update unit 10 performs the calculation of equation (21) to update the first variable, vector x (k+1) Calculate.

[0098] 3 is a block diagram showing the internal configuration of first variable updating unit 10. First variable updating unit 10 includes a subtractor 101, a differential transposition calculation unit 102, an adder 103, a multiplier 104, an FFT (Fast Fourier Transform) unit 105, an inverse calculation unit 106, a coefficient generation unit 107, a multiplier 108, and an inverse FFT unit 109.

[0099] The subtractor 101 subtracts the vector y (k) The vector d calculated by the third variable update unit 30 is (k) The subtraction value shown in the following equation (37) is calculated by subtracting

[0100]

number

[0101] The differential transposition calculation unit 102 stores a matrix D in an internal storage area in advance, and calculates a transposed matrix of the matrix D stored in the internal storage area,T The differential transposition calculation unit 102 generates the generated matrix D T and the subtraction value calculated by subtractor 101 to calculate the multiplied value shown in the following equation (38).

[0102]

number

[0103] The reciprocal calculation unit 106 receives the step size γ and calculates the reciprocal of the step size γ, i.e., γ -1 The multiplier 104 multiplies the multiplied value calculated by the differential transposition calculation unit 102 and the γ -1 The adder 103 adds the input image signal vector f and the multiplied value calculated by the multiplier 104 to calculate the sum shown in equation (39).

[0104]

number

[0105] The FFT unit 105 performs a two-dimensional fast Fourier transform on the sum calculated by the adder 103. The coefficient generation unit 107 stores the matrix I and the matrix Λ in advance in an internal storage area, and calculates the matrix I and the matrix Λ stored in the internal storage area and the reciprocal calculation unit 106 calculated γ -1 Based on this, the matrix of equation (22) is generated.

[0106] Multiplier 108 multiplies the output of FFT section 105 by the matrix generated by coefficient generating section 107 to calculate a multiplied value shown in the following equation (40).

[0107]

number

[0108] The inverse FFT unit 109 performs a two-dimensional inverse fast Fourier transform on the multiplied value calculated by the multiplier 108, i.e., performs the calculation of equation (21) to obtain the first variable, vector x (k+1)Calculate.

[0109] The second variable update unit 20 updates the vector x (k+1) and the third variable vector d (k) The second variable update unit 20 performs the calculation of the equation (24) to which the equations (27) and (35) are applied to update the second variable vector y (k+1) Calculate.

[0110] 4 is a block diagram showing the internal configuration of second variable updating unit 20. Second variable updating unit 20 includes a difference calculation unit 201, an adder 202, a binary mask unit 203, an L2 norm calculation unit 204, a descending order sorting unit 205, a permutation matrix generation unit 206, a multiplier 207, a subtractor 208, a ramp function unit 209, an inverse permutation unit 210, a coefficient generation unit 211, an element product calculation unit 212, a binary mask unit 213, a binary mask unit 214 and an adder 215.

[0111] The difference calculation unit 201 stores the matrix D in advance in an internal storage area, and calculates the matrix D stored in the internal storage area and the vector x calculated by the first variable update unit 10. (k+1) The adder 202 multiplies the multiplied value calculated by the difference calculation unit 201 and the vector d (k) and calculates the sum shown in equation (23). Binary mask unit 203 pre-stores matrix B in an internal storage area, and multiplies matrix B stored in the internal storage area by the sum calculated by adder 202 to calculate the multiplied value shown in equation (26).

[0112] The L2 norm calculation unit 204 is a calculation unit that calculates the L2 norm for each pixel, and calculates the vector v (k) The descending order sorting unit 205 calculates the vector v (k) The elements of the vector whose elements are the absolute values ​​of the elements in(k) Generate.

[0113]

number

[0114] The permutation matrix generation unit 206 converts the vector v (k) from vector a (k) The matrix P is a permutation matrix of (k) The multiplier 207 multiplies the weight vector w by the step size γ to calculate the multiplied value shown in the following equation (42).

[0115]

number

[0116] The subtractor 208 subtracts the vector a generated by the descending order sorting unit 205. (k) The ramp function unit 209 performs a calculation to clip the subtraction value calculated by the subtractor 208 to a non-negative value, and calculates the calculation result shown in the following equation (43).

[0117]

number

[0118] The inverse permutation unit 210 inversely permutes the matrix P generated by the permutation matrix generation unit 206. (k) The matrix P is the transpose of (k)T The inverse permutation unit 210 generates the generated matrix P (k)T and the output of the ramp function unit 209 to obtain a vector p (k) Calculate.

[0119]

number

[0120] The coefficient generation unit 211 calculates the vector v (k) and the vector p calculated by the inverse substitution unit 210 (k) Based on this, the coefficient q n (k) Calculate.

[0121]

number

[0122] In equation (45), n=1,...,N. The element product calculation unit 212 multiplies the multiplied value shown in equation (26) calculated by the binary mask unit 203 by the coefficient q n (k) Based on this, the vector z (k) Calculate.

[0123]

number

[0124] where n = 1, ..., N in equation (46), and the vector z (k) The elements of are expressed as follows:

[0125]

number

[0126] The binary mask unit 213 stores the matrix B in advance in an internal storage area, and multiplies the matrix B stored in the internal storage area by the vector z (k)and calculates a multiplied value. Binary mask unit 214 pre-stores in an internal storage area a matrix (IB) obtained by subtracting matrix B from matrix I, and multiplies the matrix (IB) stored in the internal storage area by the sum calculated by adder 202 to calculate a multiplied value. Adder 215 adds the output of binary mask unit 213 and the multiplied value calculated by binary mask unit 214 to calculate a vector y (k+1) Calculate.

[0127]

number

[0128] The third variable update unit 30 updates the vector x (k+1) and the vector y (k+1) and the vector d calculated by the third variable update unit 30 one step before (k) The third variable update unit 30 performs the calculation of the formula (20) to obtain the third variable, vector d (k+1) Calculate.

[0129] 5 is a block diagram showing the internal configuration of the third variable updating unit 30. The third variable updating unit 30 includes a difference calculation unit 301, an adder 302, and a subtractor 303. The difference calculation unit 301 stores a matrix D in advance in an internal storage area, and calculates the matrix D stored in the internal storage area and the vector x calculated by the first variable updating unit 10. (k+1) The adder 302 multiplies the vector d (k) The subtractor 303 calculates an additional value by adding the multiplied value calculated by the adder 302 to the vector y (k+1) to obtain the third variable, vector d (k+1) Calculate.

[0130] (Processing by the signal processing unit) 6 is a flowchart showing the flow of processing by the signal processing unit 1. The first variable updating unit 10 and the update determining unit 40 of the signal processing unit 1 take in an input image signal vector f given from the outside. The second variable updating unit 20 takes in a weighting vector w having predetermined control parameters λ and α given from the outside. The step size updating unit 50 takes in a step size γ and an attenuation rate η given from the outside (step S1).

[0131] The update determination unit 40 multiplies the matrix D stored in advance in an internal storage area by the captured input image signal vector f to calculate a multiplied value shown in the following equation (49).

[0132]

number

[0133] The update determination unit 40 calculates the multiplied value as the vector y (0) and the initial value of the third variable, vector d (0) The update determination unit 40 initializes the step k to "0" (step S2).

[0134] The update determination unit 40 determines whether or not the termination condition is satisfied (step S3). The determination of whether or not the termination condition is satisfied is made based on whether or not the value of step k at that point in time matches a predetermined upper limit of the number of updates. Here, the explanation will proceed assuming that the value of step k=0 does not match the predetermined upper limit of the number of updates. The update determination unit 40 determines that the termination condition is not satisfied because the value of step k does not match the predetermined upper limit of the number of updates (step S3, No), and resets the vector y (0) to the first variable update unit 10, and the vector d (0) to the first variable update unit 10, the second variable update unit 20, and the third variable update unit 30.

[0135] In the following description, for the sake of generalization, the value of the number of steps stored in the internal storage area of ​​the update determination unit 40 is not limited to "k=0", and the subscript of each variable is indicated as (k) or (k+1). In the first variable update unit 10, a subtractor 101 subtracts the vector y (k) and vector d (k) The subtractor 101 takes in the vector y (k) from vector d (k) The subtractor 101 outputs the calculated subtraction value to the differential transposition calculation unit 102.

[0136] The differential transposition calculation unit 102 calculates a matrix D, which is a transposed matrix of the matrix D stored in the internal storage area. T The differential transposition calculation unit 102 generates the generated matrix D T The differential transpose calculation unit 102 multiplies the subtraction value calculated by the subtractor 101 by the subtraction value calculated by the subtractor 101 to calculate the multiplication value shown in Equation (38).

[0137] The reciprocal calculation unit 106 takes in an externally applied step size γ in parallel with the processing performed by the adder 101 and the differential transpose calculation unit 102. The reciprocal calculation unit 106 calculates γ, which is the reciprocal of the taken-in step size γ. -1 The reciprocal calculation unit 106 calculates the calculated γ -1 to the multiplier 104 and the coefficient generating unit 107. The multiplier 104 multiplies the multiplied value of equation (38) output by the differential transposition calculation unit 102 and γ -1 and multiplying the input image signal vector f by the multiplier 104 to calculate a multiplied value, and outputting the calculated multiplied value to the adder 103. The adder 103 takes in an input image signal vector f provided from the outside. The adder 103 adds the taken in input image signal vector f to the multiplied value output by the multiplier 104 to calculate an added value shown in equation (39). The adder 103 outputs the calculated added value to the FFT unit 105.

[0138] The FFT unit 105 performs a two-dimensional fast Fourier transform on the sum of equation (39) output by the adder 103, and outputs the calculation result to the multiplier 108. In parallel with the processes performed by the multiplier 104, the adder 103, and the FFT unit 105, the coefficient generation unit 107 multiplies the γ -1 and the matrix I and matrix Λ stored in the internal storage area, the coefficient generation unit 107 generates the matrix of equation (22).

[0139] Multiplier 108 multiplies the calculation result output by FFT unit 105 by the matrix of equation (22) output by coefficient generation unit 107 to calculate a multiplied value shown in equation (40). Multiplier 108 outputs the calculated multiplied value to inverse FFT unit 109.

[0140] The inverse FFT unit 109 performs a two-dimensional inverse fast Fourier transform shown in equation (21) on the multiplied value of equation (40) output by the multiplier 108 to obtain the vector of the first variable (k+1) The inverse FFT unit 109 calculates the calculated vector (k+1) to the second variable update section 20, the third variable update section 30, and the update determination section 40 (step S5).

[0141] In the second variable update unit 20, the difference calculation unit 201 calculates the matrix D stored in the internal storage area and the vector x (k+1) The difference calculation unit 201 multiplies the vector d (k) and the multiplied value output by difference calculation section 201 to calculate the sum shown in equation (23). Adder 202 outputs the calculated sum to binary mask section 203 and binary mask section 214 (step S6).

[0142] The binary mask unit 203 multiplies the matrix B stored in the internal storage area by the sum output by the adder 202 to calculate the multiplied value shown in equation (26). The binary mask unit 203 outputs the calculated multiplied value to the L2 norm calculation unit 204 and the element product calculation unit 212 (step S7). The L2 norm calculation unit 204 calculates the vector v (k) The L2 norm calculation unit 204 calculates the calculated vector v (k) to the permutation matrix generation unit 206, the descending order sorting unit 205, and the coefficient generation unit 211. The descending order sorting unit 205 outputs the vector v (k) The elements of the vector whose elements are the absolute values ​​of the elements in (k) The descending sort unit 205 generates the generated vector a (k) is output to the subtractor 208 (step S8).

[0143] Multiplier 207 takes in weight vector w and step size γ, which are provided from the outside, in parallel with the processes of steps S6 to S8. Multiplier 207 multiplies weight vector w by step size γ to calculate a multiplied value shown in equation (42). Multiplier 207 outputs the calculated multiplied value to subtractor 208 (step S9).

[0144] The subtractor 208 subtracts the vector a output from the descending order sorting unit 205. (k) The subtractor 208 calculates a subtraction value by subtracting the multiplied value calculated by the multiplier 207 from the subtraction value. The subtractor 208 outputs the calculated subtraction value to the ramp function unit 209. The ramp function unit 209 performs the operation shown in equation (43) on the subtraction value output by the subtractor 208 to clip it to a non-negative value, and outputs the operation result to the inverse substitution unit 210 (step S10).

[0145] In parallel with the process of step S10, the permutation matrix generation unit 206 converts the vector v (k) from vector a (k) The matrix P is a permutation matrix of (k)The permutation matrix generation unit 206 generates the generated matrix P (k) to the inverse substitution unit 210 (step S11).

[0146] The inverse permutation unit 210 converts the matrix P (k) The matrix P is the transpose of (k)T The inverse permutation unit 210 generates the generated matrix P (k)T The vector p (k) The inverse substitution unit 210 calculates the calculated vector p (k) to the coefficient generating unit 211 (step S12).

[0147] The coefficient generation unit 211 calculates the vector v (k) and the vector p (k) Based on this, the coefficient q shown in equation (45) n (k) The coefficient generating unit 211 calculates the calculated coefficient q n (k) to the element product calculation unit 212 (step S13).

[0148] The element product calculation unit 212 multiplies the multiplied value shown by the equation (26) output by the binary mask unit 203 by the coefficient q n (k) Based on this, the calculation shown in equation (47) is performed to obtain the vector z (k) The element product calculation unit 212 calculates the calculated vector z (k) to the binary mask unit 213 (step S14).

[0149] In parallel with the processing of steps S7 to S14, the binary mask unit 214 multiplies the matrix (IB) stored in the internal storage area by the sum calculated by the adder 202 and shown in equation (23) to calculate a multiplied value. The binary mask unit 214 outputs the calculated multiplied value to the adder 215. In parallel with the processing performed by the binary mask unit 214, the binary mask unit 213 multiplies the matrix B stored in the internal storage area by the vector z (k) The binary mask unit 213 multiplies the multiplied value by the adder 215. The binary mask unit 213 outputs the multiplied value to the adder 215.

[0150] The adder 215 performs the operation shown in equation (48) to add the multiplied value output by the binary mask unit 213 and the multiplied value output by the binary mask unit 214, and calculates the second variable vector y (k+1) The adder 215 calculates the calculated vector y (k+1) to the third variable update unit 30 and the update determination unit 40 (step S15).

[0151] In the third variable update unit 30, the difference calculation unit 301 calculates the matrix D stored in the internal storage area and the vector x (k+1) The difference calculation unit 301 multiplies the vector d (k) The adder 302 adds the multiplied value output by the difference calculation unit 301 to calculate an additional value. The adder 302 outputs the calculated additional value to the subtractor 303. The subtractor 303 subtracts the vector y (k+1) to obtain the third variable, vector d (k+1) The subtractor 303 calculates the calculated vector d (k+1) to the update determination unit 40 (step S16).

[0152] The update determination unit 40 determines the vector x (k+1) and the vector y (k+1)and the vector d output by the third variable update unit 30 (k+1) When the step size update unit 50 receives the step size update instruction signal from the update determination unit 40, the step size update unit 50 multiplies the step size γ stored in the internal storage area by the attenuation rate η to calculate ηγ. The step size update unit 50 rewrites the step size γ stored in the internal storage area to ηγ to set the calculated ηγ as the new step size γ. When the step size update unit 50 rewrites the step size γ in the internal storage area to ηγ, the step size update unit 50 outputs the rewritten step size γ to the first variable update unit 10 and the second variable update unit 20 (step S17).

[0153] The update determination unit 40 performs the determination process of step S3 again. When the update determination unit 40 determines that the value of the step k at that time coincides with the upper limit value of the predetermined number of updates and thus satisfies the termination condition (step S3, Yes), it determines that the vector x (k+1) is outputted to the outside as an output image signal vector x (step S4), and the process ends.

[0154] On the other hand, when the update determination unit 40 determines that the value of the step k at that time does not match the predetermined upper limit value of the number of updates and does not satisfy the termination condition (step S3, No), the update determination unit 40 determines that the value of the step k at that time does not match the predetermined upper limit value of the number of updates and does not satisfy the termination condition (step S3, No), (k+1) to the first variable update unit 10, and the vector d (k+1) to the first variable update section 10, the second variable update section 20, and the third variable update section 30, and the processing from step S5 onwards is carried out again.

[0155] (Implementation results) 7 and 8 are diagrams showing the results of edge-preserving smoothing performed using the above-described signal processing device 1. Fig. 7 shows the results when the control parameter λ for adjusting the smoothness of the gradation is set to 50, and Fig. 8 shows the results when the control parameter λ is set to 200. Figs. 7(a) and 8(a) show the results when the control parameter α for adjusting the steepness of the edges is set to about 8% of the number of pixels N, and Figs. 7(b) and 8(b) show the results when the control parameter α is set to about 4% of the number of pixels N.

[0156] In Fig. 7 and Fig. 8, the images indicated by the reference characters 60a, 60b, 70a, and 70b are images obtained by displaying the output image signal obtained after performing edge-preserving smoothing processing under each condition. In Fig. 7 and Fig. 8, the graphs indicated by the reference characters 62a, 62b, 72a, and 72b are graphs obtained by plotting the values ​​of the luminance signals of the line indicated by the reference character 61a in the image 60a, the line indicated by the reference character 61b in the image 60b, the line indicated by the reference character 71a in the image 70a, and the line indicated by the reference character 71b in the image 70b. The solid lines in the graphs indicated by the reference characters 62a, 62b, 72a, and 72b indicate the values ​​of the luminance signals of the input image signal, and the dotted lines indicate the values ​​of the luminance signals of the output image signal. As can be seen by comparing Fig. 7 and Fig. 8, the smoothing is performed such that the smoothness of the gradation is lost as the control parameter λ increases. In contrast, as can be seen by comparing Figures 7(a) and 7(b), or Figures 8(a) and 8(b), the smaller the control parameter α, the more the smoothing that is performed reduces the steepness of the edges.

[0157] The signal processing device 1 according to the above embodiment applies a sparse regularization function of a non-convex function to an input image signal, which changes the smoothness of the gradation by adjusting a control parameter λ, which is a first control parameter included in the weight of the non-convex function, and changes the steepness of the edge by adjusting a control parameter α, which is a second control parameter included in the weight of the non-convex function, to calculate a solution to an optimization problem by global optimization, thereby generating an edge-preserving smoothed output image signal from the input image signal. This makes it possible to independently adjust both the smoothness of the gradation and the steepness of the edge in edge-preserving smoothing by global optimization.

[0158] The computational complexity of the flowchart shown in Fig. 6 is the O(NlogN) computational complexity required for solving linear equations using two-dimensional fast Fourier transform and two-dimensional inverse fast Fourier transform, the O(NlogN) computational complexity required for sorting algorithm, and the O(N) computational complexity required for arithmetic operations, and the computational complexity for one iteration is O(NlogN). Therefore, the signal processing device 1 of this embodiment performs a process of solving an optimization problem including a cusp norm of a non-convex function, but the computational complexity can be suppressed to O(NlogN), and edge-preserving smoothing by global optimization with sparse regularization performance that is stronger than the L1 norm and milder than the L0 pseudo-norm and has the same robustness as the L1 norm is possible. Here, having the same robustness as the L1 norm means that, compared to the L0 norm, even when each element of the input takes a small value, it does not react too sensitively and has the performance of appropriate sparse regularization.

[0159] In the above embodiment, the update determination unit 40 ends the determination process in step S3 in FIG. 6 when the value of step k at that point in time matches the upper limit of the number of updates that is determined in advance. (k) The condition for ending the determination process in step S3 in FIG. 6 may be that the norm of the amount of change in is equal to or less than a predetermined value.

[0160]

number

[0161] The following equation (51) may be used as an objective function, and the end condition for the determination process in step S3 in FIG. 6 may be that the amount of change in the objective function calculated by the following equation (52) is equal to or smaller than a predetermined value.

[0162]

number

[0163]

number

[0164] In the above embodiment, the FFT unit 105 may perform a two-dimensional discrete Fourier transform instead of the two-dimensional fast Fourier transform, and the inverse FFT unit 109 may perform a two-dimensional inverse discrete Fourier transform instead of the two-dimensional inverse fast Fourier transform.

[0165] In the above embodiment, the image signal to be processed is a color image signal having three channels, RGB. However, the image signal to be processed may be a color image signal having four channels, CYMK, or a monochrome image signal having one channel. In this case, for an RGB color image signal, N c = 3, for CMYK color image signals, N c = 4, for monochrome image signals, N c = 1 for variable N c After defining the above, the following formulas are c The subscript "3N" in equation (1) and equation (2) is changed to "NN c " The subscript "N v ×N h ×3" is "N v ×N h×N c " The "3" above Σ in the formula (7) that indicates the range of the symbol Σ is "N c The subscript "6N" in equation (12) stands for "2NN c The subscript "6N×3N" in equation (15) becomes "2NN c ×NN c The subscript "6N×6N" in equation (16) stands for "2NN c ×2NN c The definition of the formula (18) under argmin is the same as formula (2), and the subscript "3N" is replaced with "NN c The definition of the same formula as formula (12) under argmin in formula (19) is the same as formula (12), in which the subscript "6N" is replaced by "2NN c "

[0166] In the above embodiment, the signal processing device 1 performs edge-preserving smoothing on an image signal, i.e., a two-dimensional signal, but the signal processing device 1 may also process a one-dimensional signal, such as a communication signal, and in that case, the same effect can be obtained.

[0167] The signal processing device 1 in the above-mentioned embodiment may be realized by a computer. In that case, a program for realizing this function may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the function. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into a computer system. The term "computer-readable recording medium" may also include a medium that dynamically holds a program for a short period of time, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in that case. The above-mentioned program may be a program for realizing a part of the above-mentioned function, or may be a program that can realize the above-mentioned function in combination with a program already recorded in the computer system, or may be a program that is realized using a programmable logic device such as an FPGA (Field Programmable Gate Array).

[0168] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included. [Industrial Applicability]

[0169] It can be applied to a technique for edge-preserving smoothing by global optimization. [Explanation of symbols]

[0170] REFERENCE SIGNS LIST 1... signal processing device, 2... control unit, 10... first variable update unit, 20... second variable update unit, 30... third variable update unit, 40... update determination unit, 50... step size update unit

Claims

1. A control unit that applies a sparse regularization function of a non-convex function to an input signal, the sparse regularization function being a non-convex function, to change the smoothness of a gradation by adjusting a first control parameter included in a weight of the non-convex function, and that changes the steepness of an edge by adjusting a second control parameter included in a weight of the non-convex function, thereby calculating a solution to an optimization problem by global optimization, and generates an edge-preserving smoothed output signal from the input signal. A signal processing device comprising:

2. the input signal and the output signal are image signals; The signal processing device according to claim 1 .

3. The sparse regularization function for the non-convex function is a star-shaped norm defined by the following equation (1) with ordered weights L 1 The elements of the norm weight vector w are non-negative, non-decreasing (0≦w1≦w2...≦wN) star norms. [0010] 3. A signal processing device according to claim 1 or 2.

4. The weight vector w is defined by the following equation (2): [0025] The first control parameter is λ in equation (2), The second control parameter is α in equation (2). The signal processing device according to claim 3 .

5. The optimization problem is Using the GPSL function defined by the following equation (3): [0030] It is defined as the following formula (4): [0045] The control unit calculates a solution to the optimization problem using an alternating direction multiplication method.

5. A signal processing device according to claim 3 or 4.

6. A signal processing method performed by a signal processing device, comprising: A sparse regularization function of a non-convex function is applied to an input signal, and a first control parameter included in the weight of the non-convex function is adjusted to change the smoothness of a gradation, and a second control parameter included in the weight of the non-convex function is adjusted to change the steepness of an edge. A solution to an optimization problem by global optimization is calculated by applying the sparse regularization function of a non-convex function to the input signal, thereby generating an edge-preserving smoothed output signal from the input signal. Signal processing methods.

7. A signal processing program for causing a computer to execute the signal processing device according to any one of claims 1 to 5.

Citation Information

Patent Citations

  • Image processing device, and image processing method

    JP2011049696A

  • Gradation restoration apparatus and program of the same

    JP2013235403A

  • Medical image processor and medical image diagnostic apparatus

    JP2018015572A

  • Magnetic resonance imaging apparatus and quantitative magnetic susceptibility mapping method

    WO2016076076A1