Image processing method based on fractional calculus deep neural network convolution layer

Abstract

The application provides a fractional calculus deep neural network convolutional layer image processing method, which comprises the following steps: obtaining an image to be processed, and performing convolutional processing, wherein the convolutional layer comprises c_in input channels and c_out output channels, each output channel comprises c_in convolutional kernels, each convolutional kernel is defined by the following method: generating a matrix A with c_in rows and c_out columns, taking the opposite number of all elements in the matrix A in a way of intervaling one column to obtain an order matrix A' for generating the convolutional kernel, and generating the convolutional kernel according to each element in the order matrix A' according to a fractional calculus calculation formula. The fractional convolutional layer output constructed by the application contains more levels of high-frequency and low-frequency information of the input image in the feature map, and is more conducive to representing the difference of the image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image processing method based on fractional calculus deep neural network convolutional layers. Background Technology

[0002] Currently, information science is facing profound changes and rapid development, with signal analysis and information processing being prime examples. Many new ideas, methods, and technologies are constantly emerging, while traditional methods and techniques are also being continuously expanded. For nearly three centuries, fractional calculus, an important branch of mathematics, has gradually become systematic, but it remains largely unknown to scholars in the engineering field. It wasn't until Mandelbrot proposed the fractal theory and applied Riemann-Liouville fractional calculus to analyze and study Brownian motion in fractal media that fractional calculus was applied to engineering calculations in many disciplines, particularly gaining widespread attention and application in chemistry, electromagnetism, control science, materials science, and mechanics.

[0003] In signal processing theory, traditional calculus is a fundamental mathematical operation widely used in information analysis and processing. Calculus is a powerful tool for describing Euclid space, playing a crucial role, especially in the detection and extraction of signal singularities. It is also an indispensable mathematical tool in areas such as image edge extraction, power fault detection, electroanalytical chemical processing, and medical image diagnosis.

[0004] Commonly used calculus operations and differential equations are of integer order, such as first-order and second-order differentials. However, with the development of computing power and information technology, more and more nonlinear problems have become the focus of research, such as chaos and fractal phenomena. These are common phenomena in engineering. Many traditional methods have been studied and explained to some extent, but they are far from sufficient and cannot adequately explain the causes of these phenomena. Therefore, it is necessary to expand existing methods and research tools to explore new properties of these problems.

[0005] Further research has shown that fractional calculus is a powerful tool for describing fractional-dimensional spaces. Traditional methods often fall short for many complex phenomena, while the introduction of fractional calculus can lead to new discoveries and conclusions.

[0006] For mechanics and building science, it has been proven that the model constructed using fractional calculus is currently the best method to describe the entire process of creep compliance and relaxation modulus.

[0007] In digital images, since both high-frequency and low-frequency information are inherent characteristics of the image itself, extracting these two information is a crucial task in digital image processing. Existing technologies utilize filters designed using fractional calculus for image processing; however, these filters cannot simultaneously extract both high-frequency and low-frequency information, and they cannot automatically adjust the fractional order during calculation, making it difficult to represent the differences within the image. Summary of the Invention

[0008] To address the technical problem in existing technologies where automatic adjustment and simultaneous extraction of high-frequency and low-frequency information in images are impossible, making it difficult to represent the differences in images, one objective of this invention is to provide an image processing method based on a fractional-order calculus deep neural network convolutional layer. The image processing method includes:

[0009] Obtain the image to be processed.

[0010] The acquired image is processed by convolutional layers of a deep neural network, wherein...

[0011] The convolutional layer includes c_in input channels and c_out output channels, with each output channel including c_in convolutional kernels.

[0012] Each convolutional kernel is defined as follows:

[0013] Set up a parameter matrix A with c_in rows × c_out columns. All elements in matrix A are real numbers between 0 and 1 randomly generated by the system during network initialization. Take the negative of all elements in matrix A with each column skipped to obtain a fractional-order matrix A' used to generate the convolution kernel.

[0014] The convolution kernel is generated by calculating the elements of the convolution kernel matrix using each element in the order matrix A' according to the fractional calculus formula.

[0015] Furthermore, all elements of the convolution kernel matrix are calculated using the following method:

[0016]

[0017] Among them, M v Let i be the convolution kernel matrix, and j be the coordinates of the elements of the convolution kernel matrix.

[0018]

[0019] Where v is an element in the order matrix A'.

[0020] Furthermore, the size of the convolution kernel is m_size×m_size.

[0021] This invention provides an image processing method based on fractional-order calculus deep neural network convolutional layers, which reduces the number of model parameters. Compared to the number of parameters in a typical convolutional layer, which is c_in×c_out×m_size×m_size, where c_in is the number of input channels, c_out is the number of output channels, and m_size represents the size of the convolutional kernel, the number of parameters in the fractional-order convolutional layer of this invention is only c_in×c_out.

[0022] This invention provides an image processing method based on fractional-order calculus deep neural network convolutional layers. It has a wide feature bandwidth, extracts high-frequency information by convolutional kernels derived from positive fractional-order v, and extracts low-frequency information by convolutional kernels derived from negative fractional-order v. The feature map output by the constructed fractional-order convolutional layer contains more layers of high-frequency and low-frequency information of the input image, which is more conducive to representing the differences in the image.

[0023] This invention provides an image processing method based on fractional-order calculus deep neural network convolutional layers. The order of the constructed fractional-order convolutional layers is automatically adjusted as the model learns, and the convolutional layers can output arbitrary amplitude feature maps as needed. Attached Figure Description

[0024] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0025] Figure 1 The schematic diagram illustrates the structure of a deep neural network convolutional layer in one embodiment of the present invention.

[0026] Figure 2 A feature diagram of the negative order output channel output is shown in one embodiment of the present invention.

[0027] Figure 3 A feature diagram of the positive order output channel output is shown in one embodiment of the invention. Detailed Implementation

[0028] To make the above and other features and advantages of the present invention clearer, the invention will be further described below with reference to the accompanying drawings. It should be understood that the specific embodiments given herein are for the purpose of explanation to those skilled in the art and are exemplary only, not restrictive.

[0029] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] To address the problem that existing technologies cannot automatically adjust the extraction of high-frequency and low-frequency information in images, making it difficult to represent image differences, an image processing method based on fractional calculus deep neural network convolutional layers is provided according to an embodiment of the present invention.

[0031] To more clearly explain the content of this invention, the concept of fractional calculus will first be explained.

[0032] Definition of fractional calculus

[0033] The order of fractional calculus is a generalization from integer to fractional. Fractions refer to all non-integers, and its research foundation also started with the study of integer-order differentials.

[0034] The same applies to the Grumwald-Letnikov definition. The v-th derivative of GL is defined as:

[0035]

[0036] In equation (1) above, v belongs to the integer part and v∈R. The signal function s(t)∈[a, t](a<t, a∈R, t∈R), assume that the (n+1)th order continuous derivative of the signal function s(t) exists and n∈Z, where Z represents an integer.

[0037] When v > 0, n must take at least one value. In the formula, the superscript G denotes the definition of G_L, v represents the order, and the subscripts a and t represent the upper and lower bounds of the integral. a is the initial value of time t. When v takes a negative real value, the formula becomes an integral formula.

[0038] The Riemann-Liouville definition improves the GL definition by starting with the properties satisfied by fractional calculus, thus simplifying calculations. The differential formula of the RL definition is as follows:

[0039]

[0040] The fractional derivative defined by Riemann-Liouville involves first performing an (nv)-th order integral, followed by a v-th order derivative. The fractional integral defined by RL is:

[0041]

[0042] The Caputo definition is an improvement on another form of the Grumwald-Letnikov definition (1). Its ultimate goal is to make the Laplace transform easier to calculate, which is beneficial for solving fractional differential equations and easier to apply in engineering control.

[0043] The Caputo definition is another formal improvement on the Grumwald-Letnikov definition, and its fractional differential formula is defined as follows:

[0044]

[0045] When v is a positive integer or a negative real number, the Caputo definition and the Riemann-Liouville definition are equivalent, and the relationship between the two definitions is as follows:

[0046]

[0047] The primary challenge in applying fractional differential theory is to discretize it and transform it into a form that can be processed by a digital computer before it can be applied.

[0048] Fractional Differential Template Design

[0049] Since image information is a two-dimensional discrete signal, the primary problem to solve in applying fractional differential theory to image processing is the discretization of the definition formula of fractional differential. Currently, Taylor series expansion is mainly used for approximation. This method is commonly used in numerical computation and has good results. Similarly, the discretization of fractional calculus is also carried out using Taylor series expansion.

[0050] This invention provides an image processing method based on fractional calculus deep neural network convolutional layers, using Grumwald-Letnikov fractional derivatives to design fractional derivative templates.

[0051] As introduced above, we know that the Grumwald-Letnikov definition is derived from the definition of integer derivatives, extending the order v from integers to fractions.

[0052] Regarding equation (1) already introduced above, let f(x, y) represent the image signal, and simplify equation (1) to obtain:

[0053]

[0054] Where Gamma function If f(t) is continuous in the interval t∈[a, t], then dividing the continuous interval of the signal into n equal parts, when h=1, we can obtain:

[0055]

[0056] Therefore, the difference expression for the fractional-order differential Taylor series expansion of a univariate signal is derived as follows:

[0057]

[0058] The fractional integral of Grumwald-Letnikov is defined as:

[0059]

[0060] Similarly, the difference expression for the fractional integral of a univariate signal can be derived:

[0061]

[0062] Comparing fractional differential operators and fractional integral operators, when the order v takes a negative value, the fractional differential operator becomes a fractional integral operator.

[0063] The coefficients of the two types of operators are combined and uniformly represented as:

[0064]

[0065] When the order v takes a positive value, the above formula corresponds to the coefficients of the fractional differential operator; when the order v takes a negative value, the above formula corresponds to the coefficients of the fractional integral operator.

[0066] Fractional calculus, deep neural network convolutional layer, image processing

[0067] Similar to the role of integer-order calculus in image processing, fractional-order differential operators still sharpen images, while fractional-order integrals smooth and blunt them, corresponding to the high-frequency information of edge details and the low-frequency information of smoothing, respectively. To fully utilize as much high- and low-frequency information as possible in an image for a specific processing task, this invention constructs a multi-channel convolutional neural network layer based on fractional-order calculus that can simultaneously and effectively extract both high- and low-frequency information from an image.

[0068] According to an embodiment of the present invention, a deep neural network convolutional layer is constructed using a designed fractional-order differential template for image processing.

[0069] According to an embodiment of the present invention, an image processing method based on a fractional-order calculus deep neural network convolutional layer is provided, comprising the following steps:

[0070] Step S1: Obtain the image to be processed.

[0071] Step S2: The acquired image to be processed is processed by convolutional layer of a deep neural network.

[0072] like Figure 1 The diagram shown is a schematic of the structure of a deep neural network convolutional layer in one embodiment of the present invention. The deep neural network convolutional layer includes c_in input channels and c_out output channels, and each output channel includes c_in convolutional kernels.

[0073] For example, if the image to be processed is a color image X with three input channels (red, green, and blue), and n feature maps Y are needed as outputs, then the convolutional layer requires n output channels. Each output channel Y... i (i = 1, ..., n) all require convolution calculations for each input channel, meaning each output channel requires three convolution kernels. Convolution calculations are performed on the three input channels: red, green, and blue.

[0074] According to an embodiment of the present invention, each convolutional kernel is defined as follows:

[0075] Generate a matrix A with c_in rows × c_out columns. The first element of matrix A is a real number between 0 and 1 randomly generated by the system during network initialization (and is subsequently adjusted as a network parameter during training). Take the negative of all the first elements of matrix A with an interval of one column to obtain the order matrix A' used to generate the convolution kernel.

[0076] Taking a deep neural network convolutional layer with input channels c_in=3 and output channels c_out=8 as an example, the following matrix A is generated:

[0077]

[0078] Each first element v (order) in matrix A is a randomly generated real number between 0 and 1. The order matrix A', which is obtained by taking the negative of each first element v (order) in matrix A with each column skipped, is as follows:

[0079]

[0080] By using each first element v (order) in the order matrix A', the second element of the convolution kernel matrix is ​​calculated to generate the convolution kernel.

[0081] According to an embodiment of the present invention, the second element in the convolution kernel matrix A' is calculated by the following method:

[0082]

[0083] Among them, M v Let i and j be the second elements of the convolution kernel matrix, and let i and j be the coordinates of the second elements of the convolution kernel matrix.

[0084]

[0085] Where v is the first element in the order matrix A'.

[0086] According to an embodiment of the present invention, the convolution kernel has a size of m_size × m_size. In one embodiment, taking a 3×3 convolution kernel as an example, the 3×3 convolution kernel matrix is ​​represented as:

[0087]

[0088] Substitute each first element v (order) in the order matrix A' into all the second elements of the 3×3 convolution kernel matrix to obtain 3×8=24 3×3 convolution kernels with 8 output channels. Use the obtained convolution kernels to perform convolution calculations on the color image X to be processed.

[0089] For example, when the first element v = 0.500, the corresponding convolution kernel matrix is:

[0090]

[0091] Similarly, by substituting each first element v of the order matrix A' into all the second elements of the 3×3 convolution kernel matrix, we obtain 24 convolution kernels.

[0092] In another embodiment, taking a 5×5 convolution kernel as an example, the 5×5 convolution kernel matrix is ​​represented as follows:

[0093]

[0094] Substitute each first element v (order) in the order matrix A' into all the second elements of the 5×5 convolution kernel matrix to obtain 3×8=24 5×5 convolution kernels with 8 output channels. Use the obtained convolution kernels to perform convolution calculations on the color image X to be processed.

[0095] For example, when the first element v = 0.500, the corresponding convolution kernel matrix is:

[0096]

[0097] Similarly, by substituting each first element v of the order matrix A' into all the second elements of the 5×5 convolution kernel matrix, we obtain 24 convolution kernels.

[0098] According to an embodiment of the present invention, the number of input channels c_in and the number of output channels c_out determine the order matrix A'. The c_in first elements v (order) of each column of the order matrix A' generate c_in convolution kernels, which perform convolution calculations on the c_in input channels respectively to generate a feature map of an output channel.

[0099] Therefore, each column of the order matrix A' generates c_in convolution kernels corresponding to one output channel, that is, each column of the order matrix A' corresponds to one output channel, and the order matrix A' has a total of c_out columns, generating a total of c_out output channels.

[0100] When the column intervals of the order matrix A' are reversed, the second element of the c_in convolution kernel matrices generated by the negative column is always positive. Then, the low-frequency information of the input image is extracted through the output channel generated by the negative column of the order matrix A'.

[0101] When the column intervals of the order matrix A' are reversed, the second element of the c_in convolution kernel matrices generated by the positive column is negative except for the center position which is 1. Then, the high-frequency information of the input image is extracted through the output channel generated by the positive column of the order matrix A'.

[0102] In the order matrix A', each column corresponds to one output channel, meaning one output channel corresponds to c_in first elements v (order), which in turn correspond to c_in convolution kernels. Thus, the c_in convolution kernels corresponding to one output channel constitute a three-dimensional filter with size m_size × m_size × c_in.

[0103] Let mask be the filter corresponding to the j-th output channel. j Let x be the input signal with the c_in channel. Then the output of the j-th output channel can be expressed as:

[0104] out j =conv(x, mask) j )

[0105] Where conv(x, mask) j ) indicates the use of a filter mask j Perform convolution calculations on the data x.

[0106] like Figure 2The illustration shows a feature map output by the negative-order output channel in one embodiment of the present invention. The input is a color image (with 3 channels), and the output is an 8-channel image convolution process. Four output channels are generated using the negative column in the order matrix A', resulting in the output feature map.

[0107] Figure 2 In the image processing, (a) is the feature map of the output channel generated by the three convolution kernels v generated by the first element v of the first column (negative column) in the order matrix A'.

[0108] Figure 2 In (b), the first element v of the third column (negative column) of the order matrix A' is the feature map of the output channel after image processing by the three convolution kernels generated.

[0109] Figure 2 In the image processing, (c) represents the feature map output by the three convolution kernels generated by the first element v of the fifth column (negative column) in the order matrix A'.

[0110] Figure 2 In the image processing, (d) represents the feature map output by the three convolution kernels generated by the first element v of the seventh column (negative column) in the order matrix A'.

[0111] like Figure 3 The invention shown in one embodiment uses a positive-order output channel to generate a feature map. The input is a color image (with 3 channels), and the output is an 8-channel image convolution. Four output channels are generated using the positive number columns in the order matrix A', resulting in the output feature map.

[0112] Figure 3 In the image processing, (d) represents the feature map output by the three convolution kernels generated by the first element v of the second column (positive number column) in the order matrix A'.

[0113] Figure 3 In the matrix A', (e) is the feature map of the output channel generated by the three convolution kernels v after image processing, which is the first element of the fourth column (positive number column).

[0114] Figure 3 In the image processing, f is the feature map of the output channel generated by the three convolution kernels v generated by the first element v of the sixth column (positive number column) in the order matrix A'.

[0115] Figure 3 In the image processing, (g) is the feature map of the output channel generated by the three convolution kernels v generated by the first element of the eighth column (positive number column) in the order matrix A'.

[0116] From the appendix Figure 2 and attached Figure 3It is evident that the feature map output by the negative-order output channel is relatively blurry, which is a smoothing effect on the original input image, while the feature map output by the positive-order output channel has relatively clear texture and edge information, which is a sharpening effect on the original input image.

[0117] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. An image processing method based on fractional calculus deep neural network convolutional layers, characterized in that, The image processing method includes: Obtain the image to be processed. The acquired image is processed by convolutional layers of a deep neural network, wherein... The convolutional layer includes c_in input channels and c_out output channels, with each output channel including c_in convolutional kernels. Each convolutional kernel is defined as follows: Set up a parameter matrix A with c_in rows × c_out columns. Each element in matrix A is a real number between 0 and 1 randomly generated by the system during network initialization. Taking the negative of all elements in matrix A, with each element separated by one column, yields the order matrix A' used to generate the fractional-order convolution kernel. The convolution kernel is generated by calculating the elements of the convolution kernel matrix using each element of the order matrix A' according to the fractional calculus formula; the elements of the convolution kernel matrix are calculated using the following method: Among them, M v Let i and j be the convolution kernel matrix, and i and j be the coordinates of the elements of the convolution kernel matrix. Where v is an element in the order matrix A'.

2. The image processing method according to claim 1, characterized in that, The convolution kernel has a size of m_size × m_size.

Images