Construction method, system and storage medium of deep neural network

Abstract

The application discloses a construction method, system and storage medium of a deep neural network. According to the basic principle of wavelet denoising, the method adopts a Gaussian wavelet as an excitation function, so that the number of neuron nodes of the constructed deep neural network is greatly reduced and the layer-to-layer sparse connection of the hidden layer is realized. The construction method of the deep neural network comprises the following steps: obtaining output values and weight values of each neuron in a first deep neural network; determining a neuron with a product of the output value and the weight value less than a first threshold value as a low-contribution neuron; and removing the low-contribution neuron from the first deep neural network to obtain a second deep neural network. Meanwhile, the standard deviation of the noise can be estimated from the weight (wavelet coefficient) to serve as a reference threshold value, and small weights in the network are set to zero to perform denoising and realize sparse connection. In this way, the training and testing time of the network is reduced, and the problems of gradient explosion, vanishing gradient and overfitting are reduced while the training accuracy is maintained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep neural network, in particular to a deep neural network construction method, system and storage medium. BACKGROUND

[0002] At present, training a neural network model usually uses an optimization algorithm to find weight values and hyperparameters so as to more accurately map input values to output values, and a deep neural network exhibits strong performance in various applications by processing large data sets.

[0003] In a neural network, as moving backward through the hidden layers, the gradient tends to become smaller, and the neurons of the earlier layers may learn much more slowly than the neurons of the later layers, that is, there is a vanishing gradient problem; at the same time, when the gradient of the earlier layers becomes very large, since the gradient in the early layers is the product of all subsequent layers, when there are multiple layers in the neural network, the neural network may be inherently unstable, and due to the instability of the gradient in the deep neural network, there is a tendency to gradient explosion or vanishing in the earlier layers, which is a common drawback of gradient-based learning methods in deep neural networks; in addition, due to the current technical challenges of deep neural networks such as over-parameterization, high computational complexity and hardware resource requirements, there are outstanding problems of over-fitting caused by over-parameterization and noise. SUMMARY

[0004] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0005] The embodiment of the present application provides a deep neural network construction method, system and storage medium, which is based on a fully connected deep neural network, and adopts a tight frame Gaussian wavelet with nonlinear, sparsity, time-frequency localization and energy conservation as an excitation function according to the basic principle of wavelet denoising, so that the number of neuron nodes of the constructed deep neural network is greatly reduced and the layer-to-layer sparse connection of the hidden layer is realized. The deep neural network constructed by the present application is simplified and optimized, has very strong nonlinear fitting ability and can quickly converge, and reduces the occurrence of gradient explosion problem, vanishing gradient problem and overfitting problem.

[0006] In a first aspect, the embodiment of the present application provides a deep neural network construction method, which comprises:

[0007] obtaining output values and weight values of each neuron in a first deep neural network;

[0008] determining a neuron whose product of the output value and the weight value is less than a first threshold value as a low contribution neuron;

[0009] remove the low contribution neurons in the first deep neural network to obtain a second deep neural network.

[0010] Optionally, in some embodiments, before the determining the neurons whose product of the output value and the weight value is less than the first threshold as the low contribution neurons, further comprising:

[0011] obtain a noise standard deviation of the first deep neural network;

[0012] set the weight values less than the noise standard deviation in the first deep neural network to zero to make the first deep neural network achieve sparse connection.

[0013] Optionally, in some embodiments, the activation function of the first deep neural network is tight frame Gaussian wavelet, and the obtaining the noise standard deviation of the first deep neural network comprises:

[0014] obtaining the noise standard deviation of the deep neural network according to wavelet coefficients of the tight frame Gaussian wavelet, the wavelet coefficients corresponding to the plurality of weight values of the deep neural network.

[0015] Optionally, in some embodiments, before the obtaining the output value and the weight value of each neuron in the first deep neural network, further comprising:

[0016] preliminarily training a fully connected neural network with the tight frame Gaussian wavelet as the activation function to obtain the first deep neural network.

[0017] Optionally, in some embodiments, the preliminarily training the fully connected neural network with the tight frame Gaussian wavelet as the activation function comprises:

[0018] obtaining an input-output value data set;

[0019] initializing a structure, a plurality of hyperparameters and a plurality of weight values of the fully connected neural network;

[0020] training the deep neural network according to the input-output value data set to adjust the plurality of weight values and the plurality of hyperparameters of the deep neural network.

[0021] Optionally, in some embodiments, after the removing the low contribution neurons in the first deep neural network to obtain the second deep neural network, further comprising:

[0022] obtaining a network output value and an actual output value of the second deep neural network;

[0023] obtaining a mean square error between the network output value and the actual output value;

[0024] In the case that the mean square error is greater than the second threshold, the structure, the plurality of hyperparameters and the plurality of weight values of the second deep neural network are adjusted by an internal optimization method, and the plurality of weight values in the second deep neural network are updated according to the input-output value data set.

[0025] Optionally, in some embodiments, after the deep neural network is updated according to the input-output value data set to update the plurality of weight values, the method further comprises:

[0026] The steps of obtaining the output value and the weight value of each neuron in the first deep neural network, obtaining the noise standard deviation of the first deep neural network, setting the weight value less than the noise standard deviation in the first deep neural network to zero to realize sparse connection of the first deep neural network, determining the neuron whose product of the output value and the weight value is less than a first threshold as a low-contribution neuron, removing the low-contribution neuron in the first deep neural network, and obtaining a second deep neural network are performed again.

[0027] Optionally, in some embodiments, the tight-frame Gaussian wavelet function is a Mexican hat Gaussian wavelet with a tight frame.

[0028] In a second aspect, embodiments of the present application provide a system for constructing a deep neural network, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for constructing a deep neural network according to any one of the first aspect when executing the computer program.

[0029] In a third aspect, embodiments of the present application provide a computer readable storage medium storing computer executable instructions for implementing the method for constructing a deep neural network according to any one of the first aspect.

[0030] The embodiment of the present application comprises a deep neural network construction method, a system and a storage medium, wherein the deep neural network construction method comprises: performing preliminary training on the deep neural network by taking a tight frame Gaussian wavelet as an excitation function; obtaining a noise standard deviation of the deep neural network; performing noise reduction processing on the deep neural network according to the noise standard deviation, so as to make the deep neural network sparsely connected; removing low-contribution neurons in the deep neural network, wherein the product of the output value and the corresponding weight value of the low-contribution neuron is less than a first threshold value; and constructing an efficient deep neural network according to the deep neural network. The new deep neural network architecture constructed in this way not only has strong nonlinear fitting capability and can quickly converge, but also greatly reduces the computational complexity due to the reduction of network complexity, can greatly reduce the training and testing time of the network while maintaining the training accuracy, can meet the requirements of fast real-time operation, and can reduce the occurrence of problems such as gradient explosion, vanishing gradient and overfitting, and has a wide application prospect.

[0031] Other features and advantages of the present application will be set forth in the following description, and in part will be apparent from the description, or can be learned by practice of the application, the objects and other advantages of the present application can be achieved and obtained by the structure specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0032] The accompanying drawings are used to provide a further understanding of the technical scheme of the present application, and constitute a part of the specification, and are used together with the embodiments of the present application to explain the technical scheme of the present application, and do not constitute a limitation on the technical scheme of the present application.

[0033] Figure 1 is a step flow chart of the deep neural network construction method provided by an embodiment of the present application;

[0034] Figure 2 is a step flow chart of the deep neural network construction method provided by another embodiment of the present application;

[0035] Figure 3 is a step flow chart of the deep neural network construction method provided by another embodiment of the present application;

[0036] Figure 4 is an architecture diagram of the fully connected deep neural network provided by another embodiment of the present application;

[0037] Figure 5 is an input-output value relationship diagram of a single neuron provided by another embodiment of the present application;

[0038] Figure 6 is an architecture diagram of the efficient deep neural network provided by another embodiment of the present application;

[0039] Figure 7 is a waveform diagram of a typical nonlinear equation provided by another embodiment of the present application;

[0040] Figure 8 is an example diagram of a construction method of a deep neural network provided by another embodiment of the present application;

[0041] Figure 9 is a schematic diagram of a high-efficiency deep neural network construction system provided by another embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0043] It should be noted that although the functional modules are divided in the system schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a manner different from the module division in the system or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0044] The embodiments of the present application include a construction method of a deep neural network, a system and a storage medium. The construction method of the deep neural network comprises: preliminarily training the deep neural network, wherein the excitation function of the deep neural network is a tight frame Gaussian wavelet; obtaining the noise standard deviation of the deep neural network; performing noise reduction processing on the deep neural network according to the noise standard deviation, so as to make the deep neural network sparsely connected; removing low-contribution neurons in the deep neural network, wherein the product of the output value and the corresponding weight value of the low-contribution neuron is less than a first threshold value; and on the basis of the fully connected deep neural network, using a tight frame Gaussian wavelet with the characteristics of nonlinearity, sparsity, time-frequency localization and energy conservation as the excitation function, and according to the wavelet transform support based on sparse multi-scale representation and the time-frequency localization characteristics of the wavelet domain and the basic principle of wavelet noise reduction, in the network architecture of wavelet time-frequency decomposition, after preliminary training, the small weights in the network are set to zero to perform noise reduction and realize sparse connection, and at the same time, when the product of the output and the weight of a neuron is lower than a certain threshold value, the neuron is removed in the network to realize further noise reduction, the purpose being to simplify and optimize the network structure, so as to greatly reduce the possibility of overfitting.

[0045] The embodiments of the present application are further described below in combination with the drawings.

[0046] Reference Figure 1 , Figure 1is a step flow chart of a construction method of a deep neural network provided by an embodiment of the present application, and the construction method of the deep neural network includes but is not limited to the following steps:

[0047] In step S110, the output value and the weight value of each neuron in the first deep neural network are obtained.

[0048] The first deep neural network refers to a deep neural network formed after preliminary training.

[0049] In some embodiments, before obtaining the output value and the weight value of each neuron in the first deep neural network, the fully connected neural network is preliminarily trained by taking the tight frame Gaussian wavelet as an excitation function to obtain the first deep neural network.

[0050] It is conceivable that the excitation function plays an important role in normalizing data and breaking the linear mapping relationship of the grid in the neural network, so the selection of the excitation function directly affects the performance of the neural network.

[0051] Meanwhile, it should be noted that the wavelet excitation function is used to construct a wavelet neural network (WNN) to realize sparse connection and simplify and optimize the network structure. The wavelet time-frequency analysis method is introduced here, and the wavelet is an unconditional basis of various function spaces, which means that it usually compresses the energy of the signal into a small number of coefficients (sparsity) obtained by wavelet transform (WT).

[0052] It should be noted that the wavelet frame theory is an important part of the wavelet theory, because the frame shows the energy bound of the wavelet signal. In order to reduce the calculation amount of the wavelet transform, the amplification parameter a and the moving parameter b can be discretized:

[0053]

[0054] Where a0>1, b0>0. Therefore, the discrete wavelet family is:

[0055]

[0056] The wavelet frame is defined on the discrete wavelet family, that is, when there are 0

[0057]

[0058] The wavelet function ψ m,n forms a frame, where A and B are the upper and lower bounds of the frame, respectively. When A=B, the wavelet frame is a tight frame and the energy m,n of the wavelet family ψ (t) satisfies:

[0059]

[0060] At this time, the energy of the tight frame wavelet is constant and depends on the frame boundary. When the tight frame wavelet is used as the excitation function and the differential function of the neural network, the propagation of signal energy is constant, which does not make the signal propagation process diverge, thereby ensuring convergence. If the tight frame wavelet is also Gaussian, it has the characteristics of nonlinearity, sparsity, time-frequency localization and energy conservation, can realize fast convergence of the training process and has very strong nonlinear fitting ability. Therefore, the application uses the tight frame Gaussian wavelet as the excitation function, so that the computational complexity is greatly reduced, the training and testing network time can be greatly reduced while maintaining the training accuracy, which can meet the requirements of fast real-time operation and has broad application prospects.

[0061] Step S120, determining the neuron whose product of the output value and the weight value is less than the first threshold value as a low-contribution neuron;

[0062] It is conceivable that the product of the output and weight of a neuron can be regarded as the contribution of the neuron to the network, and when the value is lower than a certain set threshold value (i.e. the first threshold value), the neuron is removed from the network. This process can be trained multiple times to simplify and optimize the network structure, thereby greatly reducing the possibility of overfitting.

[0063] It is conceivable that in some embodiments, the first threshold value can be selected by a person skilled in the relevant field according to the actual situation, and the specific value does not limit the application.

[0064] Step S130, removing the low-contribution neuron in the first deep neural network to obtain a second deep neural network.

[0065] The second deep neural network refers to the deep neural network formed after the low-contribution neuron is removed.

[0066] In some embodiments, after removing the low-contribution neuron in the deep neural network, the product of the output value and the corresponding weight value of the low-contribution neuron is less than the first threshold value, the construction of the deep neural network is further completed.

[0067] It is conceivable that in some embodiments, the tight frame Gaussian wavelet with nonlinear, sparsity, time-frequency localization and energy conservation is used as the excitation function, so that the deep neural network has very strong nonlinear fitting ability and can quickly converge. At the same time, the step of setting the small weight in the network to zero for noise reduction and realizing sparse connection, and removing the low contribution neurons in the neural network can be carried out many times in the process of constructing the entire neural network, so as to simplify and optimize the network structure, greatly reduce the possibility of overfitting, greatly reduce the calculation complexity, greatly reduce the training and testing network time while maintaining the training accuracy, and meet the requirement of fast real-time operation.

[0068] Reference Figure 2 , Figure 2 is a step flow chart of a deep neural network construction method provided by another embodiment of the application. The deep neural network construction method includes but is not limited to the following steps:

[0069] Step S210, obtaining the noise standard deviation of the first deep neural network;

[0070] Step S220, setting the weight value smaller than the noise standard deviation in the first deep neural network to zero to realize sparse connection of the first deep neural network.

[0071] In some embodiments, the noise standard deviation of the deep neural network is obtained according to the wavelet coefficient of the tight frame Gaussian wavelet, wherein the wavelet coefficient corresponds to a plurality of weight values of the deep neural network.

[0072] It is worth noting that if the signal s(t) contains noise, that is, the obtained signal is s(t)+n(t), the wavelet inverse transform can be described as

[0073]

[0074] Assuming that the variance of the signal s(t) is The variance of the noise n(t) is The signal-to-noise ratio (SNR) can be described as Wavelet denoising usually estimates the standard deviation σ of the noise from the wavelet coefficient j As a reference threshold to filter out small wavelet coefficients, in order to achieve the purpose of noise reduction and improving the signal-to-noise ratio. Here is a commonly used method for estimating the standard deviation of noise:

[0075]

[0076] This denoising process is carried out in the wavelet domain based on the sparse multi-scale representation, and the characteristics of the signal will be reflected in the larger wavelet coefficient value.

[0077] It is worth mentioning that according to the wavelet transform support based on sparse multi-scale representation, and the time-frequency localization characteristics of the wavelet domain and according to the basic principle of wavelet denoising, that is, small wavelet coefficients can be regarded as noise, in the network architecture of wavelet time-frequency decomposition, after preliminary training, the standard deviation of the noise can be estimated from the weight (wavelet coefficient) as a reference threshold, and the small weight in the network can be set to zero for denoising and realizing sparse connection.

[0078] According to the noise standard deviation, the deep neural network is denoised to make the deep neural network sparsely connected.

[0079] In some embodiments, according to the noise standard deviation, the deep neural network is denoised to make the deep neural network sparsely connected, specifically including: setting the weight value smaller than the noise standard deviation in the deep neural network to zero to perform denoising and realize the sparse connection of the deep neural network.

[0080] It is worth mentioning that the training process of the neural network is to find the best weight value, and from the process of wavelet inverse transform, it is to find the best wavelet coefficient to construct the accurate input-output relationship. This process is equivalent to a filtering process, that is, wavelet filtering. Because of the sparsity of the wavelet function, the training process may make some weight values small and can be set to zero, which can not only reduce the complexity of calculation, but also reduce the connection between layers, thereby simplifying the structure of the neural network and improving the training speed.

[0081] Similarly, by using the sparsity of the compactly supported Gaussian wavelet as the excitation function, in the network architecture of wavelet time-frequency decomposition, after preliminary training, the weight w i,j (wavelet coefficient) and a set of hyperparameters θ of the fully connected network are obtained. According to the basic principle of wavelet denoising, that is, small wavelet coefficients can be regarded as noise, the standard deviation σ j of the noise can be estimated from the wavelet coefficient as a reference threshold, and the small weight in the network can be set to zero for denoising and realizing sparse connection.

[0082] In some embodiments, the construction method of the deep neural network includes but is not limited to the following steps:

[0083] Obtaining an input-output value data set;

[0084] Initializing the structure of the fully connected neural network, a plurality of hyperparameters and a plurality of weight values;

[0085] Training the deep neural network according to the input-output value data set to adjust the plurality of weight values and the plurality of hyperparameters of the deep neural network.

[0086] The hyperparameters include the number of intermediate layers in the network, the type (full connection, dropout layer, normalization layer, convolutional layer, etc.), the number of neurons in each layer, the activation function, the loss function, the optimization method, the batch size, the number of iterations, the learning rate, the regularization method and coefficient, the initialization method, and other common parameter types.

[0087] It is conceivable that training a neural network model usually uses an optimization algorithm to find the weight values and hyperparameters, and the input-output relationship of a deep neural network can be represented as: y = Ax + e, where A is a weight set composed of multiple weight values, and e is an error. Then the step-by-step update of the weight values of the weight set in the training can minimize the error. That is, when the weights change in the gradient descent direction, as a method of updating and adjusting the weights, the adjustment of the weights is based on the difference between the actual output and the input of each layer. This can speed up the convergence process, where a larger adjustment is taken when the difference is larger, and a smaller adjustment is taken when the difference is smaller.

[0088] It is conceivable that in deep learning, the weight initialization method of the neural network has a crucial influence on the convergence speed and performance of the model. The method for initializing the weights in the present application includes initializing the weights to 0, randomly initializing the weights, Xavier initialization, He initialization, and other common initialization methods. Related technical personnel can choose the method used according to the actual situation, which does not limit the present application.

[0089] Reference Figure 3 , Figure 3 is a step flowchart of a construction method of a deep neural network according to another embodiment of the present application. The construction method of the deep neural network includes but is not limited to the following steps:

[0090] Step S310, obtaining the network output value and the actual output value of the second deep neural network;

[0091] Step S320, obtaining the mean square error between the network output value and the actual output value;

[0092] Step S330, in the case where the mean square error is greater than the second threshold value, adjusting the structure, the plurality of hyperparameters, and the plurality of weight values of the second deep neural network by the built-in optimization method, and updating the plurality of weight values in the second deep neural network according to the input-output value data set.

[0093] It is conceivable that in some embodiments, including: in the case where the mean square error is less than or equal to the second threshold value, an efficient deep neural network can be constructed according to the method of the present application.

[0094] In some embodiments, after the plurality of weight values of the deep neural network are updated according to the input-output value dataset, the steps of obtaining the noise standard deviation of the deep neural network, performing the noise reduction processing on the deep neural network according to the noise standard deviation, and removing the low-contribution neurons in the deep neural network are performed again until the mean square error is less than or equal to the second threshold value.

[0095] It is worth noting that whether the simplified and optimized neural network meets the accuracy requirement can be analyzed by calculating the mean square error (MSE) between the output of the trained network and the actual output. If it does not meet the requirement, the parameters of the simplification and optimization network structure process can be adjusted again, and this process is repeated multiple times until the accuracy meets the requirement, while the network structure is simplified to the maximum extent to reduce the computational complexity and maximize the network performance.

[0096] It is worth noting that in some embodiments, the built-in optimization method includes common self-optimization methods such as grid search, random search, Bayesian optimization, dynamic resource allocation, and neural architecture search. Related technical personnel can choose the method used according to the actual situation, which does not limit the present application.

[0097] Reference Figure 4 and Figure 5 , Figure 4 is the architecture diagram of the fully connected deep neural network provided by another embodiment of the present application, Figure 5 is the input-output value relationship diagram of a single neuron, and the present application uses a wavelet excitation function to construct a wavelet neural network (WNN) to realize sparse connection and simplification and optimization of network structure. The wavelet time-frequency analysis method is introduced here. The wavelet is an unconditional basis of various function spaces, which means that it usually compresses the energy of a signal into a small number of coefficients (sparsity) obtained by wavelet transform (WT). These coefficients in the wavelet domain are sufficient to represent the original data, which helps to reduce the computational complexity and the required hardware resources. The wavelet transform is defined as the inner product of a signal s(t) and a two-parameter family of ψ a,b (t):

[0098]

[0099] wherein, denotes the complex conjugate of ψ, ψ a,b (t) is the magnification and movement of the wavelet basis function ψ(t) to obtain time-frequency localization:

[0100]

[0101] wherein, and a≠0, ψ(t) is assumed to be acceptable, i.e., to satisfy

[0102]

[0103] Here, is the frequency description of the Fourier transform of ψ(t), i.e.

[0104]

[0105] with the inverse wavelet transform can be described as

[0106]

[0107] corresponds to the output of a certain layer in a deep neural network architecture

[0108]

[0109] where the reference Figure 4 , ψ m is the wavelet activation function, w m is the weight value, which can also be regarded as a wavelet coefficient. If the wavelet function ψ is chosen as the activation function, and x is the input, y is the output, and W is the weight coefficient, the training process to fit any nonlinear relationship y = f(x) can be described as follows:

[0110] φ (1) (x, θ) = ψ (1) (W (1) x + b (1) )

[0111] φ (2) (x, θ) = ψ (2) (W (2) φ (1) (x, θ) + b (2) )

[0112] φ (l) (x, θ) = ψ (l) (W (l) φ (l-1) (x, θ) + b (l) )

[0113]

[0114] φ (L) = ψ (L) (W (L) φ (L-1) (x, θ) + b (L) )

[0115] y = W (L+1) φ (L) (x, θ) + b (L+1)

[0116] where b is bias, l∈L-1 is an integer, and the input-output relationship of the jth node of the lth layer is referenced Figure 5 .

[0117] In some embodiments, the tight frame Gaussian wavelet function is specifically a Mexican hat Gaussian wavelet with a tight frame. The Mexican hat wavelet is used because the Mexican hat (Mexican hat) wavelet is a Gaussian

[0118]

[0119] The normalized form can be expressed as

[0120]

[0121] And

[0122]

[0123] where the pan number ‖ψ‖=1, And

[0124] Therefore, the wavelet function is a wavelet frame. By calculating the frame bounds A and B, and different a0 and b0 values, the present application selects b0=0.25, m=1, n=1, A=B at this time, and the tight frame Mexican hat wavelet function is obtained as the excitation function (FP forward process):

[0125]

[0126] In the back propagation BP algorithm, the differential function needs to be used:

[0127]

[0128] If the same selection is made b0=0.25, m=1, n=1, A=B at this time, and the tight frame Mexican hat wavelet function is obtained as the excitation function (BP back propagation process):

[0129]

[0130] The tight frame Mexican hat wavelet function used as the neural network excitation function above

[0131] ψ mexh and ψ bp, both of which are Gaussian type, with nonlinear, sparsity, time-frequency localization and energy conservation. The following examples show that the use of tight frame Gaussian wavelet as an excitation function can make the deep neural network have very strong nonlinear fitting ability and can quickly converge, at the same time, according to the wavelet transform support based on sparse multi-scale representation, and the time-frequency localization characteristics of the wavelet domain and according to the basic principle of wavelet denoising, the network architecture of wavelet time-frequency decomposition can be simplified and optimized, thereby greatly reducing the possibility of overfitting and meeting the requirements of fast real-time operation.

[0132] Similarly, referring to Figure 6 , Figure 6 is the architecture diagram of the efficient deep neural network provided by another embodiment of the application, which utilizes the sparsity of the tight frame Gaussian wavelet as an excitation function. In the network architecture of wavelet time-frequency decomposition, after preliminary training, the weights (wavelet coefficients) of the full connection network and a set of hyperparameters are obtained. According to the basic principle of wavelet denoising, that is, small wavelet coefficients can be regarded as noise, the standard deviation of the noise can also be estimated from the wavelet coefficients as a reference threshold, and the small weights in the network can be set to zero for denoising and realizing sparse connection. At the same time, the product of the output and weight of a neuron can be regarded as the contribution of this neuron to the network, and when this value is lower than a certain set threshold, the neuron is removed from the network. This process can be trained multiple times to simplify and optimize the network structure, thereby greatly reducing the possibility of overfitting. Figure 6 An example of a network after being simplified and optimized by the above method is given, from which it can be seen that the number of neuron nodes in the hidden layer of the network is reduced and some nodes are sparsely connected. Due to the reduction of network complexity, the computational complexity is also greatly reduced, the training accuracy is maintained, the time for training and testing the network is greatly reduced, and the requirements of fast real-time operation can be met.

[0133] Referring to Figure 7 , Figure 7 is a waveform diagram of a typical nonlinear equation provided by another embodiment of the application.

[0134] In some embodiments, in order to prove that the efficient neural network proposed in the application has stronger nonlinear fitting ability and can quickly converge relative to the traditional neural network, the performance test of the efficient neural network proposed in the application is performed again through the nonlinear equation Figure 7 , the process is as follows:

[0135] First, a nonlinear equation f(x) is constructed:

[0136]

[0137] Its waveform is as follows: Figure 7The function, which is shown in the following equation, is a typical nonlinear function that contains two different frequency sinusoidal waves, an exponential function, a polynomial and a Rayleigh decay kernel. The Gaussian-type tight-frame Mexican-hat wavelet is used as the activation function, which has very strong nonlinear fitting ability. This can be proved in the following tests.

[0138] In the tests, the conventional neural network uses the nonlinear Sigmoid function as the activation function for comparison. The number of hidden layers is selected as 8, and the initial number of neurons is selected as 256, 128 and 64 for testing and comparative analysis. The test results of the signal without and with added noise are shown in Tables (1) and (2), respectively. In the table, the worst case of the mean square error (MSE) between the network output and the actual output is taken as 0% as a reference, and the rest represents the relative reduction percentage of the MSE.

[0139] Table (1) Nonlinear equation fitting performance (signal without noise)

[0140] Method Number of hidden layer nodes MSE Change (%) Conventional neural network 64 to 32 2.7099 0 128 to 64 2.4895 -8.13 256 to 128 2.2698 -16.24 Neural network of the present application 64 to 29 2.2453 -17.14 128 to 61 2.2453 -17.14 256 to 123 2.2448 -17.16

[0141] Table (2) Nonlinear equation fitting performance (signal with noise)

[0142]

[0143]

[0144] The above data are experimental data recorded by the applicant through a large number of experiments, and from the test results, it can be seen that the constructed wavelet neural network (WNN) uses a Gaussian type tight frame wavelet activation function to realize denervation and sparse connection, and can obtain better network performance. Especially, the method of the application is used to remove neuron nodes and realize sparse connection, which not only does not reduce the training precision, but also can improve the training precision in the presence of noise, and the operation time is greatly reduced. This shows that the method of simplifying and optimizing the network architecture of wavelet time-frequency decomposition according to the wavelet transform support based on sparse multi-scale representation and the time-frequency localization characteristics of wavelet domain and the basic principle of wavelet denoising is efficient and can reduce the influence of noise, and the operation speed is faster. Since the wavelet function has the characteristics of nonlinearity, sparsity, time-frequency localization and energy conservation, it has advantages in signal detection, feature selection and data decomposition. In particular, the derived Mexican hat wavelet as an activation function, because it is Gaussian, has a symmetric structure and sparsity, which can make the simplified network faster in calculation and obtain faster convergence. By using the Gaussian wavelet function, WNN can better fit and model complex nonlinear relationships, and the network structure is simplified and optimized, thereby greatly reducing the possibility of overfitting. Due to the reduction of network complexity, the calculation complexity is also greatly reduced, while maintaining the training precision, the training and testing network time can be greatly reduced, which can meet the requirements of fast real-time operation, so its application field is very wide.

[0145] In addition, with reference to Figure 8 , Figure 8 is an example diagram of efficient deep neural network construction by the construction method of the deep neural network according to another embodiment of the application, which includes but is not limited to the following steps:

[0146] Step S801, obtaining input-output data set (x, y);

[0147] Step S802, setting neural network structure and hyperparameters θ and initializing weights w;

[0148] Step S803, training the neural network to obtain optimal weights w;

[0149] Step S804, implementing the neural network simplification and optimization method;

[0150] Step S805, analyzing and judging whether the simplified and optimized neural network meets the accuracy requirement, if yes, executing step S807, and if no, continuing to execute step S806;

[0151] Step S806, adjusting the network structure and hyperparameters θ and weights w;

[0152] Step S807, for the above application to complete the construction of efficient deep neural network.

[0153] Wherein, it is contemplated that in step S804, the neural network simplification and optimization method is implemented, including the above-mentioned method steps S120, step S130 and step S140, the above embodiment is the working process of the deep neural network simplification and optimization method using tight frame Gaussian wavelet as the excitation function, wherein whether the simplified and optimized neural network meets the accuracy requirement can be analyzed and judged by calculating the mean square error (MSE) between the trained network output and the actual output, if it does not meet the requirement, the parameters of the simplification and optimization network structure process can be adjusted again, and the process is repeated multiple times until the accuracy meets the requirement.

[0154] In some embodiments, step S804, the neural network simplification and optimization method is implemented, including: obtaining the noise standard deviation of the deep neural network; performing noise reduction processing on the deep neural network according to the noise standard deviation; removing low-contribution neurons in the deep neural network, the product of the output value and the corresponding weight value of the low-contribution neuron being less than a first threshold value, after the neural network simplification and optimization method provided in the present application, an efficient deep neural network architecture can be constructed, which not only has strong nonlinear fitting capability and can quickly converge, but also greatly reduces the computational complexity due to the reduction of network complexity, while maintaining the training accuracy, the training and testing network time can be greatly reduced, and the requirement of fast real-time operation can be met.

[0155] At the same time, since the present application uses tight frame Gaussian wavelet as the excitation function on the basis of fully connected deep neural network, which has the characteristics of nonlinearity, sparsity, time-frequency localization and energy conservation, etc., the deep neural network has very strong nonlinear fitting capability and can quickly converge. According to the wavelet transform support based on sparse multi-scale representation, and the time-frequency localization characteristics of wavelet domain, and according to the basic principle of wavelet denoising, i.e. small wavelet coefficients can be regarded as noise, in the network architecture of wavelet time-frequency decomposition, after preliminary training, the standard deviation of noise can be estimated from the weight (wavelet coefficient) as a reference threshold, the small weight in the network is set to zero for noise reduction and sparse connection. At the same time, the product of the output and weight of a neuron can be regarded as the contribution of this neuron to the network, when this value is lower than a certain set threshold, the neuron can be removed in the network. This process can be trained multiple times to simplify and optimize the network structure, thereby greatly reducing the possibility of overfitting. The new deep neural network architecture constructed in this way simplifies the network structure to the maximum extent to reduce the computational complexity and maximize the network performance.

[0156] In addition, with reference to Figure 9An embodiment of the present application further provides a deep neural network construction system 900, an efficient deep neural network construction system 900, comprising a memory 920, a processor 910, and a computer program stored in the memory 920 and executable on the processor 910, wherein the processor 910 implements the deep neural network construction method of any one of the preceding embodiments when executing the computer program, for example, implements the method steps S110-S130 in the method of Figure 1 , the method steps S210-S220 in the method of Figure 2 , the method steps S310-S330 in the method of Figure 3 , and the method steps S801-S807 in the method of Figure 8 .

[0157] In addition, an embodiment of the present application further provides a computer readable storage medium storing computer executable instructions, wherein the computer executable instructions are executed by one or more control processors, for example, implement the method steps S110-S130 in the method of Figure 1 , the method steps S210-S220 in the method of Figure 2 , the method steps S310-S330 in the method of Figure 3 , and the method steps S801-S807 in the method of Figure 8 .

[0158] Those skilled in the art can understand that all or some steps in the above disclosed method and system can be implemented as software, firmware, hardware and appropriate combinations thereof. Some or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software can be distributed on a computer readable medium, which can include computer storage media (or non-transitory media) and communication media (or transitory media). As known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, as known to those skilled in the art, communication media typically includes computer readable instructions, data structures, program modules or other data in modulated data signals such as carrier waves or other transport mechanisms, and can include any information delivery medium.

[0159] The above describes the preferred embodiments of the present application, but the present application is not limited to the above-described embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A method for constructing a deep neural network, characterized in that, The method includes: Obtain the output and weight values ​​of each neuron in the first deep neural network; Neurons whose product of the output value and the weight value is less than a first threshold are identified as low-contribution neurons. Remove the low-contribution neurons from the first deep neural network to obtain the second deep neural network; Before determining neurons whose product of output value and weight value is less than a first threshold as low-contribution neurons, the method further includes: performing preliminary training on a fully connected neural network using a tight-frame Gaussian wavelet as the activation function to obtain a first deep neural network; obtaining the noise standard deviation of the deep neural network based on the wavelet coefficients of the tight-frame Gaussian wavelet, wherein the wavelet coefficients correspond to multiple weight values ​​of the deep neural network; and setting the weight values ​​in the first deep neural network that are less than the noise standard deviation to zero so that the first deep neural network achieves sparse connections.

2. The method for constructing a deep neural network according to claim 1, characterized in that, The preliminary training of the fully connected neural network using a tight-frame Gaussian wavelet as the activation function includes: Obtain the input and output value dataset; Initialize the structure, multiple hyperparameters, and multiple weight values ​​of the fully connected neural network; The deep neural network is trained based on the input-output value dataset to adjust multiple weight values ​​and multiple hyperparameters of the deep neural network.

3. The method for constructing a deep neural network according to claim 2, characterized in that, After removing the low-contribution neurons from the first deep neural network to obtain the second deep neural network, the process further includes: Obtain the network output value and the actual output value of the second deep neural network; Obtain the mean square error between the network output value and the actual output value; If the mean square error is greater than the second threshold, the structure, multiple hyperparameters, and multiple weight values ​​of the second deep neural network are adjusted by a built-in optimization method, and the multiple weight values ​​in the second deep neural network are updated according to the input-output value dataset.

4. The method for constructing a deep neural network according to claim 3, characterized in that, After updating the deep neural network according to the input-output value dataset to update the multiple weight values, the method further includes: The steps of obtaining the output value and weight value of each neuron in the first deep neural network, obtaining the noise standard deviation of the first deep neural network, setting the weight values ​​in the first deep neural network that are smaller than the noise standard deviation to zero so that the first deep neural network achieves sparse connection, determining the neurons whose product of the output value and the weight value is less than a first threshold as low contribution neurons, removing the low contribution neurons in the first deep neural network, and obtaining the second deep neural network are performed again.

5. The method for constructing a deep neural network according to any one of claims 1 to 4, characterized in that, The tight-frame Gaussian wavelet function is a Mexican hat Gaussian wavelet with a tight frame.

6. A system for constructing a deep neural network, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the method for constructing a deep neural network as described in any one of claims 1 to 5.

7. A computer-readable storage medium storing computer-executable instructions for performing a method for constructing a deep neural network as described in any one of claims 1 to 5.

Images