Data processing method, device and equipment

Abstract

The present invention relates to a data processing method, device and equipment, the method comprising the following steps: a. receiving data to be solved for partial differential equations, constructing a high-order neural network and a training data set and a test data set; b. converting the partial differential equation into a minimum optimization problem, and constructing a loss function and an optimization algorithm for the high-order neural network; c. training the high-order neural network and testing it on the test data set; d. inputting the data into the high-order neural network, solving the partial differential equation, obtaining a solution result and outputting it. The present invention can significantly improve the convergence speed and approximation accuracy of neural networks in solving high-frequency partial differential equations.

Description

Technical Field

[0001] The invention relates to a data processing method, device and equipment. Background Art

[0002] High-frequency problems arise in various scientific and engineering applications, such as the high-frequency Helmholtz equations generated in electromagnetism. Although these problems usually occur in low-dimensional space, their highly oscillatory nature makes it extremely difficult to find high-precision numerical solutions. In order to effectively approximate the high-frequency components in the solution, the existing technology usually uses high-order methods such as spectral methods and high-order finite element methods. However, these numerical methods generally rely on meshes, which are difficult and costly to generate in complex areas.

[0003] Due to the existence of the above problems, numerical solutions to high-frequency partial differential equations based on neural networks have attracted widespread attention in the computational science community. For example, many new methods based on the Ritz formula, the least squares formula, and the Galerkin formula have been proposed one after another, and they have also produced significant empirical results. In addition, for partial differential equations defined in high-dimensional irregular regions, classical numerical methods have problems such as slow calculation speed, instability, and dimensionality disaster, while discrete neural networks have become a potential tool for solving complex partial differential equations due to their meshless characteristics. However, when neural networks are applied to solving high-frequency partial differential equations, there are still many challenges. Specifically, during the training process, neural networks often give higher priority to the low-frequency components of the objective function, so that they cannot approximate the high-frequency components well, resulting in convergence stagnation in the later stages of training. This phenomenon that neural networks have difficulty capturing high-frequency information is called the frequency principle or spectrum deviation.

[0004] To this end, some technologies use phase-shift neural networks to shift the high-frequency components of the true solution to low frequencies in the frequency domain, and some technologies use multi-scale neural networks with compactly supported activation functions. Both methods transform the problem of approximating the high-frequency components of partial differential equation solutions into the problem of approximating the low-frequency components, so that neural networks can better solve high-order partial differential equations, and both have achieved impressive numerical experimental results. However, this conversion process often slows down the convergence of the neural network.

[0005] In order to obtain the advantages of high-order methods in processing high-frequency components, some technologies introduce high-order ideas into common neural networks to improve their ability to approximate high-frequency components. Specifically, this type of technology uses PowerNet with high-order activation functions, and introduces RePU activation functions instead of ReLU activation functions, thereby introducing high-order ideas into neural networks to achieve accurate approximation of high-order polynomials. However, the number of hidden layers in the neural network and the number of neurons in each layer will increase exponentially with the increase of the polynomial order, leading to problems such as gradient vanishing. Summary of the invention

[0006] The object of the present invention is to provide a data processing method, device and equipment.

[0007] To achieve the above-mentioned object of the invention, the present invention provides a data processing method, device and equipment, the method comprising the following steps:

[0008] a. Receive data that needs to be solved for partial differential equations, build high-order neural networks and training and testing data sets;

[0009] b. Converting the partial differential equation into a minimum optimization problem, and constructing a loss function and an optimization algorithm for the high-order neural network;

[0010] c. training the high-order neural network and testing it on the test data set;

[0011] d. Input the data into the high-order neural network, solve the partial differential equation, obtain the solution and output it.

[0012] According to one aspect of the present invention, in step (a), the high-order neural network includes a fully connected input layer, a nonlinear layer, a hidden layer and an output layer in the order of forward propagation;

[0013] The basis function of the high-order polynomial space of the nonlinear layer is a Lagrange interpolation basis function with Gauss-Lobatto integration nodes as interpolation points:

[0014]

[0015] Among them, the interpolation point x j ,j=1,2,…p+1 is determined by Gauss-Lobatto integral; i represents the serial number of basis function;

[0016] The high-order neural network including the nonlinear layer is expressed as:

[0017]

[0018] Among them, σ represents the activation function; F l ,1≤l≤L represents the hidden layer, F l (x) = W l x+b l ; θ={W l ,b l} represents the set of all trainable parameters; W l is the weight; b l is the bias vector; L is the network depth; p is the order of the high-order neural network; T p (·) indicates the introduced nonlinear transformation;

[0019] The nonlinear transformation transforms the input x into a high-order polynomial space Q p (R d ) Among them, R represents the set of real numbers; d is the dimension of the space.

[0020] According to one aspect of the present invention, in step (a), the training data set is N randomly selected data from the calculation area Ω according to a uniform distribution. r coordinate points and N randomly selected points on the boundary of the computational region Ω bc coordinate points and reselect them in each iteration;

[0021] The test data set is the coordinate points selected according to the equidistant grid on the calculation area Ω and the true solution Find the function value at the coordinate point.

[0022] According to one aspect of the present invention, in step (b), the loss function includes the residual error of the numerical solution obtained by forward propagation inside the region and the error on the boundary, which is expressed as:

[0023] loss=loss res +β·loss bcs ;

[0024] Among them, loss res is the residual of the numerical solution inside the region f is the right side of the equation; loss bcs is the error of the numerical solution at the boundary g is the value of the true solution at the boundary; β adaptively updates the gradient information of the parameter θ through the residual inside the region and the error at the boundary, which is Indicates the derivative of loss with respect to parameters;

[0025] The optimization algorithm is the Adam algorithm, and a learning rate decay strategy is set. The initial value of the learning rate is 0.001, and it decays to 90% of the original value every 1000 steps.

[0026] According to one aspect of the present invention, in step (c), training the high-order neural network comprises the following steps:

[0027] c1. Obtain the loss function through forward propagation;

[0028] c2. Back-propagating the loss function to obtain the gradient information of the high-order neural network parameters;

[0029] c3. Inputting the gradient information into the optimization algorithm to update the parameters of the high-order neural network;

[0030] c4. Repeat steps (c1) to (c3) until the error meets the stop condition or the training is completed.

[0031] According to one aspect of the present invention, in the step (c1), the parameters of the high-order neural network are initialized by Xavier initialization;

[0032] The data of the training data set is input into the high-order neural network, and forward propagation is performed to obtain the numerical solutions inside and at the boundary of the region, which are respectively denoted as u r and u bc , calculate and obtain the loss function;

[0033] In the step (c4), the high-order neural network is trained repeatedly 50,000 times.

[0034] According to one aspect of the present invention, in step (c), the test data set is input into the high-order neural network to obtain a numerical solution, and the relative error between the numerical solution and the true solution is calculated to complete the test of the performance of the high-order neural network.

[0035] A data processing device comprising:

[0036] A model building module, which is used to receive data and build a high-order neural network and training and testing data sets;

[0037] Algorithm building blocks for building loss functions and optimization algorithms for high-order neural networks;

[0038] Model training module, used to train high-order neural networks and test them on test datasets;

[0039] The equation solving module is used to input data into a high-order neural network to solve partial differential equations and output the solution results.

[0040] The device comprises a storage medium and a processor, wherein the storage medium stores a computer program, and is characterized in that the computer program implements a data processing method when executed by the processor.

[0041] According to the concept of the present invention, the high-order ideas in the finite element method are introduced into the deep neural network, and a high-order neural network is constructed to obtain stronger approximation ability, so as to improve the ability of the neural network to solve high-frequency problems.

[0042] According to a solution of the present invention, a nonlinear transformation layer with fixed parameters is added between the input layer and the first hidden layer of the neural network, so as to convert the input of the network into the basis function of the high-order polynomial space, so as to form a high-order polynomial space to accurately represent any polynomial whose order is not higher than the order of the nonlinear layer, thereby avoiding the problem of gradient disappearance caused by too many hidden layers, and also allowing the neural network to simultaneously approximate different frequency information of the target function during the training process. It can also be seen from this that the present invention does not perform high-frequency transfer, but because the introduction of the nonlinear layer can well characterize the high-frequency information, so different from the characteristics of the existing neural network that gradually fits the target function from low frequency to high frequency, the high-order neural network can almost learn the low-frequency component and the high-frequency component at the same time when approximating a function with a wide frequency range, that is, simultaneously approximate the high-frequency and low-frequency components of the solution, thereby accelerating the convergence of the high-frequency function fitting under the same parameter quantity, so that the convergence speed of the high-order neural network is faster than that of the existing neural network.

[0043] According to one solution of the present invention, for high-frequency partial differential equations, a high-order neural network can simultaneously learn the low-frequency and high-frequency information of the target function during the training process, and can obtain a smaller relative error under the same training parameters, while the relative error on the test set decreases with the increase of the order, so that the approximation error is at least two orders of magnitude lower than that of the existing neural network, thereby achieving an improvement in approximation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flowchart schematically showing a data processing method according to an embodiment of the present invention;

[0045] Figure 2 A network structure diagram schematically showing a high-order deep neural network according to an embodiment of the present invention;

[0046] Figure 3 Schematically showing the minimum test set error diagram that can be achieved by the existing neural network pinn and the high-order neural network for solving the high-frequency Poisson equation under different network architectures;

[0047] Figure 4 Schematically represents the loss function and error convergence process of the existing neural network pinn and the high-order neural network for solving the high-frequency Poisson equation;

[0048] Figure 5 A schematic diagram showing the fitting effect of the existing neural network pinn and the high-order neural network for solving the high-frequency Poisson equation. DETAILED DESCRIPTION

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments. The embodiments cannot be described one by one here, but the embodiments of the present invention are not therefore limited to the following embodiments.

[0051] See also Figure 1 The data processing method for solving high-frequency partial differential equations quickly and accurately based on a high-order neural network of the present invention first receives the data that needs to be solved for the partial differential equation, builds a high-order neural network and a training data set and a test data set, then converts the partial differential equation into a minimum optimization problem, and builds the corresponding loss function and optimization algorithm of the high-order neural network, then trains and saves the high-order neural network model, tests it on the test data set, and finally inputs the received data into the high-order neural network to solve the high-frequency partial differential equation, obtains the solution result and outputs it. The method of the present invention is described in detail in one embodiment below. In this embodiment, the high-frequency partial differential equation is a high-frequency Poisson equation with a Dirichlet boundary condition, which is expressed as:

[0052] -Δu=f,x∈Ω;

[0053]

[0054] in, Ω=[0,1]×[0,1]; △u is the Laplace operator; Indicates the boundary of the region. Here, the method of the present invention is described by taking the constructed high-frequency Poisson equation as an example. In practical problems, the high-order neural network method provided by the present invention can be used to deal with various high-frequency problems in the field of scientific computing, such as high-frequency function fitting commonly used in electromagnetic wave and seismic wave modeling problems, solving high-frequency Helmholtz equations, solving high-frequency inverse problems, etc.

[0055] See also Figure 2, the high-order neural network includes a fully connected input layer, a nonlinear layer, a hidden layer and an output layer in the order of forward propagation. Therefore, the idea of ​​high-order finite elements is integrated, and the input is converted into a basis function of a high-order polynomial space through a nonlinear layer. Regarding the selection of one-dimensional basis functions, since in the high-order finite element method, the monomial basis function becomes more and more ill-conditioned (i.e., the condition number is too large to be solved) as the number of parameters increases, the basis function of the high-order polynomial space of the nonlinear layer of the present invention is a Lagrange interpolation basis function with Gauss-Lobatto integration nodes as interpolation points:

[0056]

[0057] Among them, the interpolation point x j ,j=1,2,...,p+1 is determined by Gauss-Lobatto integral; i is the serial number of the basis function.

[0058] A high-order neural network with L hidden layers is represented as:

[0059]

[0060] Wherein, σ represents the activation function (the hyperbolic tangent activation function Tanh is used in this embodiment); F l ,1≤l≤L represents the hidden layer, F l (x) = W l x+b l ; θ={W l ,b l} represents the set of all trainable parameters; W l is the weight; b l is the bias vector; L is the depth of the network.

[0061] From the above, it can be seen that in order to introduce high-order ideas, the present invention adds a nonlinear layer between the input layer and the hidden layer to construct a high-order neural network, and the high-order neural network is expressed as:

[0062]

[0063] Where p is the order of the high-order neural network; T p (·) is the introduced nonlinear transformation. The nonlinear transformation transforms the input x into a high-order polynomial space Q p (R d ) R represents the set of real numbers; d is the dimension of the space. This method of constructing the basis functions of high-dimensional space through the tensor product of one-dimensional basis functions is the method used to construct the basis functions of quadrilateral and hexahedral units in the field of finite element analysis. In this way, by introducing a nonlinear layer on the basis of a fully connected neural network, a high-order neural network with p=2, d=2 is obtained. The input is converted into the basis function of the high-order polynomial space through the nonlinear layer, so that the high-order neural network without any hidden layer can approximate any high-order polynomial almost without error.

[0064] In this embodiment, the training data set is a certain number of coordinate points selected from the inside and the border of a given calculation area according to uniform distribution. Specifically, N points are randomly selected from the inside of the calculation area Ω according to uniform distribution. r = 5000 coordinate points, randomly select N points on the boundary of the calculation area Ω in the same way bc = 1000 × 4 coordinate points, the above coordinate points (data points) are used as training data sets, and are reselected in each iteration to ensure that the network has strong enough generalization ability. The test data set is an equidistant grid of points selected on a given calculation area. Specifically, 101x100 coordinate points are selected on the calculation area Ω according to an equidistant grid, and according to the given true solution The function values ​​at these discrete coordinate points are obtained, and these coordinate points and the corresponding function values ​​are used as test data sets to test the network after training. In practical application problems, the selection of data sets is different. For example, for the modeling problem of high-frequency electromagnetic waves, we usually input the numerical values ​​of the waves measured by the machine and their corresponding positions as training sets into the high-order neural network. If the accuracy of the high-order neural network needs to be tested, we often use the high-precision finite element method to calculate the numerical solutions of the waves at a few positions to approximate the true solution to test the performance of the network.

[0065] In the present invention, the loss function includes the residual of the numerical solution obtained by forward propagation inside the region and the error on the boundary, which is expressed as:

[0066] loss=loss res +β·loss bcs ;

[0067] Among them, loss res is the residual of the numerical solution inside the region f is the right side of the equation; loss bcs is the error of the numerical solution at the boundary g is the value of the true solution at the boundary; β adaptively updates the gradient information of the parameter θ through the internal residual of the region and the error at the boundary, specifically: Indicates the derivative of loss with respect to parameters.

[0068] The optimization algorithm for training high-order neural networks constructed in the present invention is the Adam algorithm, that is, the Adam optimizer in the stochastic gradient descent algorithm is used, and a suitable learning rate decay strategy is set. Specifically, the initial value of the learning rate is set to 0.001, and it decays to 90% of the original value every 1000 steps.

[0069] In the present invention, when training a high-order neural network, the parameters of the constructed high-order neural network are first initialized. The initialization method of this embodiment is Xavier initialization. Then the data of the training data set is input into the high-order neural network, and forward propagation is performed to obtain the numerical solutions inside the region and at the boundary, which are respectively denoted as u and r and u bc , and the above loss function is calculated. The loss function is then back-propagated to obtain the gradient information of the network parameters. At the same time, the obtained parameter gradient information is input into the above-constructed optimization algorithm (i.e., Adam optimizer) to update the parameters of the high-order neural network. Finally, the network training process of forward propagation-backward propagation-parameter update is repeated until the error meets the shutdown condition or the training is completed. In this embodiment, the high-order neural network is trained 50,000 times.

[0070] In the present invention, the test on the test data set is to input the pre-constructed test data set into the saved high-order neural network to obtain a numerical solution, and calculate the relative error between the numerical solution and the true solution to test the performance of the high-order neural network. In this embodiment, the relationship between the order of the high-order neural network and the neural network architecture, the influence of the order of the high-order neural network on the error convergence speed and the effect on the solution of high-frequency partial differential equations are explored respectively, and the results are compared with those of the existing physics-informed neural network (PINN).

[0071] See also Figure 3 , the relationship between the order of high-order neural networks and the network architecture is explored through numerical experiments. This embodiment considers the existing neural networks and high-order neural networks (p=3,7) when Depth=1,2,3, Width=50,100,150,200,300, etc., where Depth and Width represent the number of hidden layers and the number of neurons in each layer in the high-order neural network, respectively. The results show that the high-order neural network performs significantly better than the existing neural network when the network architecture is fixed, and the error decreases significantly with the increase of the order. In addition, the introduction of high order will increase the network parameters. In this regard, this embodiment compares the performance of existing neural networks and high-order neural networks in solving high-frequency Poisson equations under the condition of fixed network parameters. Obviously, the performance of high-order neural networks is significantly better than that of existing neural networks.

[0072] See also Figure 4 , explore the impact of the addition of higher orders on the convergence speed of the error in the time domain. This implementation method fixes the network architecture and the amount of training set data, and gives the loss function, training error, and test error as the training progresses. It can be seen that the loss and error of the existing neural network remain basically unchanged during the training process, which shows that the existing neural network is basically unable to fit the true solution of high frequency. However, the high-order neural network can quickly reach a relatively low error in the initial stage of training, and the overall convergence speed is much better than the existing neural network. Moreover, with the increase of the order, the error convergence speed is also significantly improved.

[0073] See also Figure 5 , explore the influence of the order of high-order neural network on the solution effect of high-frequency partial differential equations. This implementation method fixes the network architecture and the amount of training data, and gives the fitting effect of the existing neural network and the high-order neural network on the true solution of high-frequency partial differential equations. It can be seen that the existing neural network is basically unable to fit the local oscillation in the true solution, but the high-order neural network can fit it very well, indicating that the high-order neural network has a strong ability to capture high-frequency information in high-frequency partial differential equations.

[0074] The data processing device of the present invention includes: a model building module for receiving data and building a high-order neural network and a training data set and a test data set; an algorithm building module for building a loss function and an optimization algorithm for the high-order neural network; a model training module for training the high-order neural network and testing it on the test data set; an equation solving module for inputting data into the high-order neural network, solving partial differential equations, and outputting the solution results. In this example, the data processing device is the open source python machine learning library pytorch, but the data processing device of the present invention is not limited to pytorch.

[0075] The device of the present invention includes a storage medium and a processor, wherein the storage medium stores a computer program, and the computer program implements the data processing method when executed by the processor. In this example, all our calculations are implemented on 64-bit Ubuntu 20.04, using NVIDIA 3070 GPU.

[0076] In summary, in the data processing method based on high-order neural network of the present invention, the high-order neural network can quickly and accurately solve high-frequency partial differential equations, and at the same time also greatly improves the approximation ability of the network, so that the method has a convergence speed and approximation accuracy far exceeding that of existing neural networks, and can solve high-frequency partial differential equations with high precision, and can be applied to solve various high-frequency problems generated in science and engineering, such as high-frequency seismic wave equations and high-frequency Helmholtz equations generated in the field of electromagnetism.

[0077] The above is only one embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data processing method comprising the following steps: a. Receive data that needs to be solved for partial differential equations, and construct a high-order neural network and a training data set and a test data set; the partial differential equation is a high-frequency seismic wave equation or an electromagnetic high-frequency Helmholtz equation; b. Convert the partial differential equation into a minimum optimization problem, and construct the corresponding loss function and optimization algorithm; c. training the high-order neural network and testing it on the test data set; d. inputting the data into the high-order neural network, solving the partial differential equation, obtaining the solution and outputting it; In the step (a), the high-order neural network includes a fully connected input layer, a nonlinear layer, a hidden layer and an output layer in the order of forward propagation; The basis function of the high-order polynomial space of the nonlinear layer is a Lagrange interpolation basis function with Gauss-Lobatto integration nodes as interpolation points: Among them, the interpolation point x j ,j=1,2,...,p+1 is determined by Gauss-Lobatto integration; i is the serial number of the basis function; The high-order neural network including the nonlinear layer is expressed as: Among them, σ represents the activation function; F l ,1≤l≤L represents the hidden layer, F l (x) = W l x+b l ; θ={W l ,b l } represents the set of all trainable parameters; W l is the weight; b l is the amount of bias term; L is the number of hidden layers; p is the order of the high-order neural network; Tp(·) is the introduced nonlinear transformation; The nonlinear transformation transforms the input x into a high-order polynomial space Q p (R d ) Where R represents the set of real numbers; d is the dimension of the space; In step (a), the training data set is N randomly selected data from the calculation area Ω according to uniform distribution. r coordinate points and N randomly selected points on the boundary of the computational region Ω bc coordinate points and reselect them in each iteration; The test data set is the coordinate points selected according to the equidistant grid on the calculation area Ω and the true solution The function value at the coordinate point is obtained.

2. The method according to claim 1, characterized in that In the step (b), the loss function includes the residual error of the numerical solution obtained by forward propagation inside the region and the error on the boundary, which is expressed as: loss=loss res +β·loss bcs ; Among them, loss res is the residual of the numerical solution inside the region f is the right side of the equation; loss bcs is the error of the numerical solution at the boundary g is the value of the solution on the boundary condition; β adaptively updates the gradient information of the parameter θ through the residual inside the region and the error at the boundary, which is Indicates the derivative of loss with respect to parameters; The optimization algorithm is the Adam algorithm, and a learning rate decay strategy is set. The initial value of the learning rate is 0.001, and it decays to 90% of the original value every 1000 steps.

3. The method according to claim 1, characterized in that In the step (c), training the high-order neural network comprises the following steps: c1. Obtain the loss function through forward propagation; c2. Back-propagating the loss function to obtain the gradient information of the high-order neural network parameters; c3. Inputting the gradient information into the optimization algorithm to update the parameters of the high-order neural network; c4. Repeat steps (c1) to (c3) until the error meets the stop condition or the training is completed.

4. The method according to claim 3, characterized in that In the step (c1), the parameters of the high-order neural network are initialized by Xavier initialization; The data of the training data set is input into the high-order neural network, and forward propagation is performed to obtain the numerical solutions inside and at the boundary of the region, which are respectively denoted as u r and u bc , calculate and obtain the loss function; In the step (c4), the high-order neural network is trained repeatedly 50,000 times.

5. The method according to claim 1, characterized in that In the step (c), the test data set is input into the high-order neural network to obtain a numerical solution, and the relative error between the numerical solution and the true solution is calculated to complete the test of the performance of the high-order neural network.

6. A device for implementing the data processing method according to any one of claims 1 to 5, characterized in that: include: A model building module, which is used to receive data and build a high-order neural network and training and testing data sets; Algorithm building blocks for building loss functions and optimization algorithms for high-order neural networks; Model training module, used to train high-order neural networks and test them on test datasets; The equation solving module is used to input data into a high-order neural network to solve partial differential equations and output the solution results.

7. A device comprising a storage medium and a processor, wherein the storage medium stores a computer program, characterized in that: When the computer program is executed by the processor, the data processing method according to any one of claims 1 to 5 is implemented.

Images