Methods and systems for training a neural network to solve a physical problem using the finite element method (FEM)

Abstract

A finite element method (FEM) mesh and formulation representative of a physical system modelled using differential equation(s) (DE) describing the physical problem are input into a neural network (NN) configured for generating a NN output representative of a trial solution of the DE, followed by inputting the NN output into an automatic differentiation unit configured to generate a differentiated output comprising flux terms at boundary points of the FEM mesh and DE residuals at quadrature points of the FEM mesh, inputting the differentiated output into a multiplication unit configured to apply filters to the differentiated output to generate multiplication outputs, the filters having associated therewith test functions comprising nonvanishing value(s) at the boundary points, generating based on the multiplication outputs a loss function comprising the flux terms, evaluating the loss function and iteratively updating parameter(s) of the NN to minimize the loss function and obtain a trained NN.

Description

METHODS AND SYSTEMS FOR TRAINING A NEURAL NETWORK TO SOLVE A PHYSICAL PROBLEM USING THE FINITE ELEMENT METHOD (FEM)CROSS-REFERENCE TO RELATED APPLICATION(S)

[0001] The present application claims priority on United States Patent Application No. 63 / 673,842 filed July 22, 2024, the entire contents of which are incorporated herein by reference.FIELD

[0002] The improvements generally relate to the field of neural network training, and more precisely to training a neural network to solve a physical problem using the Finite Element Method (FEM).BACKGROUND

[0003] Physics-Informed Neural Networks (PINNs) have received significant interest due to their capabilities of solving partial differential equations. This opened the door to explore innovative methods to integrate PINNs with traditional numerical methods to improve their abilities in solving highly nonlinear and complex problems.

[0004] PINNs are applied in different areas where they were found to model and predict the behaviors of the systems better than traditional methods in some cases. Indeed, PINNs have a number of advantages including achieving machine learning with little or no data, training a model to respect known physical equations, providing a versatile method of numerical solution which can be combined with data from experiments or high-fidelity simulation, and handling ill-posed problems and solving inverse-problems. For example, for a given structure deformation, PINNs may be used to determine what the load is or, provided with the wave propagation equation and point measurements, what the spatial distribution of material properties that fits is.

[0005] However, conventional PINNs have a number of drawbacks including that they do not scale up easily to large problems, are slow to train, require finicky tuning to make them work, and are hard to converge to high precision.

[0006] Therefore, improvements are needed.SUMMARY

[0007] In accordance with one aspect, there is provided a computer-implemented method for training a neural network to solve a physical problem. The method comprises, at a computing device, inputting into the neural network a finite element method (FEM) mesh and a FEM formulation representative of a physical system modelled using at least one differential equation describing the physical problem, the FEM mesh comprising a plurality of finite elements defined by a plurality of boundary points and a plurality of quadrature points, the neural network configured for generating, based on the FEM mesh and FEM formulation, a neural network output representative of a trial solution of the at least one differential equation, inputting the neural network output into an automatic differentiation unit configured to apply automatic differentiation to the neural network output to generate a differentiated output, the differentiated output comprising a plurality of flux terms at the plurality of boundary points and a plurality of residuals of the differential equation at the plurality of quadrature points, inputting the differentiated output into a multiplication unit configured to apply a plurality of filters to the differentiated output to generate a plurality of multiplication outputs, the plurality of filters having associated therewith a plurality of test functions comprising at least one nonvanishing value at the plurality of boundary points, generating a loss function based on the plurality of multiplication outputs, the loss function comprising the plurality of flux terms, evaluating the loss function, and iteratively updating, based on an outcome of the evaluating, at least one parameter of the neural network to minimize the loss function and obtain a trained neural network.

[0008] In at least one embodiment in accordance with any previous / other embodiment described herein, the residuals comprise weak-form residuals.

[0009] In at least one embodiment in accordance with any previous / other embodiment described herein, the weak-form residuals comprise forcing terms at the plurality of quadrature points.

[0010] In at least one embodiment in accordance with any previous / other embodiment described herein, the weak-form residuals comprise at least one operator of the differential equation.

[0011] In at least one embodiment in accordance with any previous / other embodiment described herein, the residuals comprise ultra weak-form residuals.

[0012] In at least one embodiment in accordance with any previous / other embodiment described herein, the method further comprises forming the plurality of filters by computing a product of the plurality of test functions and a plurality of quadrature weights.

[0013] In at least one embodiment in accordance with any previous / other embodiment described herein, the method further comprises forming the plurality of filters by computing a product of a derivative of the plurality of test functions and a plurality of quadrature weights.

[0014] In at least one embodiment in accordance with any previous / other embodiment described herein, the filters are formed in local coordinates.

[0015] In at least one embodiment in accordance with any previous / other embodiment described herein, the quadrature points are Gauss quadrature points.

[0016] In at least one embodiment in accordance with any previous / other embodiment described herein, the filters are applied to the differentiated output to evaluate Gauss quadrature sums for each finite element and for each of the plurality of test functions.

[0017] In at least one embodiment in accordance with any previous / other embodiment described herein, the loss function is generated using a Petrov-Galerkin framework in which each test function is a combination of at least one of Legendre polynomials, Lagrange polynomials, Chebyshev polynomials, hyperbolic functions, trigonometric functions, and an output from a separate neural network.

[0018] In at least one embodiment in accordance with any previous / other embodiment described herein, the test functions comprise one of linear test functions, quadratic test functions, cubic test functions, quartic test functions, and high-order polynomials.

[0019] In at least one embodiment in accordance with any previous / other embodiment described herein, generating the loss function based on the plurality of multiplication outputs comprises grouping the plurality of multiplication outputs to formulate a plurality of residual loss tensors for the plurality of finite elements and the plurality of test functions, and computing an element-wise square, a sum, and an average of the plurality of loss tensors for the plurality of finite elements and the plurality of test functions, thereby generating the loss function.

[0020] In at least one embodiment in accordance with any previous / other embodiment described herein, evaluating the loss function comprises determining, based on the loss function, whether the PINN satisfies at least one criterion.

[0021] In at least one embodiment in accordance with any previous / other embodiment described herein, the at least one parameter of the neural network is iteratively updated to minimize the loss function in response to determining that the neural network fails to satisfy the at least one criterion.

[0022] In at least one embodiment in accordance with any previous / other embodiment described herein, the trained neural network is obtained and exported in response the determining that the neural network satisfies the at least one criterion.

[0023] In at least one embodiment in accordance with any previous / other embodiment described herein, the at least one criterion is a convergence criterion.

[0024] In at least one embodiment in accordance with any previous / other embodiment described herein, the trained neural network is used at least one of to solve a forward problem, to solve an inverse problem, to solve an optimization problem, to perform parameter identification, and to generate reduced-order models for creation of digital twins.

[0025] In at least one embodiment in accordance with any previous / other embodiment described herein, the method for training the neural network to solve the physical problem is applied in one of space, time, and parameter space.

[0026] In at least one embodiment in accordance with any previous / other embodiment described herein, when the method is applied in space, inputting the FEM mesh comprisesinputting one of a one-dimensional (1 D) mesh, a two-dimensional (2D) mesh, and a three- dimensional (3D) mesh.

[0027] In accordance with another embodiment, there is provided a system for training a neural network to solve a physical problem. The system comprises a processing unit and a non-transitory computer-readable medium having stored thereon program instructions executable by the processing unit for inputting into the neural network a finite element method (FEM) mesh and a FEM formulation representative of a physical system modelled using at least one differential equation describing the physical problem, the FEM mesh comprising a plurality of finite elements defined by a plurality of boundary points and a plurality of quadrature points, the neural network configured for generating, based on the FEM mesh and FEM formulation, a neural network output representative of a trial solution of the at least one differential equation, inputting the neural network output into an automatic differentiation unit configured to apply automatic differentiation to the neural network output to generate a differentiated output, the differentiated output comprising a plurality of flux terms at the plurality of boundary points and a plurality of residuals of the differential equation at the plurality of quadrature points, inputting the differentiated output into a multiplication unit configured to apply a plurality of filters to the differentiated output to generate a plurality of multiplication outputs, the plurality of filters having associated therewith a plurality of test functions comprising at least one nonvanishing value at the plurality of boundary points, generating a loss function based on the plurality of multiplication outputs, the loss function comprising the plurality of flux terms, evaluating the loss function, and iteratively updating, based on an outcome of the evaluating, at least one parameter of the neural network to minimize the loss function and obtain a trained neural network.

[0028] Many further features and combinations thereof concerning embodiments described herein will appear to those skilled in the art following a reading of the instant disclosure.DESCRIPTION OF THE FIGURES

[0029] In the figures,

[0030] Fig. 1 is a block diagram illustrating a prior art PINN, in accordance with one embodiment;

[0031] Fig. 2 is a block diagram illustrating a prior art convolutional variational PINN (VPINN), in accordance with one embodiment;

[0032] Fig. 3 is a block diagram illustrating a method for training a neural network to solve a physical problem using FEM in one-dimensional domains, in accordance with one embodiment;

[0033] Fig. 4 illustrates a uniform FEM mesh for use with the method illustrated in Fig. 3, in accordance with one embodiment;

[0034] Fig. 5 illustrates the construction of filters for use with the method illustrated in Fig. 3, in accordance with one embodiment;

[0035] Fig. 6 is a block diagram illustrating a method for training a neural network to solve a physical problem using FEM in two-dimensional domains, in accordance with one embodiment;

[0036] Fig. 7 is a schematic diagram of a satellite panel whose heat transfer conditions are determined using the method illustrated in Fig. 3, in accordance with one embodiment;

[0037] Fig. 8 shows plots illustrating the thermal response of the satellite panel of Fig. 7, in accordance with one embodiment;

[0038] Fig. 9 shows additional plots illustrating the thermal response of the satellite panel of Fig. 7, in accordance with one embodiment;

[0039] Fig. 10 is a schematic diagram of a two-dimensional section of a turbine blade whose heat transfer conditions are determined using the method illustrated in Fig. 3, in accordance with one embodiment;

[0040] Fig. 11 shows plots illustrating the temperature distribution inside the two- dimensional section of the turbine blade of Fig. 10, in accordance with one embodiment;

[0041] Fig. 12 is a flowchart of a method for training a neural network to solve a physical problem using FEM, in accordance with one embodiment; and

[0042] Fig. 13 is a block diagram of an example computing device, in accordance with one embodiment.

[0043] It will be noted that throughout the appended drawings that like features are identified by like reference numerals.DETAILED DESCRIPTION

[0044] Described herein is an improved FEM method. It is proposed herein to integrate FEM, a well-established robust numerical method, with the neural network (NN) framework to build a new method (referred to herein as the “Finite Element Neural Network Method” or FENNM) that can be applied to linear and nonlinear systems, and used for parametric identification and optimization purposes among other applications. In particular, the method described herein bridges FEM with PINNs, allowing to leverage the strengths of both methods.

[0045] As understood by those skilled in the art, FEM is considered as the cornerstone of numerical simulation in engineering and applied mathematics due to its accuracy and reliability. FEM has a well-defined theory that facilitates straightforward implementation across various FEM formulations. This motivated the development automated codes that can be rapidly adapted to a wide range of applications while maintaining consistency and high accuracy. However, FEM requires well-posed problems with predefined parameters, loading, and boundary conditions. Moreover, FEM results are typically compared to experimental measurements only after the simulation.

[0046] Whereas classical FEM must follow a rigorous mathematical solving approach to obtain a well-posed problem, the FENNM method proposed herein opens new solutions paradigms. For example, the proposed method can solve inverse problems (e.g., solving for a load instead of a deflection) in addition to being able to calculate forward solutions, optimization problems (e.g., topology optimization, dimensioning, etc.), perform parameter identification, and generate lightweight reduced-order models to be used in the development of digital twins. As understood by those skilled in the art, an inverse problem refers to the process of determining, based on a set of observations, the causal factors that produced theobservations. Examples of inverse problems that may be solved using FENNM include, but are not limited to, metal or polymer forming, 4D printing, design of deploying structures such as arterial stents, robots, and aeroelastic tailoring of composite wings.

[0047] As understood by those skilled in the art, a digital twin refers to a virtual replica (also referred to a virtual asset) of a well-defined physical system (or asset), where both systems communicate with each other in real-time (e.g., via a communication system that collects and processes data between both systems). It is desirable for the virtual replica to be validated by measurement data collected from the physical system through sensors. Once validated, both assets integrate through the communication channel in real-time, where insights can be drawn from the virtual replica about how the physical system behaves in various situations. In particular, the physical system provides real-time sensor data, while the virtual replica continuously updates its parameters based on the sensor data provided by the physical system, which facilitates understanding the behavior of the physical system in several scenarios.

[0048] Another key aspect of the proposed method is the ability to incorporate solution parameters as variables which makes the method an instant library of solutions for a range of values. The originality of the method described herein is its ability to solve forward problems without relying on any data like conventional neural networks, and solving inverse problems without committing to the tedious manipulation that has to be done with traditional techniques. The method described herein is therefore computationally efficient compared to existing techniques, allowing to achieve significant improvements in the functioning (i.e. the performance) of the computer used to solve the physical problem at hand. The method described herein can be considered as a tool that is adaptable to various applications in different fields which demonstrate its potential for widespread impact.

[0049] As will be described further below, in one embodiment, FENNM is based on the Petrov-Galerkin framework, where a nonlinear NN output represents the global trial solution that is used to compute partial differential equation (PDE) residuals, while nonvanishing FEM shape functions are used as test functions. Using the nonvanishing FEM shape functions ensures that the boundary terms in weak-form of the PDE do not vanish and are explicitly learned. In contrast to existing methods, the boundary terms guarantee that the residuals aresummed to zero after completion of training. The boundary terms act as constraining terms that steer the NN towards the solution, thus improving the optimization. Moreover, the boundary terms are applicable for imposing Dirichlet, Neumann, and intermediate boundary conditions within the residual loss function using a single NN, thus minimizing the number of competing loss terms and the complexity of the NN model. In one embodiment, the weak-form of the residual loss function is then approximated using the Gauss quadrature rule. Finally, multiplication (e.g., convolution) operations are used to perform the summations over the Gauss points for all elements in the domain, which is done in parallel for all test functions.

[0050] As understood by those skilled in the art, FEM is a technique that allows to obtain numerical approximations for differential equations that represent physical systems that are usually subjected to external loads. FEM allows to provide approximations to complex problems that are difficult to solve using other techniques. This is due to the fact that the finite element solution can be used repeatedly for all elements in the same FEM mesh and adapted to different problems with minimal modifications. The problem statement of an illustrative onedimensional (1-D) equilibrium problem is as follows:

[0051] with x as the independent variable. The essential and natural boundary conditions are, respectively: :

[0052] In FEM, the solution to a differential equation is approximated by choosing a trial function from a finite-dimensional space and reducing the residuals of the equation by weighting them with a set of test functions. The trial functions and the test functions belong to a linear space where they are the same in the Galerkin framework and are distinct in the Petrov-Galerkin framework, producing different numerical schemes depending on the choiceof the trial functions. The FEM formulation of Equation (1) using the Galerkin method for one element is:

[0053] where <]>k(x) is the kth test function, and U(x) is the trial solution. The flux term is evaluated at the element left boundary xnand the right boundary xn+i, where n indicates the current element. Equation (1) will be used as an example to demonstrate the FENNM method described herein.

[0054] In a more general setting, nonlinear trial functions extend the approximations to a nonlinear space, resulting in a more robust estimation with sparser representation and reduced computational cost. Nonlinear approximation approaches can include radial basis functions, dictionary learning, neural networks, and adaptive splines. However, while nonlinear approximation introduces additional capabilities, its nonlinear nature may bring additional complications, and achieving an optimal approximation rate can become challenging, especially in high-dimensional spaces.

[0055] Turning now to neural networks (NNs), NNs transform high-dimensional input into output through algebraic operations and non-linear mapping. They function as an optimization method by iteratively adjusting their parameters to minimize a loss function that measures the discrepancy between the network output and high-fidelity data. A key advantage of NNs is their ability to represent a wide range of nonlinear functions using a relatively small number of parameters.

[0056] Although NNs are not inherently data-driven, when used to fit data, they are unaware of the mathematical model expressing physical laws. Hence, they require a large amount of high-fidelity data to achieve accurate and reliable predictions. This becomes problematic in small-data regimes, where the available data is insufficient relative to the complexity of the system. Constructing physics-informed learning machines replaces the large data requirementby embedding the prior information of the differential equations into NNs, which led to the development of PINNs.

[0057] Fig. 1 is a block diagram 100 illustrating the working of a prior art PINN, as illustrated in: Tartakovsky, A. M., Marrero, C. O., Perdikaris, P., Tartakovsky, G. D., & Barajas-Solano, D. (2020). Physics-informed deep neural networks for learning parameters and constitutive relationships in subsurface flow problems. Water Resources Research, 56(5), e2019WR026731. PINN includes the differential equation as a residual term at random collocation points in the computational domain, which acts as a penalizing term constraining the space of solutions. Hence, inferring the solution of the differential equation is transformed into an optimization problem of the residual term which acts as a loss function at the penalizing points generated at minimal cost. To construct a PINN for the equilibrium problem defined in Equation (1), the total loss function £ is expressed in the strong-form for Equations (1) and (2) as follows:

[0058] where the subscript NN represents the NN output for that differential operator. The penalty parameters TS, TBare for the residual loss term £ and the boundary conditions loss term respectively. The penalty parameters can be manually selected before training based on simple observations. An effective approach is to express them as ratios, with the ratio determined by the specific problem. Alternatively, the parameters can be automatically adjusted during the training process. The Minimax weighting algorithm updates the network weights and biases using gradient descent while applying gradient ascent to the penalty parameters. The adaptive loss weighting algorithm evaluates the average convergence speed of each loss term and assigns higher weights to the slower terms. The collocation points N-Rare randomly sampled constructing the computational domain, and the number of boundary points NBis chosen before training.

[0059] Despite the potential of PINNs in tackling forward problems, they suffer from limitations that hinder their efficiency compared to classical methods like FEM. The loss function of the network consists of different terms, as shown in Equation (4), which can cause convergence problems as optimization becomes highly non-convex. Furthermore, vanilla PINNs struggle to converge and provide an accurate approximation for stiff problems that contain solutions with sharp changes in space.

[0060] Some techniques address the computational domain of PINNs, such as adaptive sampling strategies based on residual-based adaptive distribution. Other methods divide the spatial domain into discrete subdomains such as conservative PINNs (cPINN), in which separate PINNs are applied in each subdomain while enforcing the flux continuity along the interfaces. The parallelization powers of cPINN are extended in extended PINNs (XPINN) to the spatial and temporal domains for all types of differential equations, reducing training and computational costs. However, in all these formulations, the strong form of mathematical models is employed at random collocation points, requiring a large number of points to guarantee convergence. Although cPINN and XPINN provide parallelization capabilities, they introduce additional layers of complexity. The use of multiple networks complicates the hyperparameter tuning for each network and the challenge of connecting the networks along the interfaces. The Finite Basis PINNs (FBPINNs) address the issue of connectivity between networks at the interfaces using overlapping subdomains. They replace the differential equation with an ansatz representation which includes distance functions to account for the boundary conditions, hence eliminating the need for boundary loss terms at the interfaces. A key advantage of this method is that it divides a large PINN optimization problem into many smaller optimization problems in subdomains. However, using distance functions inside ansatz representations requires prior knowledge of the systems, which limits the generalizability of the method. Moreover, the method requires implementing many NNs simultaneously, which can be expensive and complicate the nature of the total loss function affecting optimization.

[0061] Incorporating the weighted residual of the differential equation to construct a variational loss function results in VPINNs. Fig. 2 is a block diagram 200 illustrating the working of a prior art convolutional VPINN, as illustrated in: Liu, C., & Wu, H. (2023). cv-PINN: Efficient learning of variational physics-informed neural network with domain decomposition. Extreme Mechanics Letters, 63, 102051. The VPINN of Fig. 2 has a number of limitations including being only applied in the spatial domain, the test functions not being customized, using the strong form of the PDEs, and being applied on toy problems only. In VPINNs, the loss function is developed within the Petrov-Galerkin framework, where the test functions belong to a linear space and are a combination of Legendre polynomials, while the nonlinear approximation of the NN represents the trial solution. The weak formulation reduces the regularity required in the network output by lowering the operator orders in the loss function. This approach reduces automatic differentiation computations when generating the loss terms and reduces the computational cost by minimizing backpropagation processes. The residual loss term of Equation (4) becomes:

[0062] where (.,.) indicates the inner product between the kth test function <]>k(x), k = 1 , 2, ... K and the residual of the differential equation, where K represents the number of test functions and Q is the computational domain. A variant of VPINN is the variational neural network (VarNet), which employs piecewise linear test functions of FEM in the variational formulation. Both VPINN and VarNet compute the weak-form integral over the whole domain, which reduces their approximating capabilities in complex domains.

[0063] VPINNs were extended to consider the domain decomposition in hp-VPINN, where the test functions are defined locally over nonoverlapping elements, and the NN represents the global nonlinear trial solution. The hp-refinement provides the flexibility of domain decomposition by applying h-refinement using variable element sizes with a projection onto a space of high order polynomials as p-refinement, making hp-VPINN the first method to approximate solutions like FEM by discretizing the domain into elements and incorporating hp-refinement techniques. Equation (5) becomes:

[0064] where the number of elements n = 1 , ... , Nei. There is no analytical solution for the integrals of the weighted residuals, and numerical integration techniques such as the Gauss quadrature rule is used. Hence, the residual loss in Equation (6) is approximated to:

[0065] where the qth quadrature point is located at position xqin the element and Wqis the corresponding quadrature weight using Q number of quadrature points. The test functions can be defined in local or global coordinates and the necessary transformations must be applied when evaluating the residual loss function. VPINNs require fewer quadrature points than the collocation points used in PINNs, decreasing the computational cost. Increasing the number of elements yields higher accuracy gains than increasing the quadrature points within each element, indicating that h-refinement is more effective than p-refinement for a given quadrature rule order. However, increasing the number of elements exponentially increases the computational cost in hp-VPINN, rendering it inefficient for approximating complex functions and thereby limiting its practical applications.

[0066] The computational cost in hp-VPINN is addressed by applying convolution operations to compute the strong-form weighted residual loss in cv-PINN. The product of the test functions and the quadrature weights form convolution filters passing over the strong-form residuals. By evaluating the loss function with convolution operations (e.g., in TensorFlow), rather than sequentially looping over elements for each test function as done in hp-VPINN, cv- PINN gains the advantage of parallelizing the training process. The residual loss term can be written as:

[0067] is a matrix of the convolution filters. As shown in Equation(8), the convolution operation simultaneously performs the integral approximation using the Gauss quadrature rule on all test functions.

[0068] VPINN, hp-VPINN, and cv-PINN use a combination of Legendre polynomials as test functions, which vanish at the element boundaries, leading to a loss of flux information across the elements. Moreover, they employ a large number of high-order test functions and quadrature points, failing to optimize their selection and, thereby, adding additional computational burden. The order of the test functions and their influence in selecting an adequate number of quadrature points remained an open question.

[0069] To overcome the challenges mentioned above, the instant disclosure presents FENNM using the convolution operations introduced in cv-PINN. As will be noted further below, compared to conventional PINNs, the FENNM method described herein requires smaller network architectures. In one embodiment, for a simple physical problem, the proposed FENNM method uses one (1) layer of three (3) neurons, compared two (2) layers of twenty (20) neurons for conventional PINNs. The same benefit may apply to larger problems. In addition, the proposed FENNM method trains faster than conventional techniques. In one embodiment, the proposed FENNM method may train in less than one (1) minute, compared to twenty (20) minutes for conventional PINNs. Moreover, the proposed FENNM method converges to machine precision (e.g., 10-16).

[0070] As will be described further below, FENNM leverages the power of the neural network to replace the trial solution of the finite element formulation. This is done by integrating the neural network with this formulation through the loss function. The neural network, hence, is trained to find the trial solution of the finite element formulation by respecting both the relevant physics equations and the available data (if available). This can be accomplished by utilizing convolution operations (or other suitable multiplication operations, such as using tensor products) which evaluate the weighted residuals of the loss function in parallel, making the method time efficient.

[0071] The FENNM method proposed herein may apply in space, time, and parameter space, as well as with any shape functions. The proposed method may be applied on problemswith space and time dependence. In one embodiment, the proposed method uses the weak form of the PDE to get the same mathematical formulation used in FEM.

[0072] More specifically, in one embodiment, the NN used in FENNM provides the global nonlinear space of solutions, while the test functions belong to the Lagrange test function space, and have at least one nonvanishing value at the element boundaries. Consequently, the information of the flux terms across the elements is now implemented inside the weak- form loss function. This will further constrain the NN towards the solution as it provides additional information inside the residual loss function. This information is missing from existing techniques where the network had to overcompensate to converge, affecting the optimization process during training. Moreover, this can explain the use of many test functions and Gauss points to approximate the residual loss function in existing techniques.

[0073] Although reference is made herein to FENNM using Lagrange test functions, it should be understood that other test functions may apply. It is proposed herein to generate the loss function using the Petrov-Galerkin framework. In some embodiments, each test function belongs to a linear space. It should however be understood that the test functions may comprise linear test functions, quadratic test functions, cubic test functions, quartic test functions, and high-order polynomials. Each test function may be a combination of Legendre polynomials, Lagrange polynomials, Chebyshev polynomials, hyperbolic functions, and / or trigonometric functions. Each test function may also be the output of a separate neural network, different from the one used to implement the FENNM method described herein.

[0074] Fig. 3 presents a schematic diagram of a one-dimensional FENNM solver 300, in accordance with one embodiment. A one-dimensional FEM mesh 302 and a FEM formulation are input into a NN 304 (e.g., a PINN). The FEM mesh 302 is representative of a physical system modelled using at least one differential equation (i.e. PDE) describing a physical problem to be solved using the FENNM solver 300. The FEM mesh 302 comprises a plurality of finite elements defined by a plurality of boundary points 306 and a plurality of quadrature points 308. In one embodiment, the quadrature points 308 are Gauss quadrature points. Any suitable variant of Gauss quadrature including, but not limited to, Gauss-Legendre, Gauss- Lobatto, Gauss-Radau, Gauss-Hermite, or Gauss-Laguerre may apply. The NN 304 is configured to generate an output representative of a trial solution of the differential equation.

[0075] The NN output is then fed into an automatic differentiation unit 310 which is configured to apply automatic differentiation to the NN output to generate a differentiated output (also referred to as “convolution signals”). In particular, in the automatic differentiation step, the NN output is used to generate flux terms (also referred to herein as “fluxes” or “flux values”) 312 at the finite elements’ boundary points 306 and residuals 314 of the differential equation. In one embodiment, the residuals 314 are weak-form residuals including the forcing term at the finite elements’ quadrature points 308 and the differential operators of the wearform differential equation. It should however be understood that the forcing term is optional and need not be included in all cases. In another embodiment, the residuals 314 are ultra weak-form residuals.

[0076] The differentiated output (i.e. the flux terms 312 and residuals 314) generated by the automatic differentiation unit 310 is then input into a multiplication unit 316 which is configured to apply predefined filters (also referred to herein as “convolution filters”) 318a, 318b to the differentiated output. The filters 318a are formed by computing a product of test functions 320 and quadrature weights, whereas the filters 318b are formed by computing a product of the quadrature weights and a derivative of the test functions 320. In one embodiment, the test functions 320 comprise at least nonvanishing value at the boundary points 306. In one embodiment, the predefined filters 318a, 318b (comprising the quadrature weights and the test functions 320 or their derivatives) pass over the flux terms 312 and residuals 314 in a convolution step to construct the residual loss. Specifically, the filters 318a, 318b pass over their corresponding signals to evaluate the Gauss quadrature sums for each finite element and for each test function in the convolution process. The outputs of the multiplication unit 316 (e.g., the convolution outputs) are then grouped to formulate the residual loss tensor for all finite elements and for all test functions. The residual loss tensor is then squared element- wise, summed, and averaged over the number of finite elements and test functions to construct the total residual loss function. Finally, the total residual loss is evaluated and the NN parameters are updated iteratively to minimize the total loss.

[0077] Considering Equation (8), the weak-form per finite element n for the kth test function after integrating by parts is as follows:

[0078] As the convolution filters 318a, 318b are fixed and pass over the output of the NN 304, the test functions 320 are defined in local coordinates. This approach allows for the generalization of the formulation to accommodate meshes as in 302 with adaptive finite element sizes similar to FEM. Hence, Equation (9) becomes:

[0079] where denotes the local coordinate within the interval [-1 , 1] such that x = xn+ Jx(1 + ), and Jx= (xn+1 - xn) / 2 is the one-dimensional Jacobian. The integral terms in Equation (10) are approximated using Gauss quadrature rule as:

[0080] An additional benefit for employing Lagrange test functions is that they allow for the integration of natural boundary conditions directly into the residual loss function. This integration smooths the total loss function, thus improving the convergence and optimization of FENNM. Moreover, the nonvanishing test function enables the application of intermediate forcing terms within one neural network. The total residual loss function for Neifinite elements and K test functions becomes:

[0081] The total residual loss function in Equation (12) consists of four parts. First, the loss function comprises the fluxes that arise at the boundaries on both the left and right sides of each finite element defined in the global coordinates. The global coordinates define the computational domain, establishing a reference frame for the elements, the quadrature points, the elements’ boundaries, and the boundary conditions.

[0082] Fig. 4 shows a one-dimensional grid 400 divided uniformly into five elements over the domain x e [1 ,2], with four quadrature points per element. In Fig. 4, the circles represent the quadrature points xqdistributed throughout the elements in the global coordinates, the “plus” (or “+”) sign on the left represents the left boundary condition, the “cross” (or “x”) sign on the right represent the right boundary condition, the triangles represent the elements’ right boundaries, and the squares represent the elements’ left boundaries. The grid 400 may be predefined before training begins and includes the points where the FENNM outputs are evaluated to construct the total loss function.

[0083] It should be understood that the number of quadrature points may vary depending on the application. The minimum number q of quadrature points to accurately approximate an integral of a polynomial of order p is defined as q > (p+ 1 ) / 2, where q is also the order of the Gauss quadrature rule. While a forcing term of any kind may be used, it is desirable to use an adequate number q of quadrature points to accurately approximate the forcing term. This precision in turn allows the neural network used in FENNM to minimize the residual loss. Although the FENNM network forms a nonlinear solution space to approximate the differential equation (DE) operators, the specific trial solution it produces is not accessible. Consequently, the number of quadrature points required for the approximation of the DE operators depends on the particular physical problem being addressed. However, a lower limit of quadrature points can be set to ensure an adequate approximation of the residual loss. The upper limit of quadrature points depends on the nature of the forcing term and the complexity of the solution approximated by FENNM. The number of quadrature points can be reduced by discretizing the domain into smaller finite elements, since the complexity of the solution is divided among elements.

[0084] Second, the total residual loss function in Equation (12) comprises the DE operators with lower orders after integration by parts, evaluated in the global coordinates.

[0085] Third, the total residual loss function in Equation (12) comprises the filters, which are (as noted herein above with reference to filters 318a, 318b) the product of the Gauss quadrature weights and the test functions or their derivatives according to the number of integrations by parts performed such as in Filtersl and Filters2 in Equation (12). These filters are computed in the local coordinates to ensure their generalizability over finite elements of different sizes. The construction of the filters is shown in Fig. 5, using linear Lagrange test functions with Q = 4 Gauss points as an example (where markers indicate the positions of Gauss points). In Fig. 5, Filtersl of Equation (12) (i.e. filters Wqd<|)i / d and Wqd<t>2 / d^, respectively) are represented in plots 502 and 504, while Filters2 of Equation (12) (i.e. filters Wq(|)i and Wq<|)2, respectively) are represented in plots 506 and 508. In Fig. 5, dots represent the integration weights Wq, squares represent the test functions and their derivatives, and solid circles represent the filters.

[0086] Fourth, the total residual loss function in Equation (12) comprises the forcing term, which includes any source terms and the remaining DE operators evaluated in the global coordinates.

[0087] In one embodiment, the penalty parametersand TBin the total loss function of Equation (4) are nondecreasing variables and are updated simultaneously during the training process using an optimizer (e.g., the ADAM optimizer). These penalty parameters remain constant during optimization (e.g., using L-BFGS), unless specified otherwise. Loss terms with increasing errors are automatically weighted more, forcing the neural network to minimize them. Hence, the neural network seeks to find a saddle point where it optimizes its parameters during training using gradient descent to minimize total loss, and updates the penalty terms using gradient ascent to maximize their weights.

[0088] Although reference is made herein to FENNM being applied to one-dimensional domains, it should be understood that physical problems in two-dimensional domains (where the second dimension can be space, time, or a parameter) may be solved using FENNM. Problems in three-dimensional domains (e.g., defined using a three-dimensional (3D) mesh) may also apply, although not described and illustrated herein. In other words, FENNM may be applied in space in 1 D, 2D, or 3D. FENNM may also be applied in time, as well as in parameter space, which entails that FENNM can be applied in 1 D, 2D, 3D, 4D, 5D or more. As understoodby those skilled in the art, the term “parameter space” refers to the fact that it is possible to have elements along a parameter dimension. For instance, if, for a given physical problem, the Young’s modulus of a material is not known, it would be possible to have elements along the Young’s modulus dimension so that FENNM is used to solve the physical problem for a whole range of the Young’s modulus parameter.

[0089] Fig. 6 illustrates a schematic representation of a two-dimensional (2D) FENNM solver 600, where a 2D FEM mesh 602 (comprising a plurality of Gauss points 604) is provided to a neural network 606 which generates the PDE operators that are used to construct the weak-form weighted residual and element fluxes using an automatic differentiation unit 608. The filters 610 consist of the Gauss quadrature weights and the test functions with their corresponding partial derivatives. Then, two-dimensional multiplication (e.g., convolution) operations are applied to the weak-form residuals to construct the integral part of the loss function. The elements’ boundary terms consist of fluxes, test functions, and Gauss quadrature weights. These are evaluated via one-dimensional multiplication (e.g., convolution) operations, where the Gauss quadrature weights serve as a one-dimensional filter that passes over the signals obtained by multiplying the fluxes with the test functions. These signals are equivalent to the number of test functions used in the weak formulation. After that, the outputs of the multiplication (e.g., convolution) operations are summed to construct the complete residual loss function map. Finally, the loss terms are multiplied by their corresponding penalty parameters and summed to construct the total loss function.

[0090] In this case, the residual loss can be defined as follows:

[0091] In Equation (13), Filtersl are two-dimensional filters consisting of the test functions and the Gauss quadrature weights, applied to the two-dimensional PDE operators that are notintegrated by parts. Filters2 represent two-dimensional filters consisting of the partial derivative of the test functions and the Gauss quadrature weights, applied to the two- dimensional PDE operators that are integrated by parts in the spatial coordinates. Onedimensional convolution is used to evaluate the boundary terms, where Filters represents the Gauss quadrature weights, and the fluxes comprise the test functions and PDE operators evaluated at the element edges in the temporal domain.

[0092] Case studies will now be described to illustrate real-world applications of the FENNM method described herein. For all case studies, unless specified otherwise, the penalty terms T-Rand T-B and TLare defined as nondecreasing variables and are simultaneously updated during the training process using the ADAM optimizer and remain unchanged when switching to the L- BFGS optimizer. Consequently, loss terms with increasing errors are assigned higher weights, forcing the NN to minimize them. Hence, the optimizers search for a saddle point that reduces the total loss function using gradient descent and gradient ascent to update the penalty terms.

[0093] First, a satellite panel use case will be described. This first example use case considers the solution of a two-dimensional plate, exemplifying the heat transfer conditions of a satellite panel of area 1 x 1 m2exposed to solar radiation and connected to a localized heatgenerating heat load acting as a finite surface source load. The results are compared with a finite-volume simulation executed using TMG, the advanced thermal solver.

[0094] Fig. 7 shows a schematic diagram 700 for the two-dimensional satellite panel subjected to solar radiation and a localized heat load. The energy balance equation in dimensionless form is:

[0095] where T* denotes the dimensionless temperature distribution over the panel. The following dimensionless parameters are used:

[0096] for a characteristic length L = 1 m, characteristic temperature To= 400 K, and characteristic load q0= 500 W ■ m“2. The panel has a thickness of d = 0.005 m, conductivity k = 163 Wm“1K’1, and emissivity = 0.8. The Stefan-Boltzmann constant is o = 5.6690 x 10“8W ■ m“2■ K’4and the environmental temperature is Tspace = 0 K. To address the steady-state Equation (14), it was assumed that the solar radiation remains constant. The solar radiation and the heat load are defined, respectively, as follows:

[0097] where q”oiar= 1.377805x103 W-n2is the solar flux constant and a = 0.47 is the absorptivity of the panel. The system is considered adiabatic at all boundaries.

[0098] In this example, the aim is to predict the thermal behavior of the panel in a forward setting, followed by an inverse problem formulation. As will be seen further below, FENNM can be used to solve inverse and ill-posed problems, where the unknown absorptivity and the thermal behavior of the panel are identified using one data point.

[0099] The weak-form loss function of Equation (14) takes the following form :

[0100] The adiabatic boundary conditions mean that the heat flux at the boundaries is zero. One can impose these boundary conditions inside the residual loss function using nonvanishing FEM shape functions. Hence, the total loss function is reduced to one term provided by Equation (17), which further improves the NN optimization.

[0101] Fig. 8 (a) shows the FENNM solution using 20 x 20 elements and the percentage error compared to the TMG finite-volume simulation using 100 x 100 elements in Fig. 8 (b) for the heat transfer problem described in Equation (17). FENNM predicts the system’s thermal behavior with a 0.036% error using a relatively coarser mesh size. As anticipated, the maximum error is located at the location of the local heat load. This occurs because of the discontinuous nature of the heat load, which subsequently induces sharp changes in the solution within that region. PINNs often struggle with abrupt variations and discontinuities in solutions, which can hinder convergence in different areas of the domain. In contrast, FENNM takes advantage of relaxed regularity constraints and the integration of boundary fluxes, enabling the NN to achieve convergence while maintaining precision across the rest of the domain.

[0102] The inverse solution to the satellite panel example will now be described. In a plausible application scenario, it is reasonable to assume that the panel’s absorptivity of the coatings a degrades over time in a non-trivial manner due to sun exposure. Therefore, it is desirable to simultaneously predict its value along with the thermal behavior of the system. In the context of space applications, one often relies on a limited number of sensors, making it beneficial to develop models capable of predicting system unknowns from limited datasets.

[0103] To solve for a using FENNM, it is defined as a tensor variable with an initial guess in Equation (17) that is directly substituted in Equation (5.17). Then, the data loss function is defined using the one data point reading collected from the ideal sensor such that T*sensor(0.73, 0.68) = 0.87 as follows:

[0105] Fig. 9 illustrates the FENNM estimate of the thermal response of the satellite panel along with the predicted absorptivity value a. A stability analysis was performed to verify that FENNM consistently converges to the same solution by running the NN ten times with different initializations. Fig. 9 (a) and (b) show the average FENNM estimate of the thermal response of the system and the percentage error, respectively. FENNM can reconstruct the solution throughout the entire domain using one data point for this heat transfer problem. Fig. 9 (c) shows the average evolution of the prediction a per iteration with a confidence interval of 99.7%. The ADAM optimizer has a higher fluctuation than the L-BFGS optimizer. However, ADAM is rapidly approaching the true value of a. The average training history illustrated in Fig. 9 (d) shows that FENNM reaches convergence around 10,000 iterations and reaches machine precision for the data loss term.

[0106] In this case study, the efficacy of FENNM was demonstrated in addressing two- dimensional heat transfer problems subjected to radiation and an external load. The problem is formulated in the weak-form where all the boundary conditions are imposed inside one loss function which is possible due to using the FEM nonvanishing shape functions. Despite the high nonlinearity of the PDE, the nonlinear nature of the trial solution generated by the NN captures the features of the solution using linear shape functions and a relatively coarse mesh. Moreover, FENNM can effectively reconstruct the entire solution using one data point and identify the unknown parameter for the same problem.

[0107] A second case study, which relates to using FENNM to determine heat transfer in a turbine blade, will now be described. In particular, it will be seen from this second example use case that FENNM can be seamlessly extended to complex domains using the standard FEM nonvanishing test functions. The two-dimensional section of a turbine blade shown in the plot 1000 of Fig. 10 will be considered. For simplicity of implementation, this turbine blade is considered a 2D body. It is sought to compute the temperature distribution arising from different temperatures imposed at the boundaries. The heat conduction is governed by the Laplace equation, as follows:

[0108] where k = 6.7 Wm-1 K-1. The temperatures at the boundaries of the body must respect Dirichlet boundary conditions, as follows: rright= mu c.^bottom=’jO C ■Ttop = Tlcft= 7’> ((21)

[0109] and are imposed with a boundary loss function:

[0110] The dimensionless form of Equations (20) and (21) can be obtained by using similar dimensionless parameters as in Equation (15). Hence, the strong-form residual loss function becomes:

[0111] where |J| denotes the determinant of the Jacobian matrix to transfer the coordinates from the global coordinate to the local coordinate.

[0112] Fig. 11 illustrates a comparison between the TMG finite-volume solution (a) (see plot 1100) and the FENNM prediction in (b) (see plot 1102) for the temperature distribution inside the turbine blade. A single computational mesh was generated using the Gmsh mesh generator and used for both methods to enable consistent comparison. The FENNM model satisfies the boundary conditions along the outer boundaries of the domain, despite the intricate geometry on the left and right sides. Fig. 11 (c) (see plot 1104) shows the temperature variation for the TMG and FENNM solutions along the line x = 0.07 m. The prediction of FENNM satisfies the boundary conditions at the top and bottom of the domain. However, despite a slight deviation in temperature between the two solutions within the interior of the domain, FENNM captures the inflection point similarly to that in the TMG solution.

[0113] From the foregoing, it can be seen that FENNM is capable of working with irregular meshes generated by tools like Gmsh. In particular, the use of convolution operations to compute the weighted residual within the FENNM framework for non-standard geometries was demonstrated, thus opening up possibilities for its adaptation to more complex and real-world scenarios.

[0114] Since numerical simulations can be found in multiple applications, it should be understood that the systems and methods described herein may be used in various applications other than those illustrated by the use cases of Fig. 7 and Fig. 10. Indeed and as previously noted herein, FENNM may be used for formulating and solving numerous real- world engineering problems, such as so-called “inverse problems” (e.g., parameter identification or optimization) and cases with uncertain or missing boundary conditions. If some information is known about an experimental setup or an actual machine or physical system in operation (e.g., a thrust bearing in a hydraulic turbine in service), FENNM can solve to fit measured quantities while respecting the known laws of physics. For instance, FENNM maybe used to solve an inverse problem, where a physical parameter such as the viscosity of the fluid is unknown. For this purpose, commercial FEM results may be used as so-called “synthetic experimental results” and FENNM may be trained to learn to solve the mathematical equations of the inverse problem and minimize the error of its predictions with respect to the synthetic experimental results by tuning the viscosity value.

[0115] In yet another example use case, FENNM may be used to simulate the coupled heat transfer and lubrication film of a sector of a thrust bearing having a given geometry, the thrust bearing supporting a hydraulic turbine. In this case, FENNM may first be used to solve the heat transfer on the given geometry. The weak form of the Reynolds hydrodynamic lubrication equation may then be derived and implemented in FENNM to solve the oil film problem on a geometry similar to the given geometry. The two problems of heat transfer and hydrodynamic lubrication may then be coupled together into a single FENNM implementation, amounting to coupling the governing equations together and solving for a vector of unknowns rather than a scalar field. Potential impacts for a specific application for hydraulic thrust bearing monitoring for hydroelectric turbine operators would be to enable more accurate health monitoring of their electricity generating units through a more complete interpretation of limited sensor information, hence allowing for optimal maintenance predictions and better asset management. This means a more reliable electrical grid.

[0116] FENNM may also be used in applications and industries other than those related to the energy sector. FENNM may indeed be applied in multiple fields of engineering, as well as finance and certain biological computations and modelling. FENNM may in particular be applied to problems involving large amounts of data typically found in artificial intelligence (Al) or small amounts of data, more typical of real life engineering problems. In addition, the proposed method may address linear or nonlinear models. For example, the systems and methods described herein may be used for non-destructive inspection, reverse engineering problems, reconstructing a full picture from incomplete data obtained from experiments or high-fidelity simulations, topology optimization, multi-grid simulation etc.

[0117] The systems and methods described herein may also be used for edge computing applications including, but not limited to, on robotic assembly lines, heavy mining equipment, freight ships, satellites, or other expensive engineering systems. FENNM may indeed be usedfor the development of a versatile computational framework capable of integrating both sensor data and physics-based modelling for form the kernel of a digital twin technology that can be deployed in any industry. As understood by those skilled in the art, while conventional simulation methods are effective at solving well-posed problems with full knowledge of parameters, boundary and initial conditions, there are, in practical applications, gaps in such settings and uncertainty on these inputs, leading to important discrepancies. A digital twin could consider these imperfections by combining sensor data and physics-based simulation. All states of a given physical system could be precomputed on a supercomputer and deployed as a trained FENNM neural network for the resulting digital twin to interpret the signals of onboard sensors with the physical equations governing the system dynamics to paint a complete picture of its state, health, and performance. In some embodiments, wind turbines, solar farms, hydroelectric turbines, and other low green-house gas emission sources, may be made more cost-effective through optimized operations made possible by digital twins. For example, the thrust bearing of a hydroelectric generating unit with the heat transfer happening in its lubrication film is a system that would greatly benefit from a digital twin as it can be difficult to measure parameters of such a system directly, yet monitoring the system can be very informative for maintenance predictions and preventing expensive downtime. Digital twins also hold the potential to revolutionize personalized medicine. Whether it is for adapting orthopedic prostheses to a patient’s bone structure, evaluating the operability of a patient’s aneurysm, or predicting the growth of a tumor, if the physical problem can be put into equations, then FENNM can be used to solve these equations while matching the medical imaging.

[0118] Fig. 12 illustrates a method 1200 (i.e. FENNM) for training a neural network (e.g., a PINN) to solve a physical problem. The method comprises, at step 1202, inputting into the neural network a FEM mesh (1 D, 2D, or 3D, as described herein) and a FEM formulation representative of the physical system modelled using at least one differential equation describing the physical problem. The FEM mesh comprises a plurality of finite elements defined by a plurality of boundary points and a plurality of quadrature points. The neural network is configured for generating, based on the FEM mesh and FEM formulation, a neural network output representative of a trial solution of the at least one differential equation.

[0119] Step 1204 comprises inputting the neural network output into an automatic differentiation unit configured to apply automatic differentiation to the neural network output to generate a differentiated output, the differentiated output comprising a plurality of flux terms at the plurality of boundary points and a plurality of residuals of the differential equation at the plurality of quadrature points.

[0120] Step 1206 comprises inputting the differentiated output into a multiplication (e.g., convolution) unit configured to apply a plurality of filters to the differentiated output to generate a plurality of multiplication outputs. The plurality of filters has associated therewith a plurality of test functions comprising at least one nonvanishing value at the plurality of boundary points.

[0121] Step 1208 comprises generating a loss function based on the plurality of multiplication outputs, the loss function comprising the plurality of flux terms. At step 1210, the loss function is evaluated and at step 1212, at least one parameter of the neural network is iteratively updated, based on an outcome of the evaluating performed at step 1210, to minimize the loss function and obtain a trained neural network.

[0122] Evaluating the loss function at step 1210 comprises determining, based on the loss function, whether the PINN satisfies at least one criterion (e.g., a convergence criterion). The at least one parameter of the neural network PINN is iteratively updated at step 1212 to minimize the loss function in response to determining that the neural network PINN fails to satisfy the at least one criterion. The trained neural network PINN is then obtained and exported in response the determining that the neural network PINN satisfies the at least one criterion.

[0123] In some embodiments, parts or all of the components FENNM solver 300, 600 and / or part or all of the steps of the method 1200 are implemented using a computing device 1300, as illustrated in Fig. 13. The computing device 1300 comprises a processing unit 1302 and a memory 1304 which has stored therein computer-executable instructions 1306. The processing unit 1302 may comprise any suitable devices configured to cause a series of steps to be performed such that instructions 1306, when executed by the computing device 1300 or other programmable apparatus, may cause functions / acts / steps described herein to be executed. The processing unit 1302 may comprise, for example, any type of general-purposemicroprocessor or microcontroller, a digital signal processing (DSP) processor, a CPU, an integrated circuit, a field programmable gate array (FPGA), a reconfigurable processor, other suitably programmed or programmable logic circuits, or any combination thereof.

[0124] The memory 1304 may comprise any suitable known or other machine-readable storage medium. The memory 1304 may comprise non-transitory computer readable storage medium, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. The memory 1304 may include a suitable combination of any type of computer memory that is located either internally or externally to device, for example random-access memory (RAM), read-only memory (ROM), electro-optical memory, magneto-optical memory, erasable programmable read-only memory (EPROM), and electrically-erasable programmable read-only memory (EEPROM), Ferroelectric RAM (FRAM) or the like. Memory 1304 may comprise any storage means (e.g., devices) suitable for retrievably storing machine- readable instructions 1306 executable by processing unit 1302.

[0125] The FENNM method 1200 described herein may be implemented, in part or entirely, in a high level procedural or object oriented programming or scripting language, or a combination thereof, to communicate with or assist in the operation of a computer system, for example the computing device 1300. Alternatively, the FENNM method 1200 may be implemented in assembly or machine language. The language may be a compiled or interpreted language.

[0126] Embodiments of the FENNM method 1200 described herein may also be considered to be implemented by way of a non-transitory computer-readable storage medium having a computer program stored thereon. The computer program may comprise computer-readable instructions which cause a computer, or more specifically the processing unit 1302 of the computing device 1300, to operate in a specific and predefined manner to perform the functions described herein.

[0127] Computer-executable instructions may be in many forms, including program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particulartasks or implement particular abstract data types. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0128] FENNM may advantageously be integrated into existing commercial finite element software such as ANSYS™, Simcenter™, NASTRAN™, Abaqus™ and other software of the like. In particular, existing software may be modified to add the possibility of solving physical problems using FENNM. This integration can leverage the preprocessing tool that engineers are already familiar with (e.g., the same meshing software and user interface for problem definition as used in FEM), eliminating the need for new software installation or training. Embedding FENNM into these platforms allows engineers to solve problems using the systems and methods described herein without disrupting their current workflows. This compatibility also facilitates regulatory certification processes by maintaining the use of trusted and validated software environments.

[0129] From the foregoing, it can be seen that FENNM improves on existing solutions, at least regarding the manner in which boundary and initial conditions are defined. Using the weak-form, FENNM allows for natural boundary conditions, also known as Neumann boundary conditions, to be imposed directly in the residual equation of the FEM formulation. In this manner, initial conditions (e.g., related to velocity and position for a given physical system) can be imposed. Using the ultra-weak form, FENNM allows for natural and essential boundary conditions (i.e. Dirichlet and Neumann boundary conditions) to be imposed directly in the residual equation. In this manner, all initial conditions (e.g., related to velocity and position in the given physical system) can be imposed. Mixed conditions (e.g., Robin-type) may also be imposed using the weak-form or the ultra-weak form, depending on the application. Being able to use FENNM to impose boundary conditions also allows to impose temporary loads within the solution domain (e.g., modeling of internal point loads or sources), which cannot be achieved using existing techniques such as PINN. For instance, FENNM may be used to add a temporary load in a structural problem, or a thermal source may be added in a heat transfer problem. Furthermore, FENNM may, in some embodiments, allow for intelligent FEM mesh refinement. For this purpose, the weak-form of equations may be used for training the neural network while the strong form residual equation is not used. In this manner, the strong form residual equation can be evaluated a posteriori (i.e. post training of the neural network) toobtain a proxy of the error distributed in the solution domain, thereby refining the FEM mesh at certain points. The computation of the weak form residual equation may then be used to implement automated mesh refining techniques. This makes FENNM particularly suitable for developing and implementing physical solutions that are both accurate and computationally efficient.

[0130] In some embodiments, the proposed method shifts the complexity from the training to the implementation. Once the physical problem is implemented correctly, seamless training may be achieved with little to no tuning. In addition, simple architectures and low numbers of quadrature points are sufficient for the proposed method, thus improving computational and time efficiencies. The proposed FENNM method is also versatile when applied to different cases and implementation of new problems requires limited modifications. Moreover, promising extensions, such as Nitsche’s method, are possible.

[0131] The advantages of FENNM can be extended to the possibility of merging FENNM with more sophisticated and existing FEM solvers in the sense that the problem setup remains the same. However, instead of running a classical simulation to compute the desired quantity, the NN is trained to approximate the solution directly. Once training is complete, FENNM provides the information necessary for engineers to perform post-processing steps. In addition, within the same solver, the NN can be replaced with a different architecture and retrained independently on new data to update its parameters without disrupting the rest of the solver. As a result, since the solution resides in the NN and not on the grid, it will be straightforward to perform multigrid training. This is because once the strong-form residual is computed for a certain grid, a new grid is proposed directly with local mesh refinement without performing an interpolation between the grid points.

[0132] Furthermore, the FENNM framework is developed for one-dimensional and two- dimensional domains that include time and parameter as additional dimensions. This implies that FENNM can be scaled to higher dimensional (e.g., 3D) domains as a natural extension and use it for tasks that involve uncertainty quantification and sensitivity analysis by treating the model’s parameters as additional dimensions. Hence, FENNM can efficiently predict how the variations of these parameters, boundary conditions, or source terms affect the solution without requiring repeated training or using different neural network architectures.

[0133] The structure of the loss function in FENNM is well-suited for multi-physics problems. FENNM can tackle coupled PDEs simultaneously within a unified framework by integrating the governing equations into a joint loss function as in vector-valued PDEs, thereby facilitating smooth interactions among the physical fields.

[0134] The FENNM model could be trained offline for a range of parameters or operating and boundary conditions using all the required computational resources. Once the FENNM is trained, it acts as a lightweight model consisting of weights and biases. This makes it well- suited for applications where computational, storage, or power resources are limited, such as in edge computing. In addition, it could generalize to a range of parameters, loads, or boundary conditions, eliminating the need to retrain at the edge. Therefore, FENNM can deliver predictions about system updates, parameters, or changes in properties in real-time at the edge using data collected from sensors directly, without the need to store large libraries of solutions or run heavy simulations for comparison.

[0135] Ultimately, the FENNM framework provides a strong foundation for digital twin technology, among other applications. Compared with the previous techniques, FENNM provides solutions that are consistent with the governing laws of physics with minimal tuning and for a wider range of applications, thanks to the benefits gained from the weak-form and the nonvanishing test functions. This is crucial for digital twin applications where reliability, predictability, and interoperability are required for safety and sustainability reasons. In addition, by having the NN as a function approximator, one ensures that the model can update its parameters to better match the evolution of physical assets over time using new sensor data.

[0136] Thus, in some embodiments, the advantages provided by the method proposed herein include, but are not limited to, the fact that it allows to tackle nonlinearities (i.e. to solve non-linear problems) without prior treatments or simplifications, may be easily used for parameter identification, can be applied for parameter space and sensitivity analysis, is well- suited for ill-posed problems and optimization problems such as topology optimization, can be applied for reverse engineering problems, can be applied for non-destructive testing, can serve as a base model for digital twins’ technology, has a simpler structure than PI NN, and requires less tuning than PINN. Since FEM is applied in all fields of engineering, the proposedmethod may also be applied to all sectors of engineering including, but not limited to, forward solution software, design software, energy sector, manufacturing, economics and finance, and communication. The possible commercial benefits of the proposed method include, but are not limited to, better product development (by having a more detailed description of the systems), better product qualities and reliability, optimization and customization of solutions, predictive maintenance, and competitive advantages.

[0137] The above description is meant to be exemplary only, and one skilled in the art will recognize that changes may be made to the embodiments described without departing from the scope of the invention disclosed. Still other modifications which fall within the scope of the present invention will be apparent to those skilled in the art, in light of a review of this disclosure.

[0138] Various aspects of the systems and methods described herein may be used alone, in combination, or in a variety of arrangements not specifically discussed in the embodiments described in the foregoing and is therefore not limited in its application to the details and arrangement of components set forth in the foregoing description or illustrated in the drawings. For example, aspects described in one embodiment may be combined in any manner with aspects described in other embodiments. Although particular embodiments have been shown and described, it will be apparent to those skilled in the art that changes, and modifications may be made without departing from this invention in its broader aspects. The scope of the following claims should not be limited by the embodiments set forth in the examples but should be given the broadest reasonable interpretation consistent with the description as a whole.

Claims

WHAT IS CLAIMED IS:

1. A computer-implemented method for training a neural network to solve a physical problem, the method comprising: at a computing device: inputting into the neural network a finite element method (FEM) mesh and a FEM formulation representative of a physical system modelled using at least one differential equation describing the physical problem, the FEM mesh comprising a plurality of finite elements defined by a plurality of boundary points and a plurality of quadrature points, the neural network configured for generating, based on the FEM mesh and FEM formulation, a neural network output representative of a trial solution of the at least one differential equation; inputting the neural network output into an automatic differentiation unit configured to apply automatic differentiation to the neural network output to generate a differentiated output, the differentiated output comprising a plurality of flux terms at the plurality of boundary points and a plurality of residuals of the differential equation at the plurality of quadrature points; inputting the differentiated output into a multiplication unit configured to apply a plurality of filters to the differentiated output to generate a plurality of multiplication outputs, the plurality of filters having associated therewith a plurality of test functions comprising at least one nonvanishing value at the plurality of boundary points; generating a loss function based on the plurality of multiplication outputs, the loss function comprising the plurality of flux terms; evaluating the loss function; and iteratively updating, based on an outcome of the evaluating, at least one parameter of the neural network to minimize the loss function and obtain a trained neural network.

2. The method of claim 1 , wherein the residuals comprise weak-form residuals.

3. The method of claim 2, wherein the weak-form residuals comprise forcing terms at the plurality of quadrature points.

4. The method of claim 2, wherein the weak-form residuals comprise at least one operator of the differential equation.

5. The method of claim 1 , wherein the residuals comprise ultra weak-form residuals.

6. The method of any one of claims 1 to 5, further comprising forming the plurality of filters by computing a product of the plurality of test functions and a plurality of quadrature weights.

7. The method of any one of claims 1 to 5, further comprising forming the plurality of filters by computing a product of a derivative of the plurality of test functions and a plurality of quadrature weights.

8. The method of claim 6 or 7, wherein the filters are formed in local coordinates.

9. The method of any one of claims 1 to 8, wherein the quadrature points are Gauss quadrature points.

10. The method of claim 9, wherein the filters are applied to the differentiated output to evaluate Gauss quadrature sums for each finite element and for each of the plurality of test functions.

11. The method of any one of claims 1 to 10, wherein the loss function is generated using a Petrov-Galerkin framework in which each test function is a combination of at least one of Legendre polynomials, Lagrange polynomials, Chebyshev polynomials, hyperbolic functions, trigonometric functions, and an output from a separate neural network.

12. The method of any one of claims 1 to 11 , wherein the test functions comprise one of linear test functions, quadratic test functions, cubic test functions, quartic test functions, and high-order polynomials.

13. The method of any one of claims 1 to 12, wherein generating the loss function based on the plurality of multiplication outputs comprises: grouping the plurality of multiplication outputs to formulate a plurality of residual loss tensors for the plurality of finite elements and the plurality of test functions; and computing an element-wise square, a sum, and an average of the plurality of loss tensors for the plurality of finite elements and the plurality of test functions, thereby generating the loss function.

14. The method of any one of claims 1 to 13, wherein evaluating the loss function comprises determining, based on the loss function, whether the PINN satisfies at least one criterion.

15. The method of claim 14, wherein the at least one parameter of the neural network is iteratively updated to minimize the loss function in response to determining that the neural network fails to satisfy the at least one criterion, further wherein the trained neural network is obtained and exported in response the determining that the neural network satisfies the at least one criterion.

16. The method of claim 14 or 15, wherein the at least one criterion is a convergence criterion.

17. The method of any one of claims 1 to 16, wherein the trained neural network is used at least one of to solve a forward problem, to solve an inverse problem, to solve an optimization problem, to perform parameter identification, and to generate reduced- order models for creation of digital twins.

18. The method of any one of claims 1 to 17, wherein the method for training the neural network to solve the physical problem is applied in one of space, time, and parameter space.

19. The method of any one of claims 1 to 18, wherein, when the method is applied in space, inputting the FEM mesh comprises inputting one of a one-dimensional (1 D) mesh, a two-dimensional (2D) mesh, and a three-dimensional (3D) mesh.

20. A system for training a neural network to solve a physical problem, the system comprising: a processing unit; and a non-transitory computer-readable medium having stored thereon program instructions executable by the processing unit for: inputting into the neural network a finite element method (FEM) mesh and a FEM formulation representative of a physical system modelled using at least one differential equation describing the physical problem, the FEM mesh comprising a plurality of finite elements defined by a plurality of boundary points and a plurality of quadrature points, the neural network configured for generating, based on the FEM mesh and FEM formulation, a neural network output representative of a trial solution of the at least one differential equation; inputting the neural network output into an automatic differentiation unit configured to apply automatic differentiation to the neural network output to generate a differentiated output, the differentiated output comprising a plurality of flux terms at the plurality of boundary points and a plurality of residuals of the differential equation at the plurality of quadrature points; inputting the differentiated output into a multiplication unit configured to apply a plurality of filters to the differentiated output to generate a plurality of multiplication outputs, the plurality of filters having associated therewith a plurality of test functions comprising at least one nonvanishing value at the plurality of boundary points; generating a loss function based on the plurality of multiplication outputs, the loss function comprising the plurality of flux terms;evaluating the loss function; and iteratively updating, based on an outcome of the evaluating, at least one parameter of the neural network to minimize the loss function and obtain a trained neural network.

Images