Mixed precision controller and method for the same

Abstract

A Mixed Precision controller configured to operate in a High-Performance Computing, HPC, system, where the HPC system includes a high-precision processor array and a low-precision processor array. The Mixed Precision controller is configured to receive a system of partial differential equations and utilize the high-precision processor array and the low-precision processor array to solve the system of partial differential equations utilizing a fixed-point iterative scheme, g. The Mixed Precision controller is characterized in that the Mixed Precision controller is configured to utilize a defect correction term which is defined as the difference between a high-precision residual, RH, and a low-precision residual, RL, where the defect correction term is added to an independent constants vector of the system of partial differential equations. The use of the MP controller in the HPC system enables high-precision accuracy of final solution while reducing computational cost and overall execution time of non-linear partial differential equations.

Description

[0001] MIXED PRECISION CONTROLLER AND METHOD FOR THE SAME

[0002] TECHNICAL FIELD

[0003] The present disclosure relates generally to the field of computing systems; and more specifically, to a mixed precision controller and a method for use in the mixed precision controller.

[0004] BACKGROUND

[0005] In the past decade, a transformative shift in computational paradigms has occurred, propelled by the ascendancy of accelerators, such as Graphics Processing Units (GPUs), Neural Processing Units (NPUs), and Tensor Processing Units (TPUs). These accelerators have emerged as indispensable tools for a wide range of applications, particularly in the realm of Artificial Intelligence (Al) and scientific computing. The contemporary landscape of data centres reflects a notable departure from conventional architectures, embracing a heterogeneous amalgamation of Central Processing Units (CPUs), the GPUs, and other accelerators, such as NPUs, to reach and cross the exa-scale barrier. The aforementioned accelerators are typically designed to conduct computations at low floating-point precision. Such strategic design choice serves the dual purpose of mitigating memory access bottlenecks and maximizing Floating Point Operations per Second (FLOPs) within the confines of limited silicon area. However, such designs have technical issues for most scientific applications, which require higher precision, specifically, double / FP64 floating-point precision, to yield the results that are not only practically usable but are also meaningful. Examples of such scientific applications include simulation of turbulent flows, combustion, weather forecast, molecular dynamics, and the like.

[0006] Currently, certain attempts have been made to obtain the maximum utilization of aforementioned accelerators for scientific applications, such as use of mixed precision algorithms. The mixed precision algorithms are typically used to solve linear Partial Differential Equations (PDEs). While solving the linear PDEs, the mixed precision algorithms utilize iterative refinement to converge to high precision accuracy. Conventionally, for non-linear PDEs, linearization is required to use iterative refinement. However, the linearization of non-linear PDEs incurs an additional computational complexity and usually, results in somewhat overall loss of accuracy. Thus, there exists a technical problem of inefficient and computationally intensive methods used for solving non-linear PDEs.

[0007] Therefore, in light of the foregoing discussion, there exists a need to overcome the aforementioned drawbacks associated with the conventional methods of solving the non-linear PDEs.

[0008] SUMMARY

[0009] The present disclosure provides a mixed precision controller and a method for use in the mixed precision controller. The present disclosure provides a solution to the existing problem of inefficient and computationally intensive methods used for solving the non-linear PDEs. An aim of the present disclosure is to provide a solution that overcomes at least partially the problems encountered in prior art, and provide an improved mixed precision controller and a method for use in the mixed precision controller.

[0010] The object of the present disclosure is achieved by the solutions provided in the enclosed independent claims. Advantageous implementations of the present disclosure are further defined in the dependent claims. In one aspect, the present disclosure provides a Mixed Precision, MP, controller configured to operate in a High-Performance Computing, HPC, system, where the HPC system comprises a high-precision processor array and a low-precision processor array, where the Mixed Precision controller is further configured to receive a system of partial differential equations and utilize the high-precision processor array and the low-precision processor array to solve the system of partial differential equations utilizing a fixed-point iterative scheme, g. where the Mixed Precision controller is characterized in that the Mixed Precision controller is further configured to utilize a defect correction term which is defined as the difference between a high-precision residual, RH. and a low-precision residual, RL. where the defect correction term is added to an independent constants vector of the system of partial differential equations.

[0011] The use of the MP controller in the HPC system enables the high-precision accuracy while reducing the computational cost and overall execution time of non-linear partial differential equations. The MP controller is configured to execute the low- precision floating-point operations along with the high-precision floating-point operations and, the low-precision floating-point operations are executed comparatively faster than the high-precision floating-point operations. Thus, the MP controller enables a faster execution time while solving the system of partial differential equations. The use of the MP controller in the HPC system enables the execution of a heterogeneous architecture combining the high-precision processor array (e.g., CPUs) and the low-precision processor array (e.g., accelerators, such as GPUs, NPUs, TPUs, and the like) on the same node resulting an enhanced performance and higher energy efficiency. Moreover, the use of the MP controller in the HPC system reduces the memory traffic (e.g., from the memory to either the high-precision processor array or to the low-precision processor array) and the network traffic (from one node to another node in the HPC system). Moreover, the use of the MP controller in the HPC system enables a reduction in the memory footprint (i.e., less data is either stored in the memory or retrieved from the memory).

[0012] In an implementation form, the system of partial differential equations is expressed as

[0013] Ax = b, where A denotes a discrete differential operator matrix, x denotes a vector of unknowns, and b denotes the vector of independent constants, wherein the MP controller is further configured to assign the defect-correction term, D, to be 0, determine an initial guess, x°, for the high-precision processor array, and advance the solution on the high-precision processor array for several steps using a computationally cheap method, to improve the original guess, truncate that updated guess x°, for the low-precision processor array, and while no solution of sufficient accuracy has been determined repeatedly for n=l, 2, 3, ... determine the high-precision residual for iteration n, RHn= Axn— b, determine the low-precision residual for iteration n RLn= Axn— b, wherein A denotes a low-precision floating-point evaluation of the discrete differential operator, A, xndenotes a low-precision floating-point evaluation of the unknown variables vector, x, for iteration n, and b denotes a low-precision floating-point evaluation of the independent constants vector, b, determine the defect-correction term for iteration n, Dnas the difference between the high-precision residual and the low-precision residual for iteration n, Dn= RHn— RLn, set xk=0= xn, apply the fixed-point iterative scheme, g. in order to compute an approximate solution for Axk= b — D utilizing the low-precision processor array 0,1, ..., K, determine the high-precision residual for iteration n+1, RHn+1= Axn+1— b, and determine that a solution of sufficient accuracy has been determined as xn+1if the high-precision residual for iteration n, RHn, is below an accuracy threshold.

[0014] The MP controller does not require any linearization of non-linear PDEs for convergence. The MP controller allows the convergence of a given PDE without the overhead of iteratively refining the solution towards the desired level of accuracy. The MP controller enables the utilization of the heterogeneous hardware for scientific HPC applications by retaining the desired precision level of accuracy. The use of the iterative algorithm ensures the faster execution time of the non-linear partial differential equations.

[0015] In a further implementation form, the Mixed Precision controller is further configured to not apply any linearization of the system of partial differential equations.

[0016] The MP controller does not require any linearization of non-linear PDEs and hence, manifests reduced computational complexity and an enhanced energy efficiency.

[0017] In a further implementation form, the high-precision processor array has a precision that is higher than the precision of the low- precision processor array.

[0018] The use of the high-precision processor array maintains the desired accuracy level of final solution of the system of partial differential equations and the use of the low-precision processor array enables a faster execution time of the system of partial differential equations.

[0019] In a further implementation form, the high-precision processor array has a precision of FP64 or higher and the low-precision processor array has a precision of FP16 or FP 32.

[0020] In another aspect, the present disclosure provides a method for a Mixed Precision controller in a High-Performance Computing, HPC, system, comprising a high-precision processor array and a low-precision processor array, where the method comprises receiving a system of partial differential equations and utilize the high-precision processor array and the low-precision processor array to solve the system of partial differential equations utilizing a fixed-point iterative scheme, g. wherein the method is characterized in that the method further comprises utilizing a defect correction term which is defined as the difference between a high-precision residual, RH. and a low-precision residual, RL. wherein the defect correction term is added to an independent constants vector of the system of partial differential equations.

[0021] The method achieves all the technical advantages and effects of the Mixed Precision controller.

[0022] In a yet another aspect, the present disclosure provides a computer program product comprising program instructions for performing the method, when executed by one or more processors in an HPC system.

[0023] The one or more processors in the HPC system achieves all the advantages and effects of the method after execution of the method.

[0024] It is to be appreciated that all the aforementioned implementation forms can be combined.

[0025] It has to be noted that all devices, elements, circuitry, units and means described in the present application could be implemented in the software or hardware elements or any kind of combination thereof. All steps which are performed by the various entities described in the present application as well as the functionalities described to be performed by the various entities are intended to mean that the respective entity is adapted to or configured to perform the respective steps and functionalities. Even if, in the following description of specific embodiments, a specific functionality or step to be performed by external entities is not reflected in the description of a specific detailed element of that entity which performs that specific step or functionality, it should be clear for a skilled person that these methods and functionalities can be implemented in respective software or hardware elements, or any kind of combination thereof. It will be appreciated that features of the present disclosure are susceptible to being combined in various combinations without departing from the scope of the present disclosure as defined by the appended claims.

[0026] Additional aspects, advantages, features and objects of the present disclosure would be made apparent from the drawings and the detailed description of the illustrative implementations construed in conjunction with the appended claims that follow.

[0027] BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The summary above, as well as the following detailed description of illustrative embodiments, is better understood when read in conjunction with the appended drawings. For the purpose of illustrating the present disclosure, exemplary constructions of the disclosure are shown in the drawings. However, the present disclosure is not limited to specific methods and instrumentalities disclosed herein. Moreover, those skilled in the art will understand that the drawings are not to scale. Wherever possible, like elements have been indicated by identical numbers.

[0029] Embodiments of the present disclosure will now be described, by way of example only, with reference to the following diagrams wherein:

[0030] FIG. 1 is a block diagram that illustrates various exemplary components of a High-Performance Computing (HPC) system, in accordance with an embodiment of the present disclosure;

[0031] FIG. 2 is a flowchart of a method of mixed-precision defect correction, in accordance with an embodiment of the present disclosure;

[0032] FIG. 3 illustrates a mathematical representation of a mixed-precision defect correction method, in accordance with an embodiment of the present disclosure;

[0033] FIG. 4 is a flowchart of a method of mixed-precision defect correction, in accordance with another embodiment of the present disclosure;

[0034] FIG. 5 represents an illustration of a mixed-precision defect correction approach, in accordance with an embodiment of the present disclosure; and

[0035] FIG. 6 is a flowchart of a method for a Mixed Precision controller in a HPC system, in accordance with an embodiment of the present disclosure.

[0036] In the accompanying drawings, an underlined number is employed to represent an item over which the underlined number is positioned or an item to which the underlined number is adjacent. A non-underlined number relates to an item identified by a line linking the non-underlined number to the item. When a number is non-underlined and accompanied by an associated arrow, the non-underlined number is used to identify a general item at which the arrow is pointing. DETAILED DESCRIPTION OF EMBODIMENTS

[0037] The following detailed description illustrates embodiments of the present disclosure and ways in which they can be implemented. Although some modes of carrying out the present disclosure have been disclosed, those skilled in the art would recognize that other embodiments for carrying out or practicing the present disclosure are also possible.

[0038] FIG. 1 is a block diagram that illustrates various exemplary components of a High-Performance Computing (HPC) system, in accordance with an embodiment of the present disclosure. With reference to FIG. 1, there is shown a block diagram 100 of a High-Performance Computing (HPC) system 102. There is shown a Mixed Precision (MP) controller 104 configured to operate in the HPC system 102. The HPC system 102 further comprises a high-precision processor array 106, a low-precision processor array 108 and a memory 110. The MP controller 104 is further configured to receive a system of Partial Differential Equations (PDEs) 112. Each of the MP controller 104, the high-precision processor array 106, the low-precision processor array 108 and the memory 110 are communicably coupled to each other.

[0039] The HPC system 102 may be referred to as a computing architecture specially designed to handle computational tasks requiring substantial computational power and throughput. The HPC system 102 typically include large-scale parallel processing of data, using multiple interconnected processors or cores to execute computations concurrently.

[0040] The MP controller 104 may include suitable logic, circuitry, interfaces and / or code that is configured to execute a part of computational workload using high-precision floating point arithmetic and remaining computational workload using low- precision floating point arithmetic with the objective of retaining high-precision accuracy of overall result. Alternatively stated, the MP controller 104 may be used to recover the result with accuracy equivalent to the one that would have been obtained if only high precision operations are used. The MP controller 104 may be configured to execute a combination of high (e.g., FP64) and low (e.g., FP16 / FP32) precision floating point operations within a given algorithm to obtain overall high-precision accuracy while reducing computational cost and overall execution time as well. The use of MP controller 104 eliminates several constraints on memory bandwidth, memory capacity and throughput. By judiciously reducing the precision, the amount of data required to be fetched from memory (e.g. the memory 110) to the computational units is reduced, mitigating the constraint on memory bandwidth of the HPC system 102. The total amount of data required to be stored in the memory is reduced hence, memory capacity of the HPC system 102 is also enhanced. Moreover, the throughput of the HPC system 102 is enhanced by utilizing faster low-precision arithmetic, either on the CPU or on the available accelerators, such as GPUs, NPUs, TPUs, and the like.

[0041] The high-precision processor array 106 may include suitable logic, circuitry, interfaces and / or code that is configured to execute high-precision Floating Point Operations per Second (FLOPs), for example, FP64. The FLOPs are often used to measure computing capabilities of various processing units. The high-precision processor array 106 may include one or more CPUs. The FLOPs may be defined as any mathematical operation, such as +, -, *, / , or assignment that involves floating-point (or noninteger) numbers.

[0042] The low-precision processor array 108 may include suitable logic, circuitry, interfaces and / or code that is configured to execute high-precision Floating Point Operations per Second (FLOPs), for example, FP16, FP32, BFLOAT16, and the like. The low- precision processor array 108 may include one or more accelerators, such as GPUs, NPUs, TPUs, and a combination thereof.

[0043] The memory 110 may include suitable logic, circuitry, interfaces and / or code that is configured to store machine code and / or instructions executable by the high-precision processor array 106 and the low-precision processor array 108. Examples of implementation of the memory 110 may include, but are not limited to, an Electrically Erasable Programmable Read-Only Memory (EEPROM), Random Access Memory (RAM), Read Only Memory (ROM), Hard Disk Drive (HDD), Flash memory, a Secure Digital (SD) card, Solid-State Drive (SSD), a computer readable storage medium, and / or CPU cache memory. The memory 110 may store an operating system and / or a computer program product to operate the MP controller 104. A computer readable storage medium for providing a non-transient memory may include, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. In an implementation, the fixed-point iterative scheme, g and the defect correction term, D may be stored in the memory 110.

[0044] In operation, the MP controller 104 is configured to operate in the HPC system 102, where the HPC system 102 comprises the high-precision processor array 106 and the low-precision processor array 108, where the MP controller 104 is further configured to receive the system of partial differential equations 112 and utilize the high-precision processor array 106 and the low- precision processor array 108 to solve the system of partial differential equations 112 utilizing a fixed-point iterative scheme, g, where the MP controller 104 is characterized in that the MP controller 104 is further configured to utilize a defect correction term which is defined as the difference between a high-precision residual, RH. and a low-precision residual, RL. where the defect correction term is added to an independent constants vector of the system of partial differential equations 112. The MP controller 104 may be configured to execute a mixed precision algorithm by use of the high-precision processor array 106 and the low-precision processor array 108. The mixed precision algorithm is executed to solve non-linear Partial Differential Equations (PDEs) on a heterogeneous hardware. In the mixed precision algorithm, a part of computational work is offloaded to low precision hardware (i.e., the low-precision processor array 108) while retaining the high-precision accuracy of a final result (i.e., FP64).

[0045] In a conventional algorithm, all of the computational work is executed by the high-precision hardware (i.e., FP64) and thereby, inducing various constraints on memory bandwidth, memory capacity and throughput of a computing system. In contrast to the conventional algorithm, in the mixed algorithm, a part of the computational work (i.e., low-precision floating point operations) is executed by the low-precision processor array 108 and remaining part (i.e., high-precision floating point operations) is executed by the high-precision processor array 106, while maintaining the high-precision accuracy of the final result (i.e., FP64). In this way, the mixed precision algorithm eliminates various constraints on the memory bandwidth, the memory capacity and throughput of the HPC system 102 while reducing the computational load of the HPC system 102 and the overall execution time of the computational load.

[0046] In order to solve the non-linear PDEs, the MP controller 104 is configured to receive the system of partial differential equations 112 and utilize the fixed-point iterative scheme, g, for solving the system of partial differential equations 112. The MP controller 104 is further configured to utilize the defect correction term for solving the system of partial differential equations 112. The defect correction term may be defined as an additional source term added to the independent constants vector of the system of partial differential equations 112 in order to converge the solution of the system of partial differential equations 112. The defect correction term is used to drive the solution towards the desired high precision accuracy.

[0047] In accordance with an embodiment, the system of partial differential equations is expressed as

[0048] Ax = b, where A denotes a discrete differential operator matrix, x denotes a vector of unknowns, and b denotes the vector of independent constants, wherein the MP controller 104 is further configured to assign the defect-correction term, D, to be 0, determine an initial guess, x°, for the high-precision processor array 106, and advance the solution on the high-precision processor array 106 for several steps using a computationally cheap method, to improve the original guess, truncate that updated guess x°, for the low-precision processor array 108, and while no solution of sufficient accuracy has been determined repeatedly for n=l, 2, 3, ... determine the high-precision residual for iteration n, RHn= Axn— b, determine the low-precision residual for iteration n RLn= Axn— b, wherein

[0049] A denotes a low-precision floating-point evaluation of the discrete differential operator, A, xndenotes a low-precision floating-point evaluation of the unknown variables vector, x, for iteration n, and b denotes a low-precision floating-point evaluation of the independent constants vector, b, determine the defect-correction term for iteration n, Dnas the difference between the high-precision residual and the low- precision residual for iteration n, Dn= RHn— RLn, set xk=0= xn, apply the fixed-point iterative scheme, g. in order to compute an approximate solution for Axk= b — D utilizing the low- precision processor array 108 0,1, ..., K, determine the high-precision residual for iteration n+1, RHn+1= Axn+1— b, and determine that a solution of sufficient accuracy has been determined as xn+1if the high-precision residual for iteration n, RHn, is below an accuracy threshold.

[0050] The system of partial differential equations 112 may also be referred to as a discrete system of partial differential equations and may be represented in a compact vector form according to Equation (1 )

[0051] Ax = b (1) where A denotes the discrete differential operator matrix, x denotes the vector of unknowns, and b denotes the vector of independent constants. In addition to receiving the system of partial differential equations 112, the MP controller 104 is configured to utilize the fixed-point iterative scheme, g, for solving the system of partial differential equations 112 using any (single) floating point precision. The fixed-point iterative scheme, g, can be represented by Equation (2) where, n denotes the iteration index.

[0052] The MP controller 104 is configured to solve the system of partial differential equations 112 in the following steps. Firstly, the MP controller 104 is configured to assign the defect-correction term equal to zero, D = 0. Thereafter, the MP controller 104 is configured to compute the initial guess, x°, by either applying the fixed-point iterative scheme, g, on the high-precision processor array 106 (e.g., the CPUs) or by any other means. The MP controller 104 is configured to advance the solution of the high-precision processor array 106 using the computationally cheap method (e.g., a Jacobi method) to improve the initial guess, x°. The computationally cheap method may be defined as an algorithm (or an approach) that requires low computational resources, such as processing power, memory, or time to execute. The “cheapness” can vary depending on some specific constraints, for example, in some cases, the “cheap” may mean an algorithm that has low time complexity, low space complexity or low energy consumption.

[0053] The MP controller 104 is further configured to truncate the updated guess x° for the low-precision processor array 108. While no solution of sufficient accuracy has been determined, the MP controller 104 is configured to repeat the below mentioned steps repeatedly forn = 1, 2, 3, ... The MP controller 104 is further configured to determine the high-precision residual for nth iteration, RHn= Axn— b and the low precision residual for nth iteration, RLn= Axn— b, where, A denotes the low-precision floating-point evaluation of the discrete differential operator, A, xndenotes the low-precision floating-point evaluation of the unknown variables vector, x, for iteration n, and b denotes the low-precision floating-point evaluation of the independent constants vector, b. The MP controller 104 is further configured to determine the defect-correction term for nth iteration, Dnas the difference between the high-precision residual (i.e., RHn) and the low-precision residual (i.e., RLn) for iteration nth iteration, as Dn= RHn— RLn. The MP controller 104 is further configured to set the xk=0= xn. The MP controller 104 is further configured to apply the fixed-point iterative scheme, g, as xk+1= g(A,xk,b — Dn) for k = 0,l,...,K, in order to compute the approximate (i.e., low-precision) solution for Axk= b — D, where K is determined empirically based on local convergence measures. The fixed-point iterative scheme, g is applied using the low-precision processor array 108. The addition of Dnto the independent constant vector serves to offset the error introduced by the low-precision arithmetic, drives the solution towards high-precision accuracy. The MP controller 104 is further configured to correct the fine-precision approximation according to xn+1= xK. The MP controller 104 is further configured to compute the high-precision residual for (n + 1)' th iteration, as RHn+1= Axn+1— b which, serves as a measure of the overall solution accuracy. The steps from assigning the defect correction term, D = 0 up to the computation of the high-precision residual (i.e., RHn+1) for (n + 1) are repeated until the magnitude of the high-precision residual (i.e., RHn+1) become sufficiently small that means the solution has converged to the desired level of accuracy. In between the n iterations, few relaxation steps can be applied to the solution, on the high precision processor array 106 to remove some of the numerical oscillations before moving to the next iteration.

[0054] In accordance with an embodiment, the Mixed Precision controller 104 is further configured to determine the constant K empirically based on local convergence measures. The constant K used in computation of the approximate (i.e., low-precision) solution for Axk= b — D, is determined empirically based on the local convergence measures. The efficiency of the mixed precision algorithm depends on the amount of convergence that is obtained at each low-precision array cycle. The constant K should be chosen to minimize the number of iterations on the low-precision processor array 108 while maintaining the overall convergence of the solution. The constant K depends on an application scenario as well as on the hardware components.

[0055] In accordance with an embodiment, the Mixed Precision controller 104 is further configured to determine the defect correction term, D utilizing the high-precision processor array 106. Alternatively stated, the MP controller 104 is configured to determine the defect correction term, D by executing the high-precision floating-point operations (e.g., FP64) by use of the high-precision processor array 106.

[0056] In accordance with an embodiment, the Mixed Precision controller 104 is further configured to determine the initial guess, x°, for the high-precision processor array 106 by utilizing an iterative algorithm. The initial guess, x°, is typically set to a unit vector. To make sure that the initial guess, x°, is then closer to the actual solution of interest, the iterative algorithm (or an iterative solver) is applied before starting the cycles with the more advanced numerical methods. Typically, the iterative algorithm may be defined as an algorithm that uses repeated iterations to allow the solution to converge towards the exact solution for a given problem.

[0057] In accordance with an embodiment, the iterative algorithm is a Jacobi algorithm. In an implementation, the iterative algorithm is the Jacobi algorithm. Typically, the Jacobi algorithm is used to solve the linear PDEs, such as shown in Equation (3)

[0058] AA= 0dx2 +dy2(3)

[0059] After spatial discretization with uniform grid space h. Equation (4) is obtained i+lj + i-lj + ij+l + ij-l_ij = 0 (4)

[0060] The Jacobi method is based on adding a pseudo-temporal iteration for calculating <p, via the following algorithm, as shown in Equation (5) where i,j are spatial coordinates in computational space, and k is the index of the pseudo-temporal iteration. The convergence depends on the properties of the PDEs and the specific boundary conditions.

[0061] In accordance with an embodiment, the Mixed Precision controller 104 is further configured to not apply any linearization of the system of partial differential equations 112. The MP controller 104 does not apply any linearization of the non-linear partial differential equations therefore, the MP controller 104 can be directly used for the original (or non-linear) partial differential equations hence, enables a faster convergence and maximal accuracy.

[0062] In accordance with an embodiment, the partial differential equations are elliptic. In an implementation, the non-linear partial differential equations may be elliptic in nature.

[0063] In accordance with an embodiment, the high-precision processor array 106 has a precision that is higher than the precision of the low-precision processor array 108. The high-precision processor array 106 is configured to execute the high-precision floating-point operations, e.g., FP64, having higher precision, while the low-precision processor array 108 is configured to execute the low-precision floating-point operations, e.g., FP32, FP16, etc., having a comparatively lower precision than the high-precision floating-point operations.

[0064] In accordance with an embodiment, the high-precision processor array 106 has a precision of FP64 or higher and the low- precision processor array 108 has a precision of FP16 or FP 32. Typically, the FP64 may be defined as a double precision floating-point format, which is a standard for representing and performing arithmetic operations on real numbers in computing systems (e.g., the HPC system 102). In the FP64 format, the floating-point numbers are typically represented using 64 bits, with a specific allocation for a sign bit, exponent and significand. The FP64 is particularly used in scientific computing and engineering applications where precision is paramount. The FP 32 may be defined as a single-precision floating-point format, which is a standard for representing and performing arithmetic operations on real numbers in computing systems (e.g., the HPC system 102). In the FP32 format, the floating-point numbers are typically represented using 32 bits, with a specific allocation for a sign bit, exponent and significand. The FP32 is commonly used in a variety of applications, including graphics processing, machine learning and scientific computing. The FP32 offers lesser precision than FP64, the FP32 is more efficient in terms of memory usage and computational resources. The various tasks in aforementioned application scenarios can tolerate the lower precision without the significant loss of accuracy hence, making the FP32 a practical choice for a wide spectrum of applications.

[0065] Thus, the use of the MP controller 104 in the HPC system 102 enables the high-precision accuracy while reducing the computational cost and the overall execution time of non-linear partial differential equations. The use of the MP controller 104 in the HPC system 102 reduces the memory traffic (e.g., from the memory 110 to either the high-precision processor array 106 or to the low-precision processor array 108) and the network traffic (from one node to another node in the HPC system 102). Moreover, the use of the MP controller 104 in the HPC system 102 enables a reduction in the memory footprint (i.e., less data is either stored in the memory 110 or retrieved from the memory 110). The MP controller 104 is configured to execute the low- precision floating-point operations along with the high-precision floating-point operations and, the low-precision floating-point operations are executed comparatively faster than the high-precision floating-point operations. Therefore, the floating-point arithmetic operations are executed at a faster rate (e.g., at least factor of 2). The low-precision processor array 108 (i.e., the GPUs, the NPUs and the TPUs) have more available computing power in low precision. For example, the application scenarios including Al training (convolution, transformer, etc.) involves the multiplication of small matrices that are insensitive to precision and can be done by use of low-precision floating-point operations (i.e., by FP16). Moreover, the MP controller 104 does not require any linearization of non-linear PDEs for convergence. The MP controller 104 uses the non-linear defect correction term for solving the system of partial differential equations 112. The existing solvers can also be augmented with the non-linear defect correction term to work on the heterogeneous architectures. The mixed precision algorithm may also be combined with any existing method that is based on the fixed-point iteration for solving the linear system of equations. The use of MP controller 104 in the HPC system 102 enables an efficient hardware implementation by minimizing the data transfer from host to accelerators (e.g., either from CPUs to NPUs, GPUs, and the like or from the CPUs to DRAM) and reducing the accelerator memory usage via reuse of the accelerator memory. The MP controller 104 allows the convergence of a given problem without the overhead of iteratively refining the solution towards the desired level of accuracy. The MP controller 104 enables the utilization of the heterogeneous hardware for scientific HPC applications by retaining the desired precision level of accuracy.

[0066] FIG. 2 is a flowchart of a method of mixed-precision defect correction, in accordance with an embodiment of the present disclosure. FIG. 2 is described in conjunction with elements from FIG. 1. With reference to FIG. 2, there is shown a method 200 that includes a sequence of operations 202 to 214. The MP controller 104 (of FIG. 1) is configured to execute the method 200.

[0067] At operation 202, an initial guess, x°, is determined for the high-precision processor array 106 (e.g., FP64).

[0068] At operation 204, an elementary method (e.g., the Jacobi method) is used to obtain an approximate solution on the high- precision processor array 106 (i.e., FP64).

[0069] At operation 206, a defect-correction term, D, is computed as a difference between a high-precision residual (i.e., RH) and a low-precision residual (i.e., RL as D = RH — RL on the high-precision processor array 106 (i.e., FP64).

[0070] At operation 208, a fixed-point iterative scheme, g, asxk+1= g(A,xk,b — Dn) fork = 0,1,..., K, is applied in order to compute an approximate solution for Axk= b — D utilizing the low-precision processor array 108 (e.g., FP32 / FP16), have been described in detail, for example, in FIG. 1.

[0071] At operation 210, an estimated current solution is corrected on the high-precision processor array 106 (i.e., FP64).

[0072] At operation 212, a loop for n iterations is executed until a target residual magnitude is reached. The operations from 204 to 210 are executed iteratively.

[0073] At operation 214, a converged solution of non-linear PDEs (e.g., elliptic PDEs) is recovered with high-precision at the high- precision processor array 106 (i.e., FP64).

[0074] FIG. 3 illustrates a mathematical representation of a mixed-precision defect correction method, in accordance with an embodiment of the present disclosure. FIG. 3 is described in conjunction with elements from FIGs. 1 and 2. With reference to FIG. 3, there is shown a mathematical representation 300 that includes a sequence of operations 302 to 314.

[0075] At operation 302, an initial guess, x°, is determined.

[0076] At operation 304, an elementary method (e.g., the Jacobi method) is used to obtain an approximate solution xn.

[0077] At operation 306, a defect-correction term, D, is computed as a difference between a high-precision residual (i.e., RH = Axn— b ) and a low-precision residual (i.e., RL = Axn— 5) as D = RH — RL on the high-precision processor array 106 (i.e., FP64).

[0078] At operation 308, a fixed-point iterative scheme, g, is applied in order to compute an approximate solution for Axk= b — D utilizing the low-precision processor array 108 (e.g., FP32 / FP16). At operation 310, an estimated current solution xn+1= xKis corrected on the high-precision processor array 106 (i.e., FP64).

[0079] At operation 312, a loop for n iterations is executed until a target residual magnitude is reached. The operations from 304 to 310 are executed iteratively.

[0080] At operation 314, a converged solution of non-linear PDEs (e.g., elliptic PDEs) that is Ax = b is recovered with high-precision at the high-precision processor array 106 (i.e., FP64).

[0081] FIG. 4 is a flowchart of a method of mixed-precision defect correction, in accordance with another embodiment of the present disclosure. FIG. 4 is described in conjunction with elements from FIGs. 1, 2, and 3. With reference to FIG. 4, there is shown a method 400 that includes a sequence of operations 402 to 418. The MP controller 104 (of FIG. 1) is configured to execute the method 400.

[0082] The method 400 is similar to the method 200 (of FIG. 2), the operations 402, 404, 406, 410, 414, 416 and 418 are similar to the operations 202, 204, 206, 208, 210, 212 and 214, respectively. The method 400 may be stated as an optimized version of the method 200. The method 400 can be optimized by minimizing the communication between the high-precision processor array 106 (i.e., the CPUs) and the low-precision processor array 108 (i.e., the accelerators, such as the GPUs, the NPUs, the TPUs, and the like) and by reducing the overhead involved in transferring the data to / from the low-precision processor array 108 (i.e., the accelerators) during the iterations. This is accomplished in the following way: at each iteration, only the defect correction vector, Dn, is transferred from the host to the accelerator. For example, at operation 408, the defect correction vector, Dn, is transferred from the high-precision processor array 106 (i.e., the CPUs) to the low-precision processor array 108 (i.e., the accelerators). Moreover, at operation 412, the current approximate solution xKis transferred from the low-precision processor array 108 (i.e., the accelerators) back to the high-precision processor array 106 (i.e., the CPUs). Furthermore, only the low-precision approximation of the defect correction vector, Dn, is transferred to the low-precision processor array 108 (i.e., the accelerators), thus further reducing the required communication bandwidth.

[0083] FIG. 5 represents an illustration of a mixed-precision defect correction approach, in accordance with an embodiment of the present disclosure. FIG. 5 is described in conjunction with elements from FIGs. 1, 2, 3, and 4. With reference to FIG. 5, there is shown an illustration 500 of the mixed-precision defect correction approach.

[0084] As shown in FIG. 5, the mixed-precision defect correction approach is applied to an existing multi-grid method that originally runs on CPUs with the FP64 accuracy (i.e., the high-precision processor array 106). In the mixed-precision defect correction approach, only a small part of the computational workload is still done on the CPU with the high-precision (i.e., the high- precision processor array 106), while the rest of the workload is offloaded to the accelerators (i.e., the low-precision processor array 108) and done with low-precision arithmetic. By doing so, the overall computational time required is significantly reduced, whilst maintaining the original level of accuracy.

[0085] FIG. 6 is a flowchart of a method for a Mixed Precision controller in a High-Performance Computing (HPC) system, in accordance with an embodiment of the present disclosure. FIG. 6 is described in conjunction with elements from FIGs. 1, 2, 3, 4, and 5. With reference to FIG. 6, there is shown a method 600 that includes steps 602 to 606. The MP controller 104 (of FIG. 1) is configured to execute the method 600.

[0086] There is provided the method 600 for the MP controller 104 for use in the HPC system 102 (of FIG. 1). The HPC system 102 comprises the high-precision processor array 106 and the low-precision processor array 108, have been described in detail, for example, in FIG. 1. At step 602, the method 600 comprises receiving the system of partial differential equations 112. In an implementation, the partial differential equations may be non-linear in nature (e.g., elliptic PDEs). In another implementation, the partial differential equations may be linear in nature (e.g., heat equation).

[0087] At step 604, the method 600 further comprises utilizing the high-precision processor array 106 and the low-precision processor array 108 to solve the system of partial differential equations 112 utilizing a fixed-point iterative scheme, g. The fixed-point iterative scheme, g, is used to solve the system of partial differential equations 112 using any single floating-point precision.

[0088] At step 606, the method 600 further comprises utilizing a defect correction term which is defined as the difference between a high-precision residual, RH. and a low-precision residual, RL. where the defect correction term is added to an independent constants vector of the system of partial differential equations 112. The defect correction term may be defined as a correction term that is used to offset the error introduced by the low-precision arithmetic, such that the accuracy of the overall solution is retained. The defect correction term is used to solve the system of partial differential equations 112 up to the desired level of accuracy, using a combination of low-precision and high-precision floating-point operations.

[0089] The steps 602 to 606 are only illustrative and other alternatives can also be provided where one or more steps are added, one or more steps are removed, or one or more steps are provided in a different sequence without departing from the scope of the claims herein.

[0090] In an aspect, the present disclosure provides a computer program product comprising program instructions for performing the method 600, when executed by one or more processors in the HPC system 102. In another aspect, the present disclosure provides a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method 600 for the MP controller 104 in the HPC system 102.

[0091] Modifications to embodiments of the present disclosure described in the foregoing are possible without departing from the scope of the present disclosure as defined by the accompanying claims. Expressions such as "including", "comprising", "incorporating", "have", "is" used to describe and claim the present disclosure are intended to be construed in a non-exclusive manner, namely allowing for items, components or elements not explicitly described also to be present. Reference to the singular is also to be construed to relate to the plural. The word "exemplary" is used herein to mean "serving as an example, instance or illustration". Any embodiment described as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments and / or to exclude the incorporation of features from other embodiments. The word "optionally" is used herein to mean "is provided in some embodiments and not provided in other embodiments". It is appreciated that certain features of the present disclosure, which are, for clarity, described in the context of separate embodiments, may also be provided in combination in a single embodiment. Conversely, various features of the present disclosure, which are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable combination or as suitable in any other described embodiment of the disclosure.

Claims

CLAIMS1. A Mixed Precision, MP, controller (104) configured to operate in a High-Performance Computing, HPC, system (102), wherein the HPC system (102) comprises a high-precision processor array (106) and a low-precision processor array (108), wherein the Mixed Precision controller (104) is further configured to receive a system of partial differential equations (112) and utilize the high-precision processor array (106) and the low-precision processor array (108) to solve the system of partial differential equations (112) utilizing a fixed-point iterative scheme, g. wherein the Mixed Precision controller (104) is characterized in that the Mixed Precision controller (104) is further configured to utilize a defect correction term which is defined as the difference between a high-precision residual, RH , and a low-precision residual, RL. wherein the defect correction term is added to an independent constants vector of the system of partial differential equations (112).

2. The Mixed Precision controller (104) according to claim 1, wherein the system of partial differential equations (112) is expressed asAx = b, where A denotes a discrete differential operator matrix, x denotes a vector of unknowns, and b denotes the vector of independent constants, wherein the Mixed Precision controller (104) is further configured to assign the defect-correction term, D, to be 0, determine an initial guess, x°, for the high-precision processor array (106), and advance the solution on the high-precision processor array (106) for several steps using a computationally cheap method, to improve the original guess, truncate that updated guess x°, for the low-precision processor array (108), and while no solution of sufficient accuracy has been determined repeatedly for n=l, 2, 3, ... determine the high-precision residual for iteration n, RHn= Axn— b, determine the low-precision residual for iteration n RLn= Axn— b, whereinA denotes a low-precision floating-point evaluation of the discrete differential operator, A, xndenotes a low-precision floating-point evaluation of the unknown variables vector, x, for iteration n, and b denotes a low-precision floating-point evaluation of the independent constants vector, b, determine the defect-correction term for iteration n, Dnas the difference between the high- and the low- precision residual for iteration n, Dn= RHn— RLn, set xk=0= xn, apply the fixed-point iterative scheme, g. in order to compute an approximate solution for Axk= b — D utilizing the low-precision processor array (108) 0,1, ..., K,determine the high-precision residual for iteration n+1, RHn+1= Axn+1— b, and determine that a solution of sufficient accuracy has been determined as xn+1if the high-precision residual for iteration n, RHn, is below an accuracy threshold.

3. The Mixed Precision controller (104) according to any preceding claim, wherein the Mixed Precision controller (104) is further configured to determine the constant K empirically based on local convergence measures.

4. The Mixed Precision controller (104) according to any preceding claim, wherein the Mixed Precision controller (104) is further configured to determine the defect correction term, D utilizing the high-precision processor array (106).

5. The Mixed Precision controller (104) according to any preceding claim, wherein the Mixed Precision controller (104) is further configured to determine the initial guess, x°, for the high-precision processor array (106) by utilizing an iterative algorithm.

6. The Mixed Precision controller (104) according to claim 5, wherein the iterative algorithm is a Jacobi algorithm.

7. The Mixed Precision controller (104) according to any preceding claim, wherein the Mixed Precision controller (104) is further configured to not apply any linearization of the system of partial differential equations (112).

8. The Mixed Precision controller (104) according to any preceding claim, wherein the partial differential equations are elliptic.

9. The Mixed Precision controller (104) according to any preceding claim, wherein the high-precision processor array (106) has a precision that is higher than the precision of the low-precision processor array (108).

10. The Mixed Precision controller (104) according to claim 9, wherein the high-precision processor array (106) has a precision of FP64 or higher and the low-precision processor array (108) has a precision of FP16 or FP32.

11. A method (600) for a Mixed Precision controller (104) in a High-Performance Computing, HPC, system (102), comprising a high-precision processor array (106) and a low-precision processor array (108), wherein the method (600) comprises receiving a system of partial differential equations (112) and utilize the high-precision processor array (106) and the low-precision processor array (108) to solve the system of partial differential equations (112) utilizing a fixed-point iterative scheme, g, wherein the method (600) is characterized in that the method (600) further comprises utilizing a defect correction term which is defined as the difference between a high-precision residual, RH , and a low-precision residual, RL. wherein the defect correction term is added to an independent constants vector of the system of partial differential equations (112).

12. A computer program product comprising program instructions for performing the method (600) according to claim 11, when executed by one or more processors in the HPC system (102).

Images