A hardware acceleration method for fourier neural operators

Abstract

The application discloses a hardware acceleration method for a Fourier neural operator, and sets up independent FFT modules and DFT / NN modules in an FNO hardware accelerator; the FFT modules are used to perform fast Fourier transform (FFT) calculation on batch input data; the DFT / NN modules are used to perform discrete Fourier transform (DFT) calculation based on a Bluestein algorithm, and a parallel random access mode of a rotation factor is reconstructed into a parallel regular access mode; in the FNO hardware accelerator, based on static parameter analysis on input dimensions, transform point numbers and frequency truncated output point numbers, the FFT modules or the DFT / NN modules are adaptively selected as units for performing calculation, and corresponding operator fusion is performed based on a predefined operator fusion mode. The application enhances the adaptation capability of the FNO to different transform point numbers and dimension structures, and effectively unifies the calculation performance, hardware efficiency and application flexibility.

Description

Technical Field

[0001] This invention relates to the field of neural network inference hardware technology, and in particular to a hardware acceleration method for Fourier neural operators. Background Technology

[0002] Fourier Neural Operators (FNOs), with their spectral-based modeling capabilities and excellent generalization performance, are widely used for learning operators for solving partial differential equations. This method maps the input function to the frequency domain using the Fast Fourier Transform (FFT), performs linear transformations in the frequency domain such as Complex Matrix Multiplication (CGEMM) to learn low-dimensional, global solution operators, and then reconstructs the spatial solution through an inverse transform. Its core computational model can be abstracted as a fixed pipeline of "Fast Fourier Transform (FFT) → Complex Matrix Multiplication (CGEMM) → Inverse Fast Fourier Transform (IFFT)".

[0003] Currently, the most widely used implementation of FNO is based on PyTorch, relying on separate calls to the Discrete Fast Fourier Transform (cuFFT), Discrete Basic Linear Computation (cuBLAS), and custom frequency filtering kernels. This implementation lacks deep optimization at the architectural level and suffers from the following efficiency bottlenecks on modern GPU platforms: 1) Partial frequency utilization: CGEMM only operates on selected low-frequency components, requiring additional memory copying of the kernel to extract and insert relevant frequencies, introducing significant memory access overhead; 2) cuFFT lacks native frequency filtering support: cuFFT does not support frequency filtering, and its closed-source design makes it difficult to integrate custom filtering logic; 3) Excessive memory transfers: cuFFT and cuBLAS are independent black-box libraries, and intermediate results at each stage need to be written back to global memory, resulting in additional data transfers and limiting the possibility of data reuse and kernel fusion. The optimization of pipelines combining Fast Fourier Transform (FFT) and Convolution (FFT-Convolution), and combining FFT and Stencil computation (FFT-Stencil), relies on the alignment of data layout before and after the time and frequency domain transformations. However, the frequency filtering truncation and zero-padding operations required by FNO disrupt this premise, making the optimization schemes designed for it unsuitable for the FNO's "Fast Fourier Transform (FFT) - General Matrix Multiplication (GEMM) - Inverse Fast Fourier Transform (IFFT)" pipeline. Furthermore, FNO inputs are mostly two-dimensional or higher, making it difficult for traditional one-dimensional FFT accelerators to fully utilize their inherent parallelism, resulting in low data reuse efficiency.

[0004] Existing optimization schemes for FNO implementation mainly include: GPU implementation of TurboFNO, ASIC implementation of PhyCU, and ASIC implementation of ReBA. However, these optimization schemes have the following limitations: 1) Limited FFT point support: TurboFNO is based on FFT and only supports 2. a Points, even if it supports 2 a x3 b x5 c x7 d 1) Point-based cuFFT is also difficult to adapt to non-power-2 resolutions commonly used in applications such as the Burger equation, such as 85, 141, 211, and 421. The DFT method used by PhyCU and ReBA is more versatile. 2) DFT computational power expansion efficiency is low: cuFFT and TurboFNO perform one-dimensional PE calculation along the batch dimension, requiring only a single rotation factor, but the number of PEs has an upper limit. PhyCU and ReBA use PxR two-dimensional PE calculation, where P is the parallel dimension and R is the accumulation dimension. R copies of the rotation factor need to be stored to avoid parallel access conflicts, resulting in low local storage utilization. 3) Lack of high-order fusion strategies: PhyCU and ReBA do not consider operator fusion. Although TurboFNO has implemented basic FFT-GEMM-IFFT fusion, in two-dimensional scenarios, it has neither explored the possibility of FFT-FFT and IFFT-IFFT fusion nor systematically compared and optimized various fusion strategies.

[0005] In summary, existing methods for implementing and accelerating Fourier neural operators still have shortcomings in terms of computational efficiency, flexibility, and adaptability. Summary of the Invention

[0006] To address the aforementioned technical problems in the existing technology, this invention proposes a hardware acceleration method for Fourier neural operators, the specific technical solution of which is as follows:

[0007] A hardware acceleration method for Fourier neural operators, based on an FNO hardware accelerator, the method comprising:

[0008] Each FFT module and DFT / NN module is set up independently in the FNO hardware accelerator;

[0009] The FFT module is used to perform Fast Fourier Transform (FFT) calculations on the batch input data;

[0010] Using the DFT / NN module, Discrete Fourier Transform (DFT) calculations are performed based on the Bluestein algorithm, and the access pattern of the rotation factor is reconstructed from parallel random access to parallel regular access.

[0011] Moreover, in the FNO hardware accelerator, based on the analysis of static parameters such as input dimension, number of transform points, and frequency-truncated output points, an FFT module or a DFT / NN module is adaptively selected as the unit for performing calculations, and corresponding operator fusion is performed based on a predefined operator fusion mode.

[0012] Furthermore, the neural network calculations in the Fourier neural operator include complex matrix multiplication CGEMM and real matrix multiplication RGEMM.

[0013] Furthermore, the DFT / NN module has a two-dimensional processing element (PE) array, and the PE array has a parallel dimension and an accumulation dimension; when performing DFT calculations, the input data is stored in a high-bandwidth input data storage, and the rotation factors are stored in a low-bandwidth fixed parameter storage. The parallel dimension corresponds to the channel dimension of the input data, and the accumulation dimension corresponds to the transform dimension for which the DFT calculations are to be performed.

[0014] Furthermore, the FFT module is configured to perform FFT calculations on a batch input [nParallel, nFFT], where nParallel is the number of tasks processed in parallel, nFFT is the number of transform points for each task, and the value of nParallel is aligned with the requirements of the DFT / NN module.

[0015] Furthermore, the adaptive selection of the FFT module or the DFT / NN module as the unit for performing calculations based on the analysis of static parameters such as input dimension, number of transform points, and frequency-truncated output points specifically includes:

[0016] Determine whether the number of transform points N is a specification supported by the FFT module:

[0017] If it belongs to the specifications supported by the FFT module, compare the frequency-truncated output points M with log2(N). If M < log2(N), then select the DFT / NN module; otherwise, select the FFT module.

[0018] If it does not belong to the specifications supported by the FFT module, perform DFT calculations based on the Bluestein algorithm, and compare M with 24log2(N). If M > 24log2(N), then select the path to implement the Bluestein algorithm via three FFT / IFFT operations; otherwise, select the path to perform DFT calculations by the DFT / NN module.

[0019] Furthermore, after selecting the unit for performing calculations, a computational flow graph of the FNO operator is formed according to the number of dimensions D of the input data, and in combination with the storage capacity constraints of the FNO hardware accelerator, the computational flow graph is matched and compared with the predefined operator fusion mode to select the combination of the computational flow graph and the fusion mode with the optimal theoretical memory access overhead.

[0020] Furthermore, the predefined operator fusion modes include FFT-FFT fusion, IFFT-IFFT fusion, FFT-GEMM-IFFT fusion, and IFFT(FFT,FFT) fusion.

[0021] Furthermore, the FNO hardware accelerator also includes:

[0022] Local shared storage is used to temporarily store intermediate calculation results between the FFT module and the DFT / NN module to support cross-module operator fusion at the hardware level.

[0023] The Load module is used to load data from external storage to local shared storage, input data storage, and fixed parameter storage, as well as to load the results of intermediate operators from local shared storage to input data storage and fixed parameter storage. When performing DFT calculations, the Load module performs premultiplication calculations, regardless of whether the input data is loaded from external storage or local shared storage.

[0024] Store module: Used to move output data to local shared storage or external storage.

[0025] Furthermore, the Bluestein algorithm is used to reconstruct the access pattern of the rotation factor from parallel random access based on the product index nk to parallel regular access based on the difference index nk, where n is the input index and k is the output index.

[0026] Beneficial effects: By using the Bluestein algorithm to transform the DFT calculation and reconstruct the twiddle factor access pattern, memory access conflicts in large-scale PE arrays are fundamentally avoided, the locality of twiddle factors is optimized, and the storage space of twiddle factors is decoupled from the scale of the computing array, thus achieving efficient expansion of DFT computing power.

[0027] With the help of the dual-module architecture of FFT and DFT / NN and local shared storage, it supports the fusion of multiple operators at the hardware level, significantly reducing intermediate data transfer.

[0028] This enhances FNO's adaptability to different numbers of transformation points and dimensional structures, achieving an effective balance between computational performance, hardware efficiency, and application flexibility. Attached Figure Description

[0029] Figure 1 This is a diagram of the Fourier neural operator model architecture in this embodiment;

[0030] Figure 2 This is a diagram of the scalable DFT computation architecture based on the Bluestein algorithm in this embodiment;

[0031] Figure 3 This is a block diagram illustrating the structural principle of the Fourier neural operator hardware accelerator in this embodiment.

[0032] Figure 4 This is a schematic diagram of the working mode of the two-dimensional PE array in the DFT / NN module of this embodiment;

[0033] Figure 5 This is a flowchart illustrating the decision-making process for the Fourier transform implementation in this embodiment.

[0034] Figure 6 This is a schematic diagram of the operator fusion method in computation flow graphs (I) and (II) of this embodiment;

[0035] Figure 7 This is a schematic diagram of the operator fusion method in the computation flow graph (III) of this embodiment. Detailed Implementation

[0036] To make the objectives, technical solutions, and technical effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0037] like Figure 1 As shown, in the FNO model, the input passes through a neural network P, then through several stacked Fourier operator layers, and finally through a neural network Q to be remapped back to the target dimension. Taking the two-dimensional Fourier neural operator FNO2D as an example, the architecture of the Fourier operator layers is divided into two branches. The upper branch is SpectralConv2D, which includes Real Fast Fourier Transform (RFFT) (W), Complex Fast Fourier Transform (CFFT) (H), frequency truncation, Complex Matrix Multiplication (CGEMM) (C), zero-padding, Inverse Complex Fast Fourier Transform (CIFFT) (H), and Inverse Real Fast Fourier Transform (RIFFT) (W). The numbers in parentheses indicate the dimension of the accumulated operations, and the letters R and C represent real and complex numbers, respectively. The lower branch is a one-dimensional convolution Conv1D, which can also be considered as Real Matrix Multiplication (RGEMM) (C). After the upper and lower branches are added together, the output of the FNO model is obtained through an activation function such as ReLU.

[0038] This embodiment discloses a hardware acceleration method for Fourier neural operators, as detailed below:

[0039] like Figure 2 As shown, a scalable DFT computation architecture based on the Bluestein algorithm is designed.

[0040] The Bluestein algorithm is a Fast Fourier Transform (FFT) algorithm capable of computing sequences of arbitrary length (including those with large prime numbers). Its core idea is to transform the Discrete Fourier Transform (DFT) computation into a convolution operation by introducing a chirp signal, thereby leveraging efficient convolutional programming and existing FFT algorithms for computation. The principle of the Bluestein algorithm is as follows:

[0041] (1)

[0042] (2)

[0043] (3)

[0044] (4)

[0045] (5)

[0046] (6)

[0047] Where x n Let n be the input sequence, n be the input index, N be the number of input points, k be the output index, and Z be the smallest power of 2 greater than 2N-1.

[0048] The computational complexity of the pre-multiplication, post-multiplication, and frequency domain dot product is O(N), and the computational complexity of two FFTs and one IFFT is O(Zlog2Z). Therefore, the computational complexity of the Bluestein algorithm is O(Nlog2N), but its constant factor is relatively large.

[0049] Continue to refer to Figure 2 The scalable DFT computing architecture includes a Load module, a matrix computing module, a vector computing module, a Store module, an input storage, a twitch factor storage, a partial sum storage, and an output storage. The Load module is responsible for moving data from external storage to the input and twitch factor storage, and also for performing pre-multiplication on the input, corresponding to the parentheses in formula (3). The matrix computing module performs DFT calculations, using a built-in two-dimensional PE array unit. It accumulates data in one dimension while operating in parallel in the other. Accumulation can be implemented using an addition tree structure, where the multiplier input is a complex number. The vector computing module, in addition to handling routine post-processing, is also responsible for performing post-multiplication on the partial sum, corresponding to the part before the summation symbol in formula (3). The matrix computing module and the partial sum storage together constitute the DFT module. The Store module is responsible for moving data from the output storage to external storage.

[0050] Considering the computational characteristics of the FNO operator, the input, rotation factors, and output are all vectorized along dimension C. For RFFT(W) and RIFFT(W), the parallel dimension of the matrix computation module corresponds to dimension C, and the accumulation dimension corresponds to dimension W. For RFFT(H) and RIFFT(H), the parallel dimension of the matrix computation module still corresponds to dimension C, and the accumulation dimension corresponds to dimension H. For a two-dimensional PE array of size PxR, R rotation factors need to be obtained simultaneously. When the output index is 1, nk=k in formula (1) is still regular, and R rotation factors can be provided in a single cycle. When the output index is 2, nk=2k in formula (1) is no longer regular. Therefore, parallel random access by product index nk will cause a conflict, and it can only be achieved through R storage arrays, and each storage array has a copy of the rotation factor. Otherwise, it is difficult to meet the requirement of obtaining data in a single cycle.

[0051] In summary, the scalable DFT computing architecture of this invention utilizes the Bluestein algorithm to transform DFT computation, reconstructing the access mode of the twiddle factor from the original parallel random access dependent on the product nk index to parallel rule access determined by the n and nk indices respectively. This fundamentally avoids memory access conflicts in large-scale PE processing unit arrays and optimizes the locality of the twiddle factor, decoupling the storage space of the twiddle factor from the scale of the computing array, thus achieving efficient expansion of DFT computing power. Specifically, firstly, the twiddle factor is loaded from external storage into the twiddle factor storage via the Load module. Then, the input is loaded from external storage into the input storage via the Load module. Before writing to the input storage, a pre-multiplication operation is performed on the input, and the required twiddle factor is obtained from the twiddle factor storage. The Load module needs to support marking the dimensions of the FFT to be executed and performing the pre-multiplication operation. Here, it is assumed that the software guarantees that the dimensions of the FFT to be executed can be transferred to the input storage in one go. Secondly, for the matrix calculation module, the data in the input storage is already the twiddle factor. The rotation factor required for each cycle This can be indexed using nk, which also uses R storage arrays at the underlying level, but these arrays only need to store one copy of the rotation factor. For cases where the same index appears within a single period, this can be achieved by accessing and copying the data, reducing the number of rotation factor reads. However, the product nk indexing method cannot take advantage of this because the same index will not appear within a single period. Finally, after the partial sum is calculated, the vector calculation module first performs a post-multiplication operation, using the required rotation factor from the rotation factor storage, and then performs other post-processing operations.

[0052] like Figure 3As shown, an FNO hardware accelerator is designed. Considering the significant differences between FFT and DFT in terms of computational accuracy and computational mode, an independent Fast Fourier Transform (FFT) module and a Discrete Fourier Transform and Neural Network (DFT / NN) fusion module are built in.

[0053] The FFT module supports fast Fourier transform on batch inputs of the form [nParallel, nFFT], where nParallel is the number of parallel processing tasks and nFFT is the number of transform points for each task. The input in FNO is at least two-dimensional, with a parallel dimension in addition to the transform dimension, and the value of the parallel dimension is not small. Traditional FFT modules mainly optimize for acceleration of one-dimensional nFFT inputs. Directly applying this to FNO has the following problems: 1) The input and output data formats are inconsistent with the data formats required by DFT / NN modules; 2) The parallel dimension is not fully utilized, resulting in low data reuse. Therefore, the value of nParallel needs to be aligned with the requirements of DFT / NN modules. The FFT module of this invention has no special requirements on the number of points nFFT and can be 2 as in cuFFT. a x3 b x5 c x7 d Points can also be a pure 2. a Point count. The former uses a hybrid-based FFT architecture, which is relatively complex to implement and cannot use output pruning techniques. It also fully utilizes the advantages of FNO frequency truncation, but it can cover a wide range of points, ensuring that the computational cost is O(Nlog2N) in most scenarios, rather than O(N) for DFT. 2 The latter is simple to implement and can also effectively implement output pruning techniques, but its coverage of points is limited.

[0054] The DFT / NN module adopts the aforementioned scalable DFT computation architecture based on the Bluestein algorithm, and supports neural network computation in the FNO model through a reconfigurable approach. For example... Figure 4The diagram illustrates the operating mode of the two-dimensional PE array in the DFT / NN module. In DFT mode, input data is stored in a high-bandwidth input data storage location, while the twiddle factor is stored in a low-bandwidth fixed-parameter storage location. Here, K is the dimension to be used for DFT computation, and N is the parallel dimension. The data type in this mode is complex numbers. For RDFT and RIDFT computations, some PE operations can be disabled through gating to reduce power consumption. In the FNO model, neural network computation is divided into two categories: 1) CGEMM: The computation form and accuracy are consistent with DFT and can be directly reused. The difference lies in the mapping between data and local storage. In CGEMM mode, the weights need to be placed in high-bandwidth input data storage and the input data is placed in low-bandwidth fixed parameter storage; 2) RGEMM: The computation form is consistent with CGEMM, but the computation accuracy is real numbers, not complex numbers. In RGEMM mode, the weights need to be placed in high-bandwidth input data storage and the input data is placed in low-bandwidth fixed parameter storage. The number of accumulations becomes 2K. In order to make full use of PE resources, by doubling the access bandwidth of the weight data, 2N data can be output, maintaining the same output bandwidth as DFT / CGEMM mode. In addition, in RGEMM mode, the PE computation unit needs to be fine-tuned. Formulas (7) and (8) show the computation formulas for complex multiplication and complex addition. Complex addition can be directly regarded as two parallel real number additions, while complex multiplication is heavily coupled and cannot be directly split into two parallel real number addition and multiplications. Assuming a+bi is the input data and c+di is the first path weight data, then ac-bd is the first path summation, which requires reconstructing subtraction into addition in NN mode; e+fi is the second path weight data, then af+be is the second path summation, which requires first converting the weight order in RGEMM mode and then connecting it to the multiplier of the imaginary part, as shown in formula (9).

[0055] (7)

[0056] (8)

[0057] (9)

[0058] The vector calculation module is responsible for post - processing operations required by the FFT module and the DFT / NN module, such as quantization, post - multiplication, etc. There is a local shared memory inside the FNO hardware accelerator, which is used to temporarily store the output results of intermediate operators, thus supporting the fusion of different - mode operators at the hardware level. The Load module can load data from external storage to local shared memory, input data storage, and fixed - parameter storage, and can also load the results of intermediate operators from local shared memory to input data storage and fixed - parameter storage. In the DFT mode, the Load module needs to perform pre - multiplication calculations whether loading input data from external storage or local shared memory. The Store module can transfer output data to local shared memory or external storage.

[0059] The host CPU and the FNO hardware accelerator perform data interaction through external storage. The host CPU can directly configure the control module of the FNO hardware accelerator and receive feedback control signals from the FNO hardware accelerator.

[0060] The method of this embodiment introduces a calculation - mode and operator - fusion decision based on static - parameter analysis. When deploying the FNO model on the hardware accelerator, according to three types of static parameters: the number of dimensions D, the number of transform points N, and the number of frequency - truncated output points M, an optimal calculation implementation scheme is automatically determined through a three - level decision process, specifically as follows:

[0061] First, determine the implementation form of the Fourier transform, as Figure 5 shown. Judge whether the number of transform points N is a specification supported by the FFT module:

[0062] If not, use Bluestein's algorithm to support these non - standard transform points, which requires three FFT calculations; more specifically, through the comparison condition of the number of frequency - truncated output points M and the number of transform points N, if M > 24log2(N), choose to use the FFT module to implement, otherwise choose the DFT module to implement;

[0063] If so, judge whether the FFT calculation can be optimized to DFT calculation through the comparison condition of the number of frequency - truncated output points M and the number of transform points N. The computational complexity of DFT is NM, and the computational complexity of FFT is Nlog2(N); more specifically, if M < log2(N), use the DFT module to implement, otherwise use the FFT module to implement.

[0064] The Bluestein algorithm consists of two FFTs and one IFFT, with a computational complexity of 3Zlog2(Z), where Z is the smallest power of 2 that is no less than 2N-1. Taking N=141 as an example, Z=512, Z / N=3.63. Considering the additional pre-multiplication and post-multiplication operations in the Bluestein algorithm, as well as its relatively complex control flow, the DFT implementation should be preferred. Therefore, in a pessimistic scenario, Z can be estimated as 4N. Thus, the computational complexity of the Bluestein algorithm can be simplified to 12Nlog2(4N) = 24Nlog2(N).

[0065] Then, a computational flow graph for the FNO operator is formed based on the number of dimensions D of the input data. For FNO2D, possible computational flow graphs include:

[0066] (1): FFT(W)-FFT(H)-CGEMM-IFFT(H)-IFFT(W);

[0067] (2): DFT(W)-DFT(H)-CGEMM-IDFT(H)-IDFT(W);

[0068] (3): IFFT(FFT,FFT)(W)-IFFT(FFT,FFT)(H)-CGEMM-IFFT(FFT,FFT)(H)-IFFT(FFT,FFT)(W).

[0069] Finally, considering the storage capacity constraints of the hardware acceleration device, a set of predefined operator fusion patterns are matched, the theoretical memory access overhead of different pattern combinations is compared, and the optimal computation flow graph and fusion strategy are selected. The predefined operator fusion patterns include FFT-FFT-IFFT, FFT-FFT, FFT-GEMM-IFFT, IFFT-IFFT, etc.

[0070] Figure 6 The computation flow graphs (I) and (II) show possible operator fusion methods. Figure 7 The possible operator fusion methods for computation flow graph (III) are shown. Figure 6 To ensure the number of transformation points conforms to the FFT module specifications, Figure 7 This is for cases where the number of transformation points does not conform to the FFT module specifications. Figure 6 In this invention, besides evaluating the fusion mode of FFT-GEMM-IFFT, it also evaluates the fusion modes of FFT-FFT and IFFT-IFFT. The number of interactions with external storage is the same in both cases, so the size of the intermediate result is key to comparing the merits of the two fusion modes. This is mainly affected by the number of input / output channels and the number of frequency-truncated output points of CGEMM. Figure 7In the process, priority is given to evaluating the fusion mode of IFFT(FFT,FFT) and the fusion possibility of IFFT(FFT,FFT)-CGEMM.

[0071] In summary, this invention constitutes a complete Fourier neural operator acceleration solution through a scalable DFT computing architecture, an independent dual-module hardware device, and a static parameter decision method. It not only fundamentally solves the memory access conflicts and low storage efficiency problems in large-scale PE arrays by reconstructing the twiddle factor access pattern using the Bluestein algorithm, achieving efficient expansion of DFT computing power; but also supports the fusion of multiple operators at the hardware level through the dual-module architecture of FFT and DFT / NN and local shared storage, significantly reducing intermediate data handling; furthermore, through static analysis based on dimension, number of points, and truncation parameters, and a three-level decision process, it can adaptively select the optimal computing path and fusion strategy. In conclusion, the method and device of this invention significantly enhance its adaptability to different transformation point numbers and dimensional structures while improving FNO computing efficiency, achieving an effective balance between computing performance, hardware efficiency, and application flexibility.

[0072] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the implementation process of the present invention has been described in detail above, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A hardware acceleration method for Fourier neural operators, based on an FNO hardware accelerator, characterized in that, The method includes: Setting respective independent FFT modules and DFT / NN modules in the FNO hardware accelerator; Performing fast Fourier transform (FFT) calculations on batch input data using the FFT module; Performing discrete Fourier transform (DFT) calculations based on the Bluestein algorithm using the DFT / NN module, and reconstructing the access pattern of rotation factors from parallel random access to parallel regular access; And, in the FNO hardware accelerator, adaptively selecting the FFT module or the DFT / NN module as the unit for performing calculations based on the analysis of static parameters such as input dimension, number of transform points, and number of frequency-truncated output points, and performing corresponding operator fusion based on a predefined operator fusion pattern; The adaptively selecting the FFT module or the DFT / NN module as the unit for performing calculations based on the analysis of static parameters such as input dimension, number of transform points, and number of frequency-truncated output points specifically includes: Judging whether the number of transform points N is a specification supported by the FFT module: If it belongs to the specification supported by the FFT module, then compare the size of the number of frequency-truncated output points M and log2(N). If M < log2(N), then select the DFT / NN module, otherwise select the FFT module; If it does not belong to the specification supported by the FFT module, then perform DFT calculations based on the Bluestein algorithm, and compare the size of M and 24log2(N). If M > 24log2(N), then select the path to implement the Bluestein algorithm via three FFT / IFFT, otherwise select the path to perform DFT calculations by the DFT / NN module.

2. The hardware acceleration method for Fourier neural operators as described in claim 1, characterized in that, The neural network calculations in the Fourier neural operator include complex matrix multiplication (CGEMM) and real matrix multiplication (RGEMM).

3. The hardware acceleration method for Fourier neural operators as described in claim 1, characterized in that, The DFT / NN module has a two-dimensional processing element (PE) array, and the PE array has a parallel dimension and an accumulation dimension; When performing DFT calculations, store the input data in a high-bandwidth input data storage, and store the rotation factors in a low-bandwidth fixed parameter storage. The parallel dimension corresponds to the channel dimension of the input data, and the accumulation dimension corresponds to the transform dimension for which DFT calculations are to be performed.

4. The hardware acceleration method for Fourier neural operators as described in claim 1, characterized in that, The FFT module is configured to perform FFT calculations on batch input [nParallel, nFFT], where nParallel is the number of tasks for parallel processing, nFFT is the number of transform points for each task, and the value of nParallel is aligned with the requirements of the DFT / NN module.

5. The hardware acceleration method for Fourier neural operators as described in claim 1, characterized in that, After selecting the unit for performing calculations, form a computational flow graph of the FNO operator according to the number of dimensions D of the input data, and combine with the storage capacity constraint of the FNO hardware accelerator, and match and compare the computational flow graph with the predefined operator fusion pattern to select the combination of the computational flow graph and the fusion pattern with the optimal theoretical memory access overhead.

6. The hardware acceleration method for Fourier neural operators as described in claim 1, characterized in that, The predefined operator fusion patterns include FFT-FFT fusion, IFFT-IFFT fusion, FFT-GEMM-IFFT fusion, and IFFT(FFT,FFT) fusion.

7. The hardware acceleration method for Fourier neural operators as described in claim 1, characterized in that, The FNO hardware accelerator also has internally: Local shared storage is used to temporarily store intermediate calculation results between the FFT module and the DFT / NN module to support cross-module operator fusion at the hardware level. The Load module is used to load data from external storage to local shared storage, input data storage, and fixed parameter storage, as well as to load the results of intermediate operators from local shared storage to input data storage and fixed parameter storage. When performing DFT calculations, the Load module performs premultiplication calculations, regardless of whether the input data is loaded from external storage or local shared storage. Store module: Used to move output data to local shared storage or external storage.

8. The hardware acceleration method for Fourier neural operators as described in claim 1, characterized in that, The Bluestein algorithm is used to reconstruct the access pattern of the rotation factor from parallel random access based on the product index nk to parallel regular access based on the difference index nk, where n is the input index and k is the output index.

Images