Massive pde neural operator pre-training method based on high-frequency enhancement module

Abstract

The application discloses a large-scale PDE neural operator pre-training method based on a high-frequency enhancement module, a partial differential equation (PDE) data set is composed into a mixed data set, a large-scale PDE neural operator is constructed, preprocessed PDE data is mapped to a latent representation space through a space-time encoder, a frequency decomposition module is used for frequency space mapping, the frequency decomposition module includes parallel high-frequency branches and low-frequency branches, and high-frequency features and low-frequency features are obtained respectively; a multi-frequency fusion module (GFM) adaptively fuses the high-frequency features and the low-frequency features through a gating mechanism; finally, a prediction head is used for processing the fused features, and final output features, i.e., predicted physical features of a next time step, are obtained. The application firstly introduces an explicit frequency division and a high-frequency enhancement mechanism, the input field is divided into low-frequency and high-frequency parts, the low-frequency branches / high-frequency branches are used for processing respectively, and thus the model can simultaneously consider global trend modeling and local gradient detail reconstruction.

Description

Technical Field

[0001] This invention relates to scientific machine learning, and more specifically to a large-scale PDE neural operator pre-training method based on a high-frequency enhancement module. Background Technology

[0002] Partial differential equations (PDEs) are core mathematical tools for describing the continuous field variations in natural and engineering systems. They characterize the evolution and distribution of physical quantities in spacetime by establishing the relationship between unknown functions and their partial derivatives. From heat conduction and fluid motion to electromagnetic fields and quantum mechanics, PDEs constitute the language for understanding the underlying physical mechanisms of the world. Their significance lies not only in providing precise mathematical expressions of phenomena but also in their ability to predict system behavior through solving them, thus supporting the development of numerous fields such as weather forecasting, aircraft design, financial modeling, and medical imaging. Existing research can be broadly divided into two categories: traditional numerical methods and data-driven methods. Traditional methods, such as FEM and FDM, approximate PDEs through discrete spatial domains, resulting in complex calculations and high computational costs.

[0003] In recent years, neural operators have emerged as a new deep learning framework. By establishing nonlinear mappings between function spaces, they enable end-to-end prediction from input parameters or initial conditions to PDE solutions. They can learn the evolution of the entire physical field and, after training, can quickly generalize to prediction tasks with different boundary conditions, equation parameters, and even geometric shapes.

[0004] Despite significant progress made by existing neural operators in PDE solving and pre-training, the following prominent shortcomings still exist:

[0005] 1. A major challenge facing neural operators is their reliance on large datasets, often obtained through expensive simulations or experiments, which can limit their application in data-scarce problems. For example, airfoil design problems typically require thousands of data samples to train neural operators, each potentially taking hours to days to generate. Similar problems exist in fluid mechanics, climate modeling, materials design, and multiphysics coupling systems, where high-fidelity simulations often require large-scale meshes, complex solvers, and lengthy integration processes. For many engineering tasks, obtaining such large training sets is not only costly but also difficult to replicate in practical deployments. Constructing sufficiently comprehensive training datasets often becomes a bottleneck limiting the performance and generalization ability of neural operator models.

[0006] Therefore, how to reduce data dependence while maintaining the model's ability to express different physical systems has become one of the core challenges that current neural operator research urgently needs to address.

[0007] 2. Limitations exist in modeling multi-scale and complex spatiotemporal dependencies.

[0008] Conventional Fourier operators tend to favor low-frequency-dominated spectral representations, with significant attenuation at high frequencies, making it difficult to effectively learn rapidly changing patterns such as local gradients, boundary layer perturbations, and small-scale turbulent structures in PDE systems. Existing methods often mix high- and low-frequency information within the same feature space, leading to attention mechanisms or frequency-domain filters that tend to optimize the low-frequency components that dominate the energy. This results in problems such as high-frequency information being overwhelmed, insufficient detail recovery, and biased local structure prediction. In real-world physics scenarios, high- and low-frequency features often serve different physical functions. For example, low frequencies dominate the overall evolution trend of the system, while high frequencies carry local structures, flow instabilities, and even important differential physical quantities.

[0009] Therefore, the unified processing architecture cannot simultaneously possess both global modeling capabilities and the ability to enhance local high-frequency details, and cannot meet the high-precision modeling requirements of cross-scale, complex physical systems.

[0010] 3. High computational complexity: Although Transformer-based operator structures have excellent local modeling capabilities and can accurately model local details and short-range dependencies through attention mechanisms, these methods are often inefficient when dealing with long-range dependencies across spatial domains, global physical propagation laws, and multi-scale coupled structures. The computational complexity increases quadratically with the image size, which can lead to a serious computational bottleneck when processing high-resolution images in PDE systems.

[0011] Therefore, such models typically rely on large-scale computing resources and are difficult to apply directly to solving practical PDEs. Summary of the Invention

[0012] Purpose of the Invention: The purpose of this invention is to address the shortcomings of existing technologies by providing a large-scale PDE neural operator pre-training method based on a high-frequency enhancement module. This method explicitly partitions the frequency components of the input field. The low-frequency component is handled by the FDM-LF (low-frequency branch), which has global modeling capabilities, for long-range dependency analysis. The high-frequency component is reconstructed using a specially designed FDM-HF (high-frequency branch) for fine-grained structure reconstruction. Utilizing the local difference characteristics of small-scale convolution kernels, this method effectively compensates for the inherent inadequacy of traditional low-frequency modules in attenuating high frequencies. Furthermore, it enhances the high-frequency response through cross-channel nonlinear interaction. In simulations of 2D compressible and incompressible Navier-Stokes equations (applications in fluid mechanics), this invention can clearly reconstruct the small and dense vortex core structure, solving the problem of detail ambiguity in turbulence simulations and greatly improving the prediction accuracy for non-smooth physical fields.

[0013] Technical solution: The present invention provides a method for pre-training large-scale PDE neural operators based on a high-frequency enhancement module, comprising the following steps:

[0014] Step 1: Obtain PDE data with different resolutions, numbers of variables, and geometric shapes to establish a hybrid dataset covering multiple physical mechanisms. Preprocess the PDE data within the hybrid dataset to obtain... The partial differential equation (PDE) data includes: 2D Navier-Stokes equations (corresponding to computational fluid dynamics scenarios, such as wing flow analysis in aerospace, aerodynamic shape optimization in the automotive industry, and viscous fluid transport in pipelines); reaction-diffusion equations (corresponding to biochemical engineering scenarios, such as concentration distribution simulation in chemical reactors, pattern formation and pollutant diffusion processes in biological morphology); and shallow water equations (corresponding to geophysical and environmental science scenarios, such as ocean circulation prediction, tsunami propagation early warning, and dam-break flood evolution simulation), etc.

[0015] Step 2: Transfer the preprocessed data The input is a large-scale PDE neural operator, which includes a spatiotemporal encoder, a frequency decomposition module (FDM), a multi-frequency fusion module (GFM), and a prediction head.

[0016] The preprocessed PDE data is mapped to the latent representation space using a spatiotemporal encoder to obtain the PDE latent representation. PDE latent representation The input is then fed into the frequency decomposition module (FDM) for frequency space mapping. The FDM module includes a parallel high-frequency branch (FDM-HF) and a low-frequency branch (FDM-LF) to obtain high-frequency and low-frequency features, respectively.

[0017] The multi-frequency fusion module GFM adaptively fuses high-frequency and low-frequency features through a gating mechanism; finally, it uses the prediction head to pair the fused features. The data is processed to obtain the final output features (such as velocity field, pressure field, fluid depth or substance concentration), which are the predicted physical features of the next time step, thereby enabling rapid and high-fidelity simulation of complex fluid dynamics, environmental hydraulics and biochemical reaction processes.

[0018] This invention employs a mixed training approach using six datasets. These partial differential equation datasets differ in resolution, number of variables, and geometry. Direct sampling from the original data would result in batches of varying sizes, impacting training efficiency. To effectively address the challenges of computational efficiency, training stability, and generalization robustness in multi-physics joint training, step 1 involves preprocessing the mixed data as follows:

[0019] 1. Select a resolution of 128×128, which can match most data;

[0020] 2. To unify the number of variables on different partial differential equations, fill all datasets along the channel dimension to match the maximum number of channels;

[0021] 3. For datasets with irregular geometries, an additional mask channel is introduced to encode the specific geometric configuration of each partial differential equation instance;

[0022] 4. When training with multiple partial differential equation datasets, the differences between the datasets can lead to an imbalance in training progress. Data sampling can be performed by balancing the probabilities between the datasets.

[0023] 5. Add small-scale noise to the data to improve the model's generalization and robustness. The final processed data format is as follows: Where B is the batch size, H is the height, W is the width, T is the custom time step, and C is the number of channels.

[0024] Furthermore, the processing method of the spatiotemporal encoder is as follows:

[0025] First, the pre-processed Divide the data into several non-overlapping patches of a specified size, project each patch into a high-dimensional space, and then add positional encoding to obtain the desired result. The expression is as follows:

[0026] ;

[0027] in It is a pre-defined patch. It is a positional encoding, where N is the number of patches and D is the embedding dimension of the patches;

[0028] Transform the obtained X dimension back into It maps to a fixed dimension, compresses temporal information, and extracts information from adjacent time steps. Extracting PDE latent representations ;

[0029] ; ;

[0030] in, It is the learnable weight matrix at time step t. This refers to the number of output channels. It is the position code at time step t.

[0031] Furthermore, the PDE latent representation output by the spatiotemporal encoder is first processed... Perform a Fourier transform to map to the frequency domain. Then, the low-frequency branch FDM-LF and the high-frequency branch FDM-HF were used respectively to... Perform feature extraction;

[0032] The low-frequency branch FDM-LF, for A parametric transformation is performed on the first 32×32 region of the frequency domain (this frequency band corresponds to the low-frequency principal components of the input signal, containing the global structure and main energy distribution of the scene). Two linear transformations are applied to each spectral block to achieve adaptive filtering and reconstruction of the low-frequency modes.

[0033] ;

[0034] Where σ is the activation function. and These are the learnable matrices with real and imaginary parts, respectively. and It corresponds to the bias term; and These represent the real and imaginary frequency domain coefficients of the feature in the frequency domain, respectively. It refers to the real part feature in the frequency domain. This refers to the imaginary part characteristic in the frequency domain;

[0035] Finally, perform an inverse Fourier transform. Low-frequency features were obtained. ;

[0036] The high-frequency branch FDM-HF constructs a frequency domain mask that preserves high frequencies and suppresses low frequencies. For the frequency domain features after Fourier transform The low-frequency region in the upper left corner (32×32) is set to zero. Then Transform to the spatial domain This extracts high-frequency information; to enhance the extracted high-frequency information, a depthwise convolution module is used. Local differential enhancement for each channel highlights high-frequency structures; subsequent pointwise convolution and linear blending along the channel dimension reconstruct high-frequency correlations across channels, expressed as:

[0037] ;

[0038] In the above formula, DWConv is depthwise convolution, GN is group normalization, and PWConv is pointwise convolution. For activation functions;

[0039] The high-frequency branch FDM-HF is then subjected to a final residual connection to obtain the high-frequency characteristics. This ensures that local details are enhanced while preserving the original high-frequency information, thereby significantly improving the model's expressive power in boundary layers, small-scale changes, and high-frequency dynamics.

[0040] Furthermore, the multi-frequency fusion module GFM learns whether each spatial location should be biased towards high or low frequencies through pixel-level gating, possessing the dynamic selection capability of local details and global trends. The specific fusion method is as follows:

[0041] First, construct the gated input: ;

[0042] in, High-frequency characteristics It is a low-frequency characteristic;

[0043] Then, the fusion weights are generated through a convolutional gating network, as shown in the following expression:

[0044] ;

[0045] That is, the softmax weights are obtained. ;

[0046] in, For the fusion weights of low-frequency features, For the fusion weights of high-frequency features, For activation functions;

[0047] Obtain the final output features : ;

[0048] This indicates element-wise multiplication.

[0049] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0050] 1. High-frequency information enhancement improves local structure recovery capability. The high-frequency enhancement branch of this invention utilizes the local difference and edge sensitivity characteristics of small-scale convolution to effectively compensate for the inherent limitations of traditional low-frequency processing in high-frequency attenuation, resulting in significant improvements in detail prediction tasks such as boundary layers, vortex core structures, and singularity-like regions. In simulations of 2D compressible and incompressible Navier-Stokes equations (applications in fluid mechanics), this invention can clearly reconstruct small and dense vortex core structures, solving the problem of detail blurring in turbulence simulations and greatly improving the prediction accuracy for non-smooth physical fields.

[0051] 2. The cross-scale modeling capability is significantly enhanced, improving the generalization performance of complex physical systems.

[0052] By employing a low-frequency-high-frequency divide-and-conquer architecture, this invention can simultaneously capture the global evolution dynamics of low-frequency control, the local physical structure changes carried by high-frequency components, and cross-scale coupling relationships. Unlike existing neural operators that use a unified feature space for hybrid modeling, this invention achieves a frequency-level processing mechanism that is more in line with the physical characteristics of PDEs, effectively avoiding the problem of mutual interference between high and low frequency information.

[0053] In the Navier-Stokes equations with scalar transport (Ns-cond), the model can accurately predict the macroscopic convection motion of the fluid velocity field (dominantly low frequency) and maintain the sharp edges of passive scalars (such as pollutant concentration or temperature) in the diffusion process (dominantly high frequency), thus exhibiting superior decoupling and generalization capabilities in complex multiphysics coupled systems.

[0054] 3. In the joint pre-training task on six different types or coefficient PDE datasets, this invention achieved four state-of-the-art results compared to other state-of-the-art models. In downstream task experiments, it successfully transferred to three types of PDE datasets, demonstrating the model's strong generalization and accuracy. Attached Figure Description

[0055] Figure 1 A schematic diagram of the overall network structure;

[0056] Figure 2 This is a schematic diagram of the processing flow of the present invention;

[0057] Figure 3 This is a schematic diagram of the NS(1e-4) physical field visualization in the embodiment;

[0058] Figure 4 This is a schematic diagram of the physical field visualization of CNS(1,0.01) in the embodiment;

[0059] Figure 5 This is a schematic diagram of the Ns-cond physical field in the embodiment. Detailed Implementation

[0060] The technical solution of the present invention will be described in detail below, but the scope of protection of the present invention is not limited to the embodiments described.

[0061] like Figure 1 and Figure 2 As shown, the large-scale PDE neural operator pre-training method based on a high-frequency enhancement module of the present invention includes the following steps:

[0062] Step 1: Obtain PDE data with different resolutions, numbers of variables, and geometries. The PDE data includes two-dimensional Navier-Stokes equations, reaction-diffusion equations, and shallow water equations. Establish a hybrid dataset, and preprocess each PDE data point within the hybrid dataset to obtain... ;

[0063] Step 2: Transfer the preprocessed data The input is a large-scale PDE neural operator, which includes a spatiotemporal encoder, a frequency decomposition module (FDM), a multi-frequency fusion module (GFM), and a prediction head.

[0064] The preprocessed PDE data is mapped to the latent representation space using a spatiotemporal encoder to obtain the PDE latent representation. PDE latent representation The input is then fed into the frequency decomposition module (FDM) for frequency space mapping. The FDM module includes a parallel high-frequency branch (FDM-HF) and a low-frequency branch (FDM-LF) to obtain high-frequency and low-frequency features, respectively.

[0065] The multi-frequency fusion module GFM adaptively fuses high-frequency and low-frequency features through a gating mechanism; finally, it uses the prediction head to pair the fused features. The process is performed to obtain the final output features, which are the predicted physical features of the next time step.

[0066] The method for preprocessing the mixed data in step 1 of this embodiment is as follows:

[0067] Step 1.1: Select a resolution of 128×128;

[0068] Step 1.2: Fill all datasets along the channel dimension to match the maximum number of channels;

[0069] Step 1.3: For datasets with irregular geometries, introduce additional mask channels to encode the specific geometric configuration for each instance of a partial differential equation;

[0070] Step 1.4: When training with multiple partial differential equation datasets, the differences between the datasets can lead to an imbalance in training progress. Data sampling is performed by balancing the probabilities between the datasets.

[0071] Step 1.5: Add a small amount of noise to the data. The final processed data will look like this: Where B is the batch size, H is the height, W is the width, T is the custom time step, and C is the number of channels.

[0072] Figure 1 In the input feature X, the total number of channels is C. It is the input feature component of the k-th head; The learnable weight matrix of the k-th head.

[0073] The processing method of the spatiotemporal encoder in this embodiment is as follows:

[0074] First, the pre-processed Divide the data into several non-overlapping patches of a specified size, project each patch into a high-dimensional space, and then add positional encoding to obtain the desired result. The expression is as follows:

[0075] ;

[0076] in It is a pre-defined patch. It is the positional encoding, B is the batch size, N is the number of patches, and D is the embedding dimension of the patches;

[0077] Transform the obtained X dimension back into It maps to a fixed dimension, compresses temporal information, and extracts information from adjacent time steps. Extracting PDE latent representations ;

[0078] ; ;

[0079] in, It is a learnable weight matrix. It is the position code at time step t. This refers to the number of output channels.

[0080] In this embodiment, the PDE latent representation output by the spatiotemporal encoder is first... Perform a Fourier transform to map to the frequency domain. Then, the low-frequency branch FDM-LF and the high-frequency branch FDM-HF were used respectively to... Perform feature extraction;

[0081] The low-frequency branch FDM-LF, for A parameterized transformation is performed on the first 32×32 region of the frequency domain, and two layers of linear transformation are applied to each spectral block to achieve adaptive filtering and reconstruction of low-frequency modes:

[0082] ;

[0083] Where σ is the activation function. and These are the learnable matrices with real and imaginary parts, respectively. and It corresponds to the bias term. and These represent the real and imaginary frequency domain coefficients of the feature in the frequency domain, respectively. It refers to the real part feature in the frequency domain. This refers to the imaginary part characteristic in the frequency domain;

[0084] Finally, perform an inverse Fourier transform. Low-frequency features were obtained. ;

[0085] The high-frequency branch FDM-HF constructs a frequency domain mask that preserves high frequencies and suppresses low frequencies. For the frequency domain features after Fourier transform The low-frequency region in the upper left corner (32×32) is set to zero. Then Transformed to the spatial domain, the expression is: Then, the depthwise convolution module is used to implement... Local difference enhancement for each channel; followed by pointwise convolution and linear blending along the channel dimension to reconstruct high-frequency correlations across channels, expressed as:

[0086] ;

[0087] In the above formula, DWConv is depthwise convolution, GN is group normalization, and PWConv is pointwise convolution. For activation functions;

[0088] The high-frequency characteristics are obtained by performing a residual connection again on the high-frequency branch FDM-HF. .

[0089] In this embodiment, the multi-frequency fusion module GFM learns whether each spatial location should be biased towards high or low frequencies through pixel-level gating, and has the ability to dynamically select between local details and global trends. The specific fusion method is as follows:

[0090] First, construct the gated input: ;

[0091] in, High-frequency characteristics It is characterized by low frequency.

[0092] Then, the fusion weights are generated through a convolutional gating network, as shown in the following expression:

[0093] ;

[0094] That is, the softmax weights are obtained. ;

[0095] in For the fusion weights of low-frequency features, The fusion weights are for high-frequency features; For activation functions;

[0096] Obtain the final output features : ;

[0097] element-wise multiplication

[0098] To further verify the feasibility and technical effect of the present invention, this embodiment applies the technical solution to the following scenarios, and the specific experimental analysis is as follows.

[0099] In this embodiment, experiments were conducted on mixed datasets with different data sources and parameters, including FNO, PDEBench, PDEArena, and CFDbench. L2RE was used as the evaluation metric, where a lower L2RE indicates better performance.

[0100] Main Results: Table 1 presents a comparison of the experimental results of this invention with other models in the pre-training dataset. The first row of Table 1 specifies the type of PDE dataset and parameter settings, while the first column lists the baseline models used for comparison. The experiments were pre-training experiments on a mixed dataset, and all models were trained from scratch on the mixed dataset.

[0101] During the pre-training phase, the present invention demonstrated strong performance on six PDE datasets, achieving state-of-the-art results on four of them. These results clearly validate the effectiveness of the present invention in handling complex PDE systems, highlighting its superior performance and versatility in PDE modeling.

[0102] Downstream tasks: To evaluate the effectiveness of the pre-trained model of this invention in improving performance in different PDE downstream tasks, experiments were conducted to test its broader applicability, and the results are shown in Table 2.

[0103] Here, three downstream tasks are selected: NS (1e-4), CNS (1,0.01), and Ns-cond. The pre-trained weights are loaded and fine-tuned for 500 rounds.

[0104] The results are shown in Table 1. Compared with advanced pre-trained models, the model of this invention achieved two SOTA results on three datasets, proving the effectiveness of the model of this invention. Then, the prediction results of the model after 500 rounds of fine-tuning on the three datasets were visualized. The first column is the true value, the second column is the predicted value, and the third column is the error.

[0105] Figure 3The result of the Vx field evolution at the last time step of NS(1e-4) is as follows. Figure 3 The first column is the actual value, the second column is the predicted value, and the third column is the error value.

[0106] Figure 4 The evolution results of the pressure field, Vx field, and Vy field at the last time step of CNS(1,0.01) are shown. Figure 4 The first column contains the actual values. Figure 4 The second column shows the predicted value, and the third column shows the corresponding error value.

[0107] Figure 5 The graph shows the evolution of the Vx and Vy fields at the last time step of Ns-cond. The first column represents the true values, the second column represents the predicted values, and the third column represents the corresponding error values.

[0108] Table 1. Comparison results of joint pre-training of different models on 6 datasets.

[0109]

[0110] Table 2. Results of 500 rounds of fine-tuning of different models on different downstream tasks with pre-trained weights.

[0111]

[0112] In summary, this invention, for the first time, constructs a high-frequency branch that utilizes the local difference and edge sensitivity of small-scale convolution to effectively compensate for the inherent limitations of traditional low-frequency processing in high-frequency attenuation. This results in significant improvements in detailed prediction tasks such as boundary layers, vortex kernel structures, and singularity-like regions. Furthermore, this low-frequency-high-frequency divide-and-conquer architecture simultaneously captures the global evolutionary dynamics of low-frequency control, the local physical structure changes carried by high-frequency components, and cross-scale coupling relationships. Unlike existing neural operators that use a unified feature space for hybrid modeling, this invention achieves a frequency-level processing mechanism that better conforms to the physical characteristics of PDEs, effectively avoiding the problem of mutual interference between high and low-frequency information.

Claims

1. A large-scale PDE neural operator pre-training method based on a high-frequency enhancement module, characterized in that, The method comprises the following steps: Step 1, obtaining partial differential equation (PDE) data with different resolutions, variable numbers and geometric shapes, the PDE data including two-dimensional Navier-Stokes equations for wing flow analysis in aerospace, pollutant diffusion reaction-diffusion equations and shallow water equations for ocean circulation prediction, establishing a mixed data set, and preprocessing each PDE data in the mixed data set; Step 2, inputting the preprocessed data into a large-scale PDE neural operator, the large-scale PDE neural operator comprising a space-time encoder, a frequency decomposition module (FDM), a multi-frequency fusion module (GFM) and a prediction head; mapping the pre-processed PDE data to a latent representation space by a spatio-temporal encoder to obtain a PDE latent representation , the PDE latent representation Then input to the frequency decomposition module FDM for frequency space mapping, the frequency decomposition module FDM includes parallel high-frequency branch FDM-HF and low-frequency branch FDM-LF, respectively, to obtain high-frequency features and low-frequency features; The multi-frequency fusion module (GFM) adaptively fuses high-frequency features and low-frequency features through a gating mechanism; The final fused features are processed by the prediction head to output predicted next time step physical features.

2. The pre-training method of large-scale PDE neural operators based on high-frequency augmentation module according to claim 1, characterized in that, The preprocessing method of the step 1 is as follows: Step 1.1, selecting a resolution of 128x128; Step 1.2, filling all data sets along the channel dimension to match the data set with the maximum channel number; Step 1.3, for data sets with irregular geometric shapes, an additional mask channel is introduced to encode the specific geometric configuration of each partial differential equation instance; Step 1.4, when multiple partial differential equation data sets are used for training, the differences between the data sets will cause uneven training progress, and the data is sampled by balancing the probabilities between the data sets; Step 1.5, Add small-scale noise to the data, and the final processed data is in the form of: ; Wherein, B is the batch size, H is the height, W is the width, T is the custom time step, and C is the number of channels.

3. The method for pre-training large-scale PDE neural operators based on a high-frequency enhancement module according to claim 2, characterized in that, Obtaining preprocessed data After that, the spatio-temporal encoder processes it, in the following way: First, after pretreatment According to the specified size, the image is divided into several non-overlapping patches. Each patch is projected into a high-dimensional space, and then position encoding is added to obtain The expression is as follows: ； wherein is a divided patch, is a position encoding, N is the number of patches, and D is the embedding dimension of the patch. re-transforming the resulting X-dimension into and mapping to fixed dimensions, compressing time information, and extracting PDE latent representations from adjacent time steps ;​ ； ； wherein, is a learnable weight matrix at time step t, is the number of output channels, is the position encoding at time step t.

4. The pre-training method of large-scale PDE neural operators based on high-frequency augmentation module according to claim 1, characterized in that, PDE latent representation output by the spatio-temporal encoder Do Fourier transform, map to frequency domain space ; then use low-frequency branch FDM-LF and high-frequency branch FDM-HF respectively feature extraction; The low-frequency branch FDM-LF, to The low-frequency branch FDM-LF, to The low-frequency branch FDM-LF, to ； where σ is an activation function, and are learnable matrices for real and imaginary parts, respectively, and are corresponding bias terms; and denote real and imaginary frequency domain coefficients of the feature in the frequency domain, respectively; denotes a real part feature in the frequency domain, denotes an imaginary part feature in the frequency domain; a final inverse Fourier transform , to obtain low frequency features ; The high-frequency branch FDM-HF constructs a frequency domain mask that preserves high frequencies and suppresses low frequencies For the frequency domain features after Fourier transform The low-frequency region of the upper left corner 32*32 is set to zero Then, the Transformed to the spatial domain, the expression is Then, the depth convolution module is used to realize Local differential enhancement of each channel; then, point-by-point convolution is performed, linear mixing is performed in the channel dimension, and the expression is: ； In the above equation, DWConv is a depthwise convolution, GN is group normalization, PWConv is a pointwise convolution, is an activation function. The high-frequency branch FDM-HF makes a residual connection again to obtain high-frequency features .

5. The pre-training method of large-scale PDE neural operators based on high-frequency augmentation module according to claim 1, characterized in that, The multi-frequency fusion module (GFM) learns through pixel-level gating whether each spatial position should be biased towards high frequency or low frequency, and has the dynamic selection ability of local details and global trends, and the specific fusion method is as follows: First, construct the gated input: ; wherein is a high frequency feature, is a low frequency feature; Then, the fusion weight is generated through the convolution gating network, and the expression is as follows: ； y = softmax (w) = exp (w) / Σ exp (w) ; wherein, is a fusion weight for low-frequency features, is a fusion weight for high-frequency features, is an activation function; obtaining final output features : ; represents an element-wise multiplication.

Images