Task adaptive parameter adjustment method and system for weather and climate basic model

Abstract

The invention relates to an artificial intelligence technology, in particular to a task adaptive parameter adjusting method and system for a weather and climate basic model. The adjusting method comprises the steps that a weather and climate basic model is initialized, the weather and climate basic model comprises an encoder, a main body network and a decoder, and the main body network comprises a parameter efficient fine adjustment framework model provided with a task self-adaptive dynamic prompt module and a random snow-consuming guided self-adaptive selection module which work cooperatively; performing task self-adaptive dynamic prompt through a task self-adaptive dynamic prompt module, generating soft prompt lexical elements, and fusing the input weather data with the soft prompt lexical elements to obtain an enhanced lexical element sequence; and through a random snow-consuming guided adaptive selection module, random snow-consuming guided adaptive selection is carried out, and parameter fine tuning of the weather and climate basic model is realized. Model parameters can be dynamically, selectively and finely adjusted according to downstream tasks, a very small number of trainable parameters are used, and the calculation and storage cost is remarkably reduced.

Description

Technical Field

[0001] This invention relates to artificial intelligence technology, and in particular to a method and system for adaptive parameter adjustment of a weather and climate basic model. Background Technology

[0002] In recent years, significant progress has been made in the field of artificial intelligence, particularly in large-scale pre-trained models, also known as foundation models (FMs). In meteorology, Weather and Climate Foundation Models (WCFMs), through pre-training on massive amounts of data, have learned general representations of atmospheric processes and demonstrated strong generalization capabilities. To apply these models to diverse downstream tasks (such as weather forecasting, forecast post-processing, and downsampling), fine-tuning is typically required.

[0003] However, as the parameter size of WCFMs models continues to expand, often reaching billions, the traditional "full fine-tuning" method faces severe challenges. Full fine-tuning requires updating all parameters of the model, which not only places extremely high demands on computing resources, making it too costly in practical applications, but also incurs huge storage overhead in multi-task scenarios, as maintaining a complete copy of the model parameters for each task would lead to huge storage costs.

[0004] To address these issues, Parameter-Efficient Fine-Tuning (PEFT) techniques have emerged. PEFT aims to achieve performance similar to or even better than full fine-tuning by updating only a small subset (typically much less than 1%) of the model's parameters. However, most current mainstream PEFT methods are designed for Natural Language Processing (NLP) or Computer Vision (CV) tasks, and their performance is often unsatisfactory when directly applied to weather and climate baseline models. The fundamental reason is that they fail to address the unique challenges posed by meteorological data:

[0005] 1. Data Heterogeneity: Unlike visual models with standardized RGB three-channel input or language models with a unified word embedding space, downstream tasks in weather and climate involve a wide variety of physical variables (such as temperature, humidity, and wind speed), spatial resolution (e.g., from 5.625° to 0.25°), and spatiotemporal coverage (e.g., global and regional). This requires models to be able to adapt to the inherent characteristics of different tasks.

[0006] 2. Differences in the Role of Parameters: Most existing PEFT methods apply the same set of trainable parameters to all downstream tasks, ignoring the fact that different parameters may play drastically different roles in different tasks. For example, the key parameters for regional precipitation prediction tasks may be very different from those required for meteorological downsampling tasks.

[0007] Therefore, there is an urgent need for a new, efficient parameter fine-tuning method specifically designed for weather and climate fundamental models, capable of adapting to the characteristics of data from different downstream tasks. Summary of the Invention

[0008] To address the issues of poor performance and adaptability of existing efficient parameter fine-tuning methods when applied to weather and climate fundamental models, this invention provides a task-adaptive parameter adjustment method and system for weather and climate fundamental models. By constructing the efficient parameter fine-tuning framework ClimatePEFT, it is possible to dynamically and selectively fine-tune model parameters according to the specific attributes of downstream tasks. This achieves or even surpasses the performance of fully fine-tuned models using only a very small number of trainable parameters, while significantly reducing computational and storage costs.

[0009] On one hand, embodiments of the present invention provide a task-adaptive parameter-efficient fine-tuning framework model, applied to the Transformer module of a pre-trained main network, and implemented on a pre-trained weather and climate basic model; the parameter-efficient fine-tuning framework model includes a collaborative task-adaptive dynamic prompting module and a random Fisher-guided adaptive selection module;

[0010] The task-adaptive dynamic prompting module is connected to the encoder and receives input data through the encoder's embedding layer; the random Fisher-Price-guided adaptive selection module is connected to the decoder.

[0011] The task-adaptive dynamic prompting module is used to enable the model to perceive and adapt to the data heterogeneity of different downstream tasks. By extracting and transforming the weight information of the embedding layer in the encoder, dynamic soft prompting words are generated and injected into the input of the pre-trained main network to achieve context-aware feature recalibration for downstream tasks.

[0012] The randomized Fisher-guided adaptive selection module is used to identify and update only the subset of parameters most critical to the current downstream task, while retaining the general knowledge learned in the pre-trained model. It uses Fisher information to quantify the importance of model parameters and determine which model parameters need to be updated and which need to be frozen during model training.

[0013] Preferably, the implementation process of the task adaptive dynamic prompting module includes:

[0014] Obtain the embedding layer weight matrix in the encoder used to map the input weather and climate data to words. The embedding layer weight matrix contains hidden dimensional information that reveals climate features, dimensional information of the input physical variables, and spatial resolution information of the block embedding core size. Reshape the embedding layer weight matrix.

[0015] An adapter is designed to extract internal patterns. The adapter captures the inherent patterns in different dimensions through upsampling, subsampling, and the GELU activation function. The output matrix is ​​obtained after the internal patterns are extracted.

[0016] External pattern extraction is performed, and the output matrix extracted from the internal pattern is applied to the layer normalization function and self-attention operation to capture the external pattern that establishes the coupling analysis between the input physical variables and spatial resolution features, so as to establish the coupling relationship between different features.

[0017] By using a fully connected linear layer, the coupling relationships between different features established by external pattern extraction are converted into soft cue words.

[0018] The generated soft cue lexicons are appended to the beginning of the input lexicon sequence of each Transformer module in the main network to guide the model to focus on the specific attributes of the current task during forward propagation.

[0019] Preferably, the implementation process of the Fisher-guided adaptive selection module includes:

[0020] Importance measurement: The Fisher information matrix is ​​approximated by the squared gradient of the model's loss function on the training data to quantify the importance of model parameters and measure the degree of impact of small changes in model parameters on the model's output distribution.

[0021] To stabilize the selection of model parameters, randomness is introduced: a random component that decays linearly with the number of training steps is added to the calculated Fisher information matrix to reduce the interference of initial noise on the judgment of the importance of model parameters.

[0022] Model parameter selection and update: Based on the Fisher information matrix values ​​after adding random components, select the top-k parameters with the highest scores; then, generate a binary Fisher mask matrix, in which the selected top-k parameters are set to 1 and the rest to 0, and the sparsity level of the Fisher mask matrix is ​​controlled by hyperparameters.

[0023] On the other hand, embodiments of the present invention provide a task-adaptive parameter adjustment method for a weather and climate basic model, comprising the following steps:

[0024] S1. Prepare input data and initialize the weather and climate basic model; the weather and climate basic model includes an encoder, a main network and a decoder, and the main network includes the above-mentioned parameter efficient fine-tuning framework model;

[0025] S2. The task-adaptive dynamic prompting module of the parameter-efficient fine-tuning framework model performs task-adaptive dynamic prompting, generates soft prompting words, and merges the input weather data with the soft prompting words to obtain the enhanced word sequence.

[0026] S3. Through the stochastic Fisher-guided adaptive selection module of the parameter-efficient fine-tuning framework model, stochastic Fisher-guided adaptive selection is performed to achieve parameter fine-tuning of the basic weather and climate model.

[0027] Preferably, step S2, which involves providing adaptive dynamic prompts for the task, includes the following steps:

[0028] S21. Extract the embedding layer weights of the encoder in the basic weather and climate model;

[0029] S22. Perform internal pattern extraction and external pattern extraction to obtain a tensor containing internal and external pattern information;

[0030] S23. Pass the tensor containing internal and external pattern information through a fully connected linear layer to generate soft cue words;

[0031] S24. Integrate the input weather data with soft cue words to obtain the enhanced word sequence.

[0032] Preferably, step S2, which involves random Fisher-guided adaptive selection, includes the following steps:

[0033] S31. Calculate Fisher information: Calculate the loss of the model during one forward propagation; based on the calculated loss, calculate the gradient of all trainable parameters θ of the model with respect to the log-likelihood, and take the square of it as an approximation of Fisher information.

[0034] S32. Introduce annealing random perturbations during model training to correct Fisher information;

[0035] S33. Based on the corrected Fisher information, select the Top-k parameters and generate a mask;

[0036] S34. Masked parameter update: When updating parameters through backpropagation, the calculated gradient is multiplied element-wise with the generated mask.

[0037] In another aspect, embodiments of the present invention provide a weather and climate forecasting method, including steps S1-S3 of the above-described task adaptive parameter adjustment method, and the following steps:

[0038] S4. The enhanced word sequence is fed into the weather and climate basic model with fine-tuned parameters. The weather and climate basic model with fine-tuned parameters is used for layer-by-layer processing and inference to output the forecast results.

[0039] Furthermore, embodiments of the present invention also provide a weather and climate forecasting system, comprising the following modules:

[0040] The parameter adjustment module is implemented using steps S1-S3 of the above-mentioned task adaptive parameter adjustment method to perform task adaptive parameter adjustment on the weather and climate basic model and obtain the weather and climate basic model after parameter fine-tuning.

[0041] The prediction module is used to feed the enhanced lexical sequence into the parameter-fine-tuned weather and climate basic model, and then use the parameter-fine-tuned weather and climate basic model to perform layer-by-layer processing and inference to output the forecast results.

[0042] The present invention has the following advantages and effects compared with the prior art:

[0043] 1. Based on a unified pre-trained meteorological model, this invention constructs a task-adaptive parameter efficient fine-tuning framework, ClimatePEFT, applicable to basic weather and climate models. Through efficient fine-tuning strategies, it performs targeted and scalable model fine-tuning for different meteorological forecasting or climate analysis tasks, improving the accuracy and adaptability of the model in multi-type and multi-scale meteorological tasks, and further promoting the low-cost deployment of meteorological artificial intelligence models in extreme weather forecasting and climate analysis.

[0044] 2. Superior performance and efficient parameters: Due to its task-adaptive fine-tuning strategy, this invention achieves performance comparable to fully fine-tuned methods that train all parameters, while only training approximately 0.28% of the parameters. In certain tasks, such as ensemble forecast post-processing, its performance even surpasses that of fully fine-tuned methods. This significantly outperforms existing PEFT methods, which exhibit a marked performance gap compared to fully fine-tuned methods in meteorological tasks.

[0045] 3. Strong task adaptability: Since the TADP module can perceive the heterogeneity of task input features and the SFAS module can identify the key parameters of specific tasks, this invention can effectively cope with the diversity of downstream tasks in the weather and climate field (such as different variables, resolutions, and spatiotemporal ranges), and solve the problem of existing general PEFT methods being unsuitable for various situations.

[0046] 4. Significantly reduced resource consumption: Since only a very small number of parameters need to be updated and stored, this invention greatly reduces the computing resources and storage space required for fine-tuning WCFMs, making it possible to deploy and apply advanced large-scale climate models under resource-limited conditions, and promoting the popularization of climate modeling tools.

[0047] 5. Component Synergy: Ablation experiments have shown that both TADP and SFAS modules perform well when used alone. Due to the synergistic gain effect between the two modules, their performance is even better when combined in the complete ClimatePEFT framework. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the ClimatePEFT framework model in an embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram of the TADP module structure of the ClimatePEFT framework model;

[0050] Figure 3 This is a schematic diagram of the SFAS module structure of the ClimatePEFT framework model;

[0051] Figure 4 This is a comparison diagram of the processing flow of the present invention ClimatePEFT and existing efficient fine-tuning methods in downsampling tasks;

[0052] Figure 5 This diagram compares the processing flow of the present invention ClimatePEFT and existing efficient fine-tuning methods in precipitation prediction tasks. Detailed Implementation

[0053] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments, but the implementation of the present invention is not limited thereto.

[0054] Example

[0055] This embodiment proposes a task-adaptive parameter tuning method and system for a basic weather and climate model. In the implementation process, a task-adaptive parameter-efficient fine-tuning framework, ClimatePEFT, is first constructed. This framework is applied to the Transformer module of the main network and implemented on a pre-trained basic weather and climate model (such as the Aurora model). Specifically, it includes two collaborative innovative modules: a Task-Adaptive Dynamic Prompting (TADP) module and a Stochastic Fisher-Guided Adaptive Selection (SFAS) module, as shown below. Figure 1 As shown in the diagram. The task-adaptive dynamic prompting module is connected to the encoder and receives input data through the encoder's embedding layer; the random Fisher-guided adaptive selection module is connected to the decoder.

[0056] from Figure 1As can be seen, the input data of the ClimatePEFT framework model first enters the embedding layer of the encoder; then the data flows through a backbone network containing multiple (L) "Transformer blocks"; finally, it is output by the decoder to complete different downstream tasks, such as "post-processing", "downsampling", and "prediction". Most parameters of the backbone network and decoder are frozen; the fine-tuning mainly occurs in the output layer related to specific downstream tasks, as well as a subset of backbone network parameters controlled by the TADP and SFAS modules. The TADP and SFAS modules act on the core modules of the backbone network (i.e., the backbone network), they act on the Transformer modules of the backbone network, and work together to achieve efficient adaptive fine-tuning.

[0057] like Figure 2 The Task Adaptive Dynamic Prompt (TADP) module enables the model to perceive and adapt to the data heterogeneity of different downstream tasks, such as different input variables and resolutions. Its core idea is to utilize the weights of the embedding layer in the encoder, as these weights contain task-specific information. The TADP module extracts and transforms this weight information to generate dynamic "soft prompts" and injects these soft prompts into the input of the pre-trained main network, achieving context-aware feature recalibration for downstream tasks. The specific implementation steps are as follows:

[0058] The input to the TADP module is the embedding layer weight matrix used in the encoder to map the input weather and climate data to tokens. Simplified representation as Where D is the hidden dimension revealing climate characteristics, V represents the total number of input physical variables, and P... h ×P w Corresponding to the core size of the patch embedding, it contains spatial resolution information. Therefore, this weight matrix includes hidden dimension (climate feature) information, input physical variable dimension information, and patch embedding core size (spatial resolution information), etc. The embedding layer weight matrix E is then reshaped into...

[0059] Internal pattern extraction: To capture internal patterns across different dimensions, three sequentially connected adapters were designed: HW-Adapter, V-Adapter, and D-Adapter. HW-Adapter takes the embedding layer weights as input and processes dimensions related to spatial resolution; V-Adapter processes dimensions related to input physical variables, such as capturing physical constraints between different variables; D-Adapter processes hidden dimensions related to climate features.

[0060]

[0061] Shift(·) transposes the last dimension to the first dimension, while Reshape(·) combines the input physical variable V and the spatial resolution feature P. h P w ; This represents the output matrix after processing by the HW-Adapter. This represents the output matrix after processing by the V-Adapter. This represents the output matrix after processing by the D-Adapter.

[0062] External pattern extraction: Output matrix of internal pattern extraction It is applied to the layer normalization function Norm(·) and the self-attention operation SA(·) to capture and establish the input physical variables (total of V) and spatial resolution features (size P). h P w External patterns for coupling analysis between different features are used to establish coupling relationships between them.

[0063]

[0064] in, E represents the normalized output matrix after dimensional flipping; query E key and E value They are normalized The query, key, and value characteristics.

[0065] Generating soft prompt tokens: A fully connected linear layer (FC-linear) is used to process the coupling relationships between different features established by external pattern extraction, converting them into soft prompt tokens.

[0066]

[0067] Where P is the length of the prompt.

[0068] Injection Model: The generated soft cue lexicons are appended to the beginning of the input lexicon sequence of each Transformer module in the main network, thereby guiding the model to focus on the specific attributes of the current task during forward propagation.

[0069] In summary, the input to the TADP module is the embedding layer weights of the encoder. The weights first undergo internal pattern extraction via three cascaded adapters (HW-Adapter, V-Adapter, D-Adapter), which capture the inherent patterns across different dimensions (such as space and variables) through upsampling, subsampling, and the GELU activation function. The features extracted from the internal patterns then undergo layer normalization and self-attention layers for external pattern extraction, establishing coupling relationships between different features. Finally, a fully connected layer generates the final "soft prompt" lexical unit, with dimensions p×D, which is then injected into the Transformer module of the main network.

[0070] like Figure 3 The Random Fisher-Guided Adaptive Selection (SFAS) module identifies and updates only the subset of parameters most critical to the current downstream task, while preserving general knowledge learned from the pre-trained weather and climate baseline model. Its core idea is to use Fisher information to quantify the importance of model parameters, determining which parameters need to be updated during model training and which need to be kept frozen. The specific implementation steps are as follows:

[0071] Importance Measurement: The Fisher information matrix is ​​calculated to quantify the importance of model parameters. Fisher information measures the impact of small changes in model parameters on the model's output distribution. In practice, the Fisher information matrix F can be efficiently approximated by the squared gradient of the loss function on the training data of the weather and climate baseline model. θ :

[0072]

[0073] Where p θ (y j |x j ) is determined by the parameter. The basic weather and climate model under given input x j The time generated about y j The output distribution is given by N, where N is the number of data points. This indicates taking the derivative with respect to θ.

[0074] Introducing randomness to stabilize model parameter selection: Since the calculation of the Fisher information matrix may be significantly affected by noise in the initial stages of model parameter fine-tuning, this embodiment introduces an annealed stochastic component to stabilize the model parameter selection process. Specifically, a random component that decays linearly with the number of training steps is added to the calculated Fisher information matrix. This injection of randomness can reduce the interference of initial noise on the judgment of the importance of model parameters, making the model parameter selection process more stable.

[0075]

[0076] in, This represents the Fisher information matrix value after adding random components, used to measure the importance of model parameters; γ is the initialization factor, and the random matrix M... sc The data is derived from a uniform distribution Uniform(0,1), where ns and ts represent the current step and the total number of steps, respectively. In this embodiment, each batch of data is set as one step.

[0077] Model parameter selection and update: Based on the Fisher information matrix values ​​after adding random components, the top-k parameters with the highest scores are selected. Then, a binary "Fish Mask Matrix" is generated, where the selected top-k parameters are set to 1, and the rest are set to 0. The sparsity level of the Fisher mask matrix is ​​controlled by the hyperparameter k. In the gradient update step of model training, only the parameters with a mask value of 1 are updated, while all other parameters are frozen.

[0078] from Figure 3 As can be seen, the SFAS module calculates Fisher information for the model parameters during the forward computation and loss backpropagation process. By randomly intervening in the Fisher information, a sparse binary Fisher mask matrix is ​​ultimately generated. This Fisher mask matrix determines which parameters are updated (mask value 1) and which parameters remain frozen (mask value 0) during model training, thereby achieving precise and sparse fine-tuning of the most critical parameters.

[0079] The following section uses a specific downstream task, such as regional precipitation prediction, and selects Aurora, a pre-trained model with 1.3 billion parameters, as the base model to illustrate the optimal implementation process of the task adaptive parameter adjustment method in this embodiment.

[0080] exist Figure 4In the "downsampling" task, when the input is "current global atmospheric state data at 5.625°", the TADP and SFAS modules in the ClimatePEFT framework will generate "dynamic cue 1" to indicate the characteristics of the current task and "Fisher mask matrix 1" to select neuron parameters related to the current task in order to upscale the low-resolution data to 1.40625°.

[0081] Accordingly, in Figure 5 In the "Precipitation Forecasting" task, given a completely different input such as "current 0.25° regional atmospheric state data", the framework can adaptively generate new "Dynamic Cue 2" and "Fisher Mask Matrix 2" to predict future regional precipitation.

[0082] Figure 4 , Figure 5 Unfilled small circles represent selected neuron parameters, while gray filled small circles represent frozen neuron parameters. Existing efficient fine-tuning methods adopt a consistent approach, selecting the same neuron parameters for different tasks, ignoring the diversity of downstream weather tasks. ClimatePEFT's ability to dynamically generate specific cues and parameter masks for different tasks intuitively demonstrates that ClimatePEFT can perform customized fine-tuning according to specific task requirements, rather than adopting a "one-size-fits-all" strategy that may exist in existing methods.

[0083] In this embodiment, the task adaptive parameter adjustment method for the weather and climate basic model includes the following steps:

[0084] S1. Prepare to input weather data and initialize the basic weather and climate model.

[0085] Load the pre-trained weather and climate baseline model Aurora, which includes an encoder, a backbone network, and a decoder. The backbone network can be a 3D Swin Transformer U-Net. Prepare the input weather data X for a specific downstream task (such as predicting the 6-hour cumulative total precipitation of a region from the ERA5-CH dataset). The input weather data for this downstream task is the current weather conditions of a region, including 70 meteorological variables, with a spatial resolution of 0.25°, and the data format is 70×240×240.

[0086] The main network includes the ClimatePEFT model, a task-adaptive parameter-efficient fine-tuning framework of this embodiment.

[0087] S2. The task-adaptive dynamic prompting module of the ClimatePEFT model, which is a parameter-efficient fine-tuning framework, performs task-adaptive dynamic prompting, generates soft prompting words, and fuses the input weather data with the soft prompting words to obtain an enhanced word sequence.

[0088] In this embodiment, the process of adaptive dynamic prompting for tasks includes the following steps:

[0089] S21. Extract the embedding layer weights of the encoder in the weather and climate basic model.

[0090] First, the embedding layer weight matrix is ​​obtained from the model's encoder. This weight matrix encapsulates the model's encoded knowledge of different input variables, spatial resolution, and climate features, representing a highly condensed representation of task characteristics. Assuming a downstream task's input data has V physical variables, the spatial resolution information is reflected in the core size P of the patch embedding. h ×P w (Preferred value is 4×4) and the hidden dimension of the model is D (preferred value is 512).

[0091] S22. Perform internal pattern extraction and external pattern extraction to obtain a tensor containing internal and external pattern information.

[0092] The weight matrix E is rearranged in dimensions and then fed into three cascaded adapters for internal pattern extraction. The V-Adapter captures the physical relationships between different input variables. The feature tensors processed by the adapters undergo layer normalization and self-attention operations to capture the physical variables (totaling V) and spatial resolution (size P). h ×P w External coupling modes between )

[0093] S23. Generate soft prompt words.

[0094] The tensor containing internal and external pattern information is passed through a fully connected linear layer (FC-linear) to generate soft prompt tokens S of a specified length.

[0095] S24. Integrate the input weather data with soft cue words to obtain the enhanced word sequence.

[0096] The input weather data X prepared in the first step is divided into multiple image patches, and then encoded into input tokens T through the embedding layer of the encoder. The soft cue token S and the input token T are concatenated in the sequence dimension to form an enhanced token sequence [S,T].

[0097] S3. Through the stochastic Fisher-guided adaptive selection module of the ClimatePEFT model, a parameter-efficient fine-tuning framework, stochastic Fisher-guided adaptive selection is performed to achieve parameter fine-tuning of the basic weather and climate model.

[0098] In this embodiment, the Fisher-guided adaptive selection process includes the following steps:

[0099] S31. Calculate Fisher information.

[0100] During parameter fine-tuning, for a batch of training data, the model performs one forward propagation and calculates the loss. Based on the calculated loss, the gradients of all trainable parameters θ with respect to the log-likelihood are calculated, and their squares are used as approximations of the Fisher information F. θ Fisher information measures the sensitivity of parameters to the distribution of the model's output; parameters with higher values ​​are more important to the task.

[0101] S32. Introduce annealing random perturbations during model training to correct Fisher information.

[0102] To prevent the incorrect selection of unimportant parameters due to noise during the initial training phase, this embodiment introduces an annealing stochastic stabilization mechanism. Specifically, in the calculated Fisher information F... θ Based on this, a random term that decays linearly with the number of training steps is added to calculate the corrected Fisher information.

[0103] S33. Based on the corrected Fisher information, select the Top-k parameters and generate a mask.

[0104] Based on the corrected Fisher information values, all parameters are sorted, and the top k most important parameters are selected. The hyperparameter k determines the sparsity; for example, it can be set to 0.001, meaning 0.1% of the parameters are selected. A binary "Fish Mask Matrix" with the same shape as the model parameters is generated, with the selected Top-k parameters set to 1 at their corresponding positions and the rest set to 0.

[0105] S34, Masked parameter update.

[0106] During backpropagation to update parameters, the calculated gradients are multiplied element-wise with the generated mask. In this way, only the gradients of the model parameters that are 1 in the mask (i.e., the selected key parameters) are preserved and updated, while the remaining model parameters are frozen.

[0107] Repeat the above steps in the training cycle until the preset number of training cycles (Epochs) is reached, which is 15 cycles in this embodiment.

[0108] Application Performance Verification: To verify the application performance of this invention framework, comprehensive experiments were conducted on three representative and challenging downstream tasks based on a pre-trained base model (Aurora) containing 1.3 billion parameters. These tasks included global data downsampling (from 5.625° to 1.40625° resolution), ensemble weather forecast post-processing, and regional (China) precipitation forecasting. The experimental results strongly demonstrate the effectiveness and efficiency of this framework (ClimatePEFT). With only about 0.28% of the trainable parameters required for fine-tuning (e.g., 3.48M in the downsampling task), the performance of this framework not only comprehensively surpasses several existing state-of-the-art parameter-efficient fine-tuning (PEFT) benchmark methods, but also successfully reaches a level comparable to computationally expensive full-scale fine-tuning methods. Furthermore, it even surpasses full-scale fine-tuning on certain specific task evaluation metrics (such as the Z500 CRPS metric for forecast post-processing). These results highlight the superior capabilities of the present invention framework, which can effectively address the unique challenges commonly found in meteorological data, such as variable heterogeneity, resolution diversity, and spatiotemporal coverage differences, and achieve efficient and adaptive fine-tuning of the parameters of the basic weather and climate model.

[0109] This embodiment also provides a weather and climate forecasting method, including steps S1-S3 of the above-mentioned task adaptive parameter adjustment method for the basic weather and climate model, and the following steps:

[0110] S4. Input the enhanced lexical sequence into the parameter-fine-tuned weather and climate basic model, use the parameter-fine-tuned weather and climate basic model to perform forward inference, and output the forecast results.

[0111] The enhanced lexical sequence from step S2 is fed into the parameter-fine-tuned weather and climate baseline model. This model processes and infers layer by layer through L Transformer modules in the main network. Due to soft cues, the weather and climate baseline model can perceive the current task, such as "regional precipitation forecast," when processing the enhanced lexical sequence, thus performing targeted feature adjustments and calibrations. Finally, the decoder outputs the forecast result.

[0112] Similarly, based on the same inventive concept, this embodiment also provides a weather and climate forecasting system, including the following modules:

[0113] The parameter adjustment module is implemented using steps S1-S3 of the above-mentioned task adaptive parameter adjustment method to perform task adaptive parameter adjustment on the weather and climate basic model and obtain the weather and climate basic model after parameter fine-tuning.

[0114] The prediction module is used to feed the enhanced lexical sequence into the parameter-fine-tuned weather and climate basic model, and then use the parameter-fine-tuned weather and climate basic model to perform layer-by-layer processing and inference to output the forecast results.

[0115] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A task-adaptive parameter-efficient fine-tuning framework model, characterized in that, The Transformer module, applied to the pre-trained main network, is implemented on the pre-trained weather and climate baseline model; the parameter-efficient fine-tuning framework model includes a collaborative task-adaptive dynamic prompting module and a random Fisher-guided adaptive selection module. The task-adaptive dynamic prompting module is connected to the encoder and receives input data through the encoder's embedding layer; the random Fisher-Price-guided adaptive selection module is connected to the decoder. The task-adaptive dynamic prompting module is used to enable the model to perceive and adapt to the data heterogeneity of different downstream tasks. By extracting and transforming the weight information of the embedding layer in the encoder, dynamic soft prompting words are generated and injected into the input of the pre-trained main network to achieve context-aware feature recalibration for downstream tasks. The randomized Fisher-guided adaptive selection module is used to identify and update only the subset of parameters most critical to the current downstream task, while retaining the general knowledge learned in the pre-trained model. It uses Fisher information to quantify the importance of model parameters and determine which model parameters need to be updated and which need to be frozen during model training.

2. The parameter-efficient fine-tuning framework model according to claim 1, characterized in that, The implementation process of the task adaptive dynamic prompting module includes: Obtain the embedding layer weight matrix in the encoder used to map the input weather and climate data to words. The embedding layer weight matrix contains hidden dimensional information that reveals climate features, dimensional information of the input physical variables, and spatial resolution information of the block embedding core size. Reshape the embedding layer weight matrix. An adapter is designed to extract internal patterns. The adapter captures the inherent patterns in different dimensions through upsampling, subsampling, and the GELU activation function. The output matrix is ​​obtained after the internal patterns are extracted. External pattern extraction is performed, and the output matrix extracted from the internal pattern is applied to the layer normalization function and self-attention operation to capture the external pattern that establishes the coupling analysis between the input physical variables and spatial resolution features, so as to establish the coupling relationship between different features. By using a fully connected linear layer, the coupling relationships between different features established by external pattern extraction are converted into soft cue words. The generated soft cue lexicons are appended to the beginning of the input lexicon sequence of each Transformer module in the main network to guide the model to focus on the specific attributes of the current task during forward propagation.

3. The parameter-efficient fine-tuning framework model according to claim 2, characterized in that, When extracting internal patterns, three adapters, HW-Adapter, V-Adapter, and D-Adapter, are designed and connected in sequence. HW-Adapter takes the embedding layer weights as input and processes the dimensions related to spatial resolution. V-Adapter processes the dimensions related to the input physical variables. D-Adapter processes the hidden dimensions related to climate features. After internal pattern extraction, the output matrix processed by D-Adapter is obtained.

4. The parameter-efficient fine-tuning framework model according to claim 1, characterized in that, The implementation process of the Fisher-guided adaptive selection module includes: Importance measurement: The Fisher information matrix is ​​approximated by the squared gradient of the model's loss function on the training data to quantify the importance of model parameters and measure the degree of impact of small changes in model parameters on the model's output distribution. To stabilize the selection of model parameters, randomness is introduced: a random component that decays linearly with the number of training steps is added to the calculated Fisher information matrix to reduce the interference of initial noise on the judgment of the importance of model parameters. Model parameter selection and update: Based on the Fisher information matrix values ​​after adding random components, select the top-k parameters with the highest scores; then, generate a binary Fisher mask matrix, in which the selected top-k parameters are set to 1 and the rest to 0, and the sparsity level of the Fisher mask matrix is ​​controlled by hyperparameters.

5. The parameter-efficient fine-tuning framework model according to claim 4, characterized in that, Fisher information matrix values ​​after adding random components for: Where γ is the initial factor, M sc Let be a random matrix, and ns and ts be the current step and the total number of steps, respectively.

6. A task-adaptive parameter adjustment method for a basic weather and climate model, characterized in that, Includes the following steps: S1. Prepare to input weather data and initialize the weather and climate basic model; wherein, the weather and climate basic model includes an encoder, a main network and a decoder, and the main network includes the parameter-efficient fine-tuning framework model described in any one of claims 1-5; S2. The task-adaptive dynamic prompting module of the parameter-efficient fine-tuning framework model performs task-adaptive dynamic prompting, generates soft prompting words, and merges the input weather data with the soft prompting words to obtain the enhanced word sequence. S3. Through the stochastic Fisher-guided adaptive selection module of the parameter-efficient fine-tuning framework model, stochastic Fisher-guided adaptive selection is performed to achieve parameter fine-tuning of the basic weather and climate model.

7. The task adaptive parameter adjustment method according to claim 6, characterized in that, Step S2, which involves adaptive dynamic prompting for tasks, includes the following steps: S21. Extract the embedding layer weights of the encoder in the basic weather and climate model; S22. Perform internal pattern extraction and external pattern extraction to obtain a tensor containing internal and external pattern information; S23. Pass the tensor containing internal and external pattern information through a fully connected linear layer to generate soft cue words; S24. Integrate the input weather data with soft cue words to obtain the enhanced word sequence.

8. The task adaptive parameter adjustment method according to claim 6, characterized in that, Step S2, which involves the adaptive selection process guided by random Fisher, includes the following steps: S31. Calculate Fisher information: Calculate the loss of the model during one forward propagation; based on the calculated loss, calculate the gradient of all trainable parameters θ of the model with respect to the log-likelihood, and take the square of it as an approximation of Fisher information. S32. Introduce annealing random perturbations during model training to correct Fisher information; S33. Based on the corrected Fisher information, select the Top-k parameters and generate a mask; S34. Masked parameter update: When updating parameters through backpropagation, the calculated gradient is multiplied element-wise with the generated mask.

9. A weather and climate forecasting method, characterized in that, This includes steps S1-S3 of the task adaptive parameter adjustment method according to any one of claims 6-8, and the following steps: S4. The enhanced word sequence is fed into the weather and climate basic model with fine-tuned parameters. The weather and climate basic model with fine-tuned parameters is used for layer-by-layer processing and inference to output the forecast results.

10. A weather and climate forecasting system, characterized in that, Includes the following modules: The parameter adjustment module is implemented by steps S1-S3 of the task adaptive parameter adjustment method described in any one of claims 6-8, and performs task adaptive parameter adjustment on the weather and climate basic model to obtain the weather and climate basic model after parameter fine-tuning. The prediction module is used to feed the enhanced lexical sequence into the parameter-fine-tuned weather and climate basic model, and then use the parameter-fine-tuned weather and climate basic model to perform layer-by-layer processing and inference to output the forecast results.

Images