Adaptive instruction generation driven large model fine-tuning method for meteorological field
By constructing a traffic meteorology seed dataset and using adaptive instruction fine-tuning and low-rank adaptive strategies to fine-tune the pre-trained language model, the problems of resource consumption and dataset creation for large-scale pre-trained language models in the meteorological field are solved, and the performance and efficiency of the model in meteorological tasks are improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- WUHAN UNIV
- Filing Date
- 2024-11-08
- Publication Date
- 2026-06-02
AI Technical Summary
Existing technologies face challenges when applying large-scale pre-trained language models to the meteorological field, including difficulties in creating large-scale instruction datasets, high resource consumption, high computational costs, and deployment difficulties, which affect the performance and efficiency of the models in the meteorological field.
A traffic meteorology seed dataset was constructed. The pre-trained language model was augmented and fine-tuned using an adaptive instruction fine-tuning strategy and a low-rank adaptive strategy to generate an instruction dataset in the meteorological field. The ChatGLM3-6B model was then fine-tuned until the preset performance conditions were met.
It improves the performance of the model in meteorological tasks, solves the problems of difficulty in creating large-scale instruction datasets and high resource consumption, and achieves efficient weather forecasting and information provision.
Smart Images
Figure CN119759510B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of software engineering and artificial intelligence, and in particular to a large model fine-tuning method driven by adaptive instruction generation for the meteorological field. Background Technology
[0002] Large-scale pre-trained language models are advanced artificial intelligence models based on deep learning technology, designed specifically for processing and understanding complex natural language tasks. They are widely used in various fields, including but not limited to machine translation, dialogue systems, sentiment analysis, and question answering systems. Techniques such as the self-attention mechanism within the Transformer architecture make these models particularly adept at capturing long-distance dependencies in text, thereby maintaining high semantic and syntactic accuracy when generating and understanding text. This capability ensures the efficiency and accuracy of the models in dealing with real-world natural language processing challenges.
[0003] When applying large pre-trained language models to specific tasks, these models typically need to be fine-tuned to adapt them to new data distributions and task requirements. Traditional fine-tuning methods involve adding an equal number of trainable parameters to the original model parameters and then further training on the target task data. This approach allows the model to fully utilize the knowledge gained during the pre-training phase, but it also has several significant drawbacks:
[0004] (1) Large-scale instruction datasets are difficult to create: Designing instructions requires creative thinking and in-depth expertise to ensure that accurate and representative instances can be written. In addition, this process involves a deep understanding of the task context, a clear understanding of the task objectives, and the precise construction of input-output pairs that match the task characteristics, which is not only time-consuming but also labor-intensive.
[0005] (2) High resource consumption: Full parameter fine-tuning requires the introduction of the same amount of trainable parameters as the original model, resulting in extremely high demand for computing resources and storage space.
[0006] (3) High computational cost: The need to update all parameters in each training step results in huge computational overhead, which is especially evident when dealing with particularly large models.
[0007] (4) Deployment difficulties: The model after fine-tuning is difficult to dynamically load and update due to the large number of parameters, which makes the deployment and delivery of the model passive and static.
[0008] (5) Poor interpretability: Directly modifying the original parameters may affect the interpretability and controllability of the model behavior, reducing the transparency of the model's prediction results. Summary of the Invention
[0009] This invention provides an adaptive instruction generation-driven large model fine-tuning method for the meteorological field, which solves the problems of difficulty in creating large-scale instruction datasets, high resource consumption, and high computational cost in related technologies. This method can fine-tune large models by using automatically generated meteorological instruction data, thereby improving the performance of the model in meteorological tasks.
[0010] According to one aspect of the present invention, an adaptive instruction generation-driven large model fine-tuning method for the meteorological field is provided, comprising:
[0011] Construct a traffic meteorology seed dataset;
[0012] Based on a preset adaptive instruction fine-tuning strategy, the traffic meteorological seed dataset is augmented to obtain an instruction dataset.
[0013] Based on the preset ChatGLM3-6B model and the instruction dataset, the pre-trained language model is fine-tuned using a preset low-rank adaptive strategy until the fine-tuned pre-trained language model reaches the preset performance conditions, thereby obtaining the target language model, which is used to perform weather forecasting and / or provide weather information that meets preset customized requirements.
[0014] As a further technical solution, based on a preset adaptive instruction fine-tuning strategy, the traffic meteorological seed dataset is augmented to obtain an instruction dataset, including:
[0015] Determine that each instruction data to be generated contains at least one instruction;
[0016] Each instruction and its corresponding input instance are input into a preset model to obtain the output instance corresponding to each instruction, and multiple given instructions are obtained based on each instruction and its corresponding output instance.
[0017] Each given instruction and its task type are determined, and an instance generation strategy corresponding to each given instruction is determined based on its task type. Data augmentation is then performed on the corresponding given instruction based on the instance generation strategy, and the instruction dataset is obtained based on the augmentation results.
[0018] As a further technical solution, the instance generation strategy corresponding to each given instruction is determined based on the task type of each given instruction, including:
[0019] If the task type is a generation task, then the input priority method will be used as the instance generation strategy.
[0020] If the task type is a categorized task, then the output priority method will be used as the instance generation strategy.
[0021] As a further technical solution, based on the preset ChatGLM3-6B model and the instruction dataset, a preset low-rank adaptive strategy is used to fine-tune the pre-trained language model until the fine-tuned pre-trained language model reaches the preset performance conditions, thereby obtaining the target language model, including:
[0022] The instruction dataset is input into the preset ChatGLM3-6B model to obtain the output text vector;
[0023] Calculate the loss between the output text vector and the target text vector, and based on the calculation result, fine-tune the pre-trained language model using the preset low-rank adaptive strategy, and determine whether the fine-tuned pre-trained language model meets the preset performance conditions.
[0024] If the fine-tuned pre-trained language model meets the preset performance conditions, then the fine-tuned pre-trained language model is used as the target language model. Otherwise, the parameters of the preset low-rank adaptive strategy are adjusted, and the step of fine-tuning the pre-trained language model using the preset low-rank adaptive strategy is re-executed until the new fine-tuned pre-trained language model meets the preset performance conditions, thus obtaining the target language model.
[0025] As a further technical solution, before inputting the instruction dataset into the preset ChatGLM3-6B model, the following is also included:
[0026] The instruction dataset is preprocessed to obtain an instruction dataset that meets the preset ChatGLM3-6B model question-and-answer text format.
[0027] As a further technical solution, a traffic meteorological seed dataset is constructed, including:
[0028] Retrieve the identifiers of multiple tasks, the name of each task, the task instructions for each task, the task type for each task, and a list of multiple task instances;
[0029] The traffic meteorological seed dataset is constructed based on the identifiers of the multiple tasks, the name of each task, the task instructions of each task, the task type of each task, and the list of multiple task instances.
[0030] According to one aspect of the present invention, an adaptive instruction generation-driven large model fine-tuning device for the meteorological field is provided, comprising:
[0031] The building block is used to construct the traffic meteorology seed dataset;
[0032] An augmentation module is used to augment the traffic meteorological seed dataset based on a preset adaptive instruction fine-tuning strategy to obtain an instruction dataset.
[0033] The fine-tuning module is used to fine-tune the pre-trained language model based on the preset ChatGLM3-6B model and the instruction dataset using a preset low-rank adaptive strategy until the fine-tuned pre-trained language model reaches the preset performance conditions, thereby obtaining the target language model, which is used to perform weather forecasting and / or provide weather information that meets preset customization requirements.
[0034] As a further technical solution, the augmentation module is also used to execute the following instructions:
[0035] Determine that each instruction data to be generated contains at least one instruction;
[0036] Each instruction and its corresponding input instance are input into a preset model to obtain the output instance corresponding to each instruction, and multiple given instructions are obtained based on each instruction and its corresponding output instance.
[0037] Each given instruction and its task type are determined, and an instance generation strategy corresponding to each given instruction is determined based on its task type. Data augmentation is then performed on the corresponding given instruction based on the instance generation strategy, and the instruction dataset is obtained based on the augmentation results.
[0038] According to one aspect of the present invention, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the large model fine-tuning method driven by adaptive instruction generation for the meteorological field.
[0039] According to one aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which is executed by a processor to implement the steps of the large model fine-tuning method driven by adaptive instruction generation for the meteorological field.
[0040] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0041] This invention first constructs a traffic meteorology seed dataset. Then, based on a preset adaptive instruction fine-tuning strategy, it augments the seed dataset to obtain an instruction dataset. Next, based on a preset ChatGLM3-6B model and the instruction dataset, it fine-tunes a pre-trained language model using a preset low-rank adaptive strategy until the fine-tuned pre-trained language model reaches preset performance conditions, resulting in a target language model. This target language model is then used for weather forecasting and / or to provide meteorological information that meets preset customized requirements. Therefore, this invention solves the problems of difficulty in creating large-scale instruction datasets and high resource consumption associated with related technologies. It enables fine-tuning of large models using automatically generated meteorological instruction data, improving the model's performance in meteorological tasks. Attached Figure Description
[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0043] Figure 1 A flowchart of a large model fine-tuning method driven by adaptive command generation for the meteorological field, provided by an embodiment of the present invention;
[0044] Figure 2 A flowchart illustrating data augmentation based on adaptive instruction fine-tuning according to an embodiment of the present invention;
[0045] Figure 3 This is a flowchart illustrating the fine-tuning of a pre-trained language model based on Low-Rank Adaptation (LoRA) according to an embodiment of the present invention.
[0046] Figure 4 A block diagram of a large model fine-tuning device for adaptive instruction generation driven by meteorological field according to an embodiment of the present invention.
[0047] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation
[0048] Embodiments of the present invention are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.
[0049] The following describes a large-scale model fine-tuning method driven by adaptive command generation for the meteorological domain, according to an embodiment of the present invention, with reference to the accompanying drawings. Addressing the problems mentioned in the background art, such as the difficulty in creating large-scale command datasets and high resource consumption, the present invention provides a large-scale model fine-tuning method driven by adaptive command generation for the meteorological domain. In this method, based on a preset adaptive command fine-tuning strategy, a traffic meteorological seed dataset is augmented to obtain a command dataset. A preset low-rank adaptive strategy is then used to fine-tune a pre-trained language model until the fine-tuned pre-trained language model reaches preset performance conditions, resulting in a target language model for meteorological forecasting and / or providing meteorological information that meets preset customized requirements. This solves the problems of difficulty in creating large-scale command datasets and high resource consumption in related technologies, enabling fine-tuning of large models using automatically generated meteorological command data, thus improving the model's performance in meteorological tasks.
[0050] Specifically, Figure 1 This is a flowchart illustrating a large model fine-tuning method driven by adaptive command generation for the meteorological field, provided in an embodiment of the present invention.
[0051] like Figure 1 As shown, this adaptive command generation-driven large model fine-tuning method for the meteorological field includes the following steps:
[0052] In step S101, a traffic meteorological seed dataset is constructed.
[0053] Specifically, the traffic meteorological seed dataset of this invention needs to integrate multi-dimensional data such as meteorological conditions, geographical information and time changes, and simulate real-world application scenarios to ensure the practicality and effectiveness of the model.
[0054] Optionally, to demonstrate the diverse potential of model fine-tuning, the present invention designs the following types of application scenario tasks:
[0055] (1) Fact judgment task: evaluate the accuracy of the model’s judgment on geographical and climate information.
[0056] (2) Extreme weather selection task: test the model’s ability to select extreme weather phenomena.
[0057] (3) Paragraph search task: Simulate actual meteorological information query, requiring the model to accurately find relevant paragraphs.
[0058] (4) Intent recognition task: predict users’ potential intents through user discourse analysis, reflecting the model’s consulting application capabilities.
[0059] (5) Travel suggestion generation task: Generate practical travel suggestions based on specific weather conditions.
[0060] Furthermore, in some embodiments, constructing a traffic meteorological seed dataset includes: obtaining identifiers of multiple tasks, names of each task, task instructions of each task, task types of each task, and a list of multiple task instances; and constructing the traffic meteorological seed dataset based on the identifiers of multiple tasks, names of each task, task instructions of each task, task types of each task, and the list of multiple task instances.
[0061] Specifically, the original, earliest defined batch of seed data (175 items) is actually text with a fixed field format. The task's unique identifier (id) distinguishes different tasks. The task name briefly describes the task's main content. The task instruction details the task's goals and requirements in natural language. The task instance list (instances) contains input and output pairs, describing the specific execution case of the task. Input refers to the instance's input content, which can be a question, description, query, etc., varying depending on the task type. Output refers to the expected correct output for a specific input, which can be a classification result, generated text, etc. The task type (is_classification) indicates whether the task is a classification task. A boolean value of `is_classification` of `true` indicates a classification task, while a boolean value of `is_classification` of `false` indicates a generation task or other type.
[0062] In step S102, based on a preset adaptive instruction fine-tuning strategy, the traffic meteorological seed dataset is augmented to obtain an instruction dataset.
[0063] Furthermore, in some embodiments, based on a preset adaptive instruction fine-tuning strategy, data augmentation is performed on the traffic meteorological seed dataset to obtain an instruction dataset, including: determining at least one instruction contained in each instruction data to be generated; inputting each instruction and its corresponding input instance into a preset model to obtain an output instance corresponding to each instruction, and obtaining multiple given instructions based on each instruction and its corresponding output instance; determining each given instruction and its task type, and determining an instance generation strategy corresponding to each given instruction based on its task type, and performing data augmentation on the corresponding given instruction based on the instance generation strategy corresponding to each given instruction, and obtaining the instruction dataset based on the augmentation result.
[0064] It should be noted that this invention employs the semi-automatic data augmentation method SELF-INSTRUCT and introduces the SELF-INSTRUCT framework. The SELF-INSTRUCT framework represents a cutting-edge natural language processing technique that aims to improve the model's ability to follow instructions through data generated by the model itself. The core idea of the SELF-INSTRUCT framework is to utilize the existing knowledge base of pre-trained language models (such as the GPT series, BERT series, etc.) and their powerful generative capabilities to bootstrap and generate new instructions and their corresponding input-output samples. This method mainly includes two parts: defining instruction data and automatically generating instruction data.
[0065] First, let's define instruction data. The instruction data to be generated by this invention contains a set of instructions. Each instruction defines a set of tasks using natural language. Task t have Input / output examples:
[0066]
[0067] in, For the mission, This refers to the sequence number of the input / output instance. For the i-th instance of task t, The target output for the i-th instance of task t. For the task The number of input and output instances.
[0068] Given task instructions and corresponding inputs, the model is expected to produce the following output:
[0069]
[0070] in, Generate output for the model. For the mission, This refers to the index of the input / output instance. For the instructions of task t, For the i-th instance of task t, The model prediction output for the i-th instance of task t.
[0071] for In many cases, there are no strict boundaries between instructions and instance inputs. For example, "Write an article about school safety" can be a valid instruction that the model responds to directly, while it can also be formulated as "Write an article on the following topic" as the instruction and "school safety" as the instance input. To encourage diversity in data formats, this invention allows for instruction cases without additional input (i.e., X is empty).
[0072] Secondly, this invention introduces the automatic generation of instruction data. The invention employs the semi-automatic data augmentation method SELF-INSTRUCT to automatically generate instruction data. The SELF-INSTRUCT framework leverages the powerful generative capabilities of the pre-trained language model to construct a self-guided task generation and self-training pipeline. This innovative method allows the pre-trained language model to expand the scope and depth of its task execution through self-exploration and generation, thereby enhancing its ability to follow and respond to human instructions through a refined instruction adjustment process.
[0073] The following is combined with Figure 2 This paper introduces the process of data augmentation (i.e. automatic generation of instruction data) based on adaptive instruction fine-tuning.
[0074] like Figure 2 As shown, firstly, a series of new task instructions are generated from the large language model.
[0075] Specifically, SELF-INSTRUCT generates new instructions in a guided manner from a small set of human-written seed instructions. This invention uses a task pool containing 175 tasks, each consisting of one instruction and one instance. At each step, the model extracts eight task instructions from the task pool as contextual example references; of these eight instructions, six are from human-written tasks and two are from tasks generated by the model in previous steps, promoting task diversity.
[0076] Secondly, distinguish whether the task is a categorized task.
[0077] Specifically, this step involves classifying the task. This invention requires two different methods to handle classified and unclassified tasks. Next, it determines whether the generated instruction represents a classified task. Using the 12 classified instructions and 19 unclassified instructions already in the seed task, the language model is guided with minimal prompts to determine whether the newly generated instruction belongs to a classified task.
[0078] Furthermore, for each generated instruction, the framework further generates corresponding input-output instances.
[0079] Specifically, after receiving a given instruction and its task type, an instance is generated independently for each instruction. For generative tasks, an input-first approach can be used, requiring the language model to first propose the input fields based on the instruction and then generate the corresponding output. This generation order is similar to how the model responds to instructions and inputs, but here contextual examples from other tasks are used. For classification tasks, an output-first approach is used, that is, possible class labels are generated first, and then inputs are generated based on the conditions of each class label.
[0080] Finally, a series of heuristic strategies are used to filter the generated samples, removing invalid or highly similar instances to existing samples.
[0081] Specifically, to ensure the diversity and quality of the generated data, this invention sets several filtering conditions. For example, a new instruction is only added to the task pool if its similarity (measured by ROUGE-L) to an existing instruction is less than 0.7. Furthermore, since certain keywords are generally not processed by language models, this invention also excludes instructions containing certain specific keywords (e.g., image, picture, graph). Simultaneously, when generating new instances for each instruction, completely identical instances or instances with the same input but different outputs are filtered out. Invalid generation is identified and filtered out based on heuristics (e.g., instructions that are too long or too short, or instances whose outputs duplicate the input), thereby ensuring that the generated tasks and instances are of high quality and usefulness.
[0082] Furthermore, in some embodiments, the instance generation strategy is determined according to the multi-task type of each instruction, including: if the task type is a generation task, then the input priority method is used as the instance generation strategy; if the task type is a classification task, then the output priority method is used as the instance generation strategy.
[0083] Specifically, different generation strategies are adopted based on the task type of the instruction. For generation tasks, an input-first approach is used, that is, the input fields are determined first, and then the corresponding outputs are generated. For classification tasks, an output-first approach is used, that is, the possible classification labels are determined first, and then the corresponding inputs are generated based on these labels.
[0084] In step S103, based on the preset ChatGLM3-6B model and instruction dataset, the pre-trained language model is fine-tuned using a preset low-rank adaptive strategy until the fine-tuned pre-trained language model reaches the preset performance conditions, thereby obtaining the target language model, which is used to perform weather forecasting and / or provide weather information that meets preset customized requirements.
[0085] It should be noted that this invention uses ChatGLM3-6B as the base model for the instruction fine-tuning stage. ChatGLM3-6B is a large-scale dialogue language model developed by Zhipu AI. It is based on the Transformer architecture and has 6.2 billion parameters. ChatGLM3-6B uses similar technology to ChatGPT and is optimized for Chinese question-and-answer and dialogue. The core idea of instruction fine-tuning is to construct a large-scale dataset containing various task instructions, with each sample formatted as "instruction + input" and "target output". Based on this instruction dataset, the pre-trained model is fine-tuned using supervised learning, with the goal of enabling the model to correctly map different task instructions to the desired output behavior. The fine-tuning process has three steps: data preprocessing, LoRA module deployment, and model training.
[0086] Specifically, the large language model used in this invention is based on ChatGLM3-6B, and is trained on a pre-constructed meteorological question-and-answer dataset using a fine-tuning approach tailored to the data characteristics and problem requirements of the meteorological and transportation domains. Through this process, the model learns relevant knowledge and question-and-answer styles in the meteorological and transportation domains, better understands human commands, and improves question-and-answer performance. The LoRA method is used for model training, further reducing training overhead. While adapting to the meteorological and transportation domains, this method also preserves as much of the original language model's dialogue capabilities and knowledge reserves as possible. Through fine-tuning training with domain-specific data, the generative language model can provide more tailored services to vertical domains.
[0087] The following is combined with Figure 3 This paper introduces the process of fine-tuning a pre-trained language model based on low-rank adaptive LoRA.
[0088] like Figure 3 As shown, firstly, the original instruction data is preprocessed before training and converted into the unified question-and-answer text format of ChatGLM3.
[0089] Next, the preprocessed text is input into ChatGLM3 to obtain the output text vector.
[0090] Finally, the loss between the output text vector and the target text vector is calculated, and the LoRA technique is used to fine-tune the pre-trained language model.
[0091] Further, in some embodiments, based on a preset ChatGLM3-6B model and instruction dataset, a preset low-rank adaptive strategy is used to fine-tune the pre-trained language model until the fine-tuned pre-trained language model reaches the preset performance conditions to obtain the target language model. This includes: inputting the instruction dataset into the preset ChatGLM3-6B model to obtain output text vectors; calculating the loss between the output text vectors and the target text vectors, and based on the calculation results, fine-tuning the pre-trained language model using the preset low-rank adaptive strategy, and determining whether the fine-tuned pre-trained language model reaches the preset performance conditions; if the fine-tuned pre-trained language model reaches the preset performance conditions, then the fine-tuned pre-trained language model is used as the target language model; otherwise, the parameters of the preset low-rank adaptive strategy are adjusted, and the step of fine-tuning the pre-trained language model using the preset low-rank adaptive strategy is re-executed until the new fine-tuned pre-trained language model reaches the preset performance conditions to obtain the target language model.
[0092] In some embodiments, before inputting the instruction dataset into the preset ChatGLM3-6B model, the method further includes: preprocessing the instruction dataset to obtain an instruction dataset that meets the preset ChatGLM3-6B model question-and-answer text format.
[0093] It should be noted that the instruction-based question-and-answer dataset constructed by the adaptive instruction fine-tuning data augmentation module is stored in JSONL file format, with each line of data stored in JSON data format. It includes three parts: instruction, input, and target output. Before training, the original instruction data needs to be preprocessed, converted into the unified question-and-answer text format of ChatGLM3, and then converted into an input format acceptable to the language model: input tensor (input_ids), attention mask (attn_mask), and labels.
[0094] Specifically, in the data preprocessing stage, data is first read line by line from the original data file. Each line is treated as a Python dictionary object containing key fields such as instructions, inputs, and target outputs. Next, the context is constructed from preset prompt text, the special marker "[gMASK]sop", user prompts, and the concatenation of instructions and inputs. The target is constructed from assistant input prompts and target outputs. The model's tokenizer converts these texts into token sequences, adding eos terminators to the end of each sequence. Sequences exceeding the preset maximum length are appropriately truncated or discarded. Then, the start and end positions of the target within the complete token sequence are calculated, and input tensors (input_ids), attention masks (attn_mask), and labels are constructed. Labels for non-target parts are marked with a special value of -100 to indicate that these parts are not included in the loss calculation. Finally, to ensure consistent input lengths for training samples in the same batch, the three tensors are appropriately padded. This entire preprocessing process not only structures the data to meet model requirements but also optimizes the efficiency and effectiveness of model training.
[0095] Furthermore, the preprocessed text is input into the ChatGLM3 model. First, the generated input tensors (input_ids), attention masks (attn_mask), and labels are provided to the model. ChatGLM3 processes these inputs through its internal multi-layer Transformer structure, using attention mechanisms and neural network components for feature extraction and learning. After processing, the model outputs a text vector.
[0096] Furthermore, the loss between the output text vector generated by the ChatGLM3 model and the predefined target text vector is calculated, typically using loss functions such as cross-entropy loss or mean squared error, to quantify the error between the model output and the actual expected output. Next, the LoRA technique is used to fine-tune the pre-trained language model.
[0097] During fine-tuning, LoRA parameters such as scaling factor, dropout rate, and rank are set to suit specific needs. Specifically, this invention uses Huggingface's peft package to load the LoRA module into the model. In actual training, the default settings for the LoRA scaling factor (alpha) are 32, the dropout rate is 0.1, and the LoRA rank is 8. LoRA fine-tuning is specified for the query / key / value mapping matrix of the Transformer model, but the bias term is not fine-tuned; only the weight matrix is fine-tuned.
[0098] Furthermore, by backpropagation and using optimizers such as AdamW to update network weights, the entire process is iterated over multiple training cycles until the fine-tuned pre-trained language model reaches the preset performance conditions, thus obtaining the target language model.
[0099] For example, this invention uses a multi-GPU parallelization approach, employing the Accelerate package to perform data-parallel training on four Nvidia V100 32GB graphics cards. AdamW is used as the training optimizer, and to improve training stability and reduce the risk of gradient explosion, this invention does not apply weight decay regularization to the bias and layer normalization weight parameters.
[0100] According to embodiments of the present invention, an adaptive instruction generation-driven large model fine-tuning method for the meteorological domain is proposed. Based on a preset adaptive instruction fine-tuning strategy, a traffic meteorological seed dataset is augmented to obtain an instruction dataset. A preset low-rank adaptive strategy is then used to fine-tune a pre-trained language model until the fine-tuned pre-trained language model reaches preset performance conditions, resulting in a target language model for meteorological forecasting and / or providing meteorological information that meets preset customized requirements. This solves the problems of difficulty in creating large-scale instruction datasets and high resource consumption associated with related technologies. It enables fine-tuning of large models using automatically generated meteorological domain instruction data, improving the model's performance in meteorological tasks.
[0101] Next, referring to the accompanying drawings, we describe the large model fine-tuning device for adaptive command generation driven by the present invention, which is designed for the meteorological field.
[0102] Figure 4 This is a block diagram of a large model fine-tuning device driven by adaptive instruction generation for the meteorological field, according to an embodiment of the present invention.
[0103] like Figure 4 As shown, the large model fine-tuning device 10 driven by adaptive command generation for the meteorological field includes: a construction module 100, an augmentation module 200, and a fine-tuning module 300.
[0104] The system includes a construction module 100 for constructing a traffic meteorology seed dataset; an augmentation module 200 for augmenting the traffic meteorology seed dataset based on a preset adaptive instruction fine-tuning strategy to obtain an instruction dataset; and a fine-tuning module 300 for fine-tuning a pre-trained language model based on a preset ChatGLM3-6B model and instruction dataset using a preset low-rank adaptive strategy until the fine-tuned pre-trained language model reaches preset performance conditions to obtain a target language model, which is used for meteorological prediction and / or to provide meteorological information that meets preset customized requirements.
[0105] Furthermore, in some embodiments, the augmentation module 200 is configured to: determine at least one instruction contained in each instruction data to be generated; input each instruction and its corresponding input instance into a preset model to obtain an output instance corresponding to each instruction, and obtain multiple given instructions based on each instruction and its corresponding output instance; determine each given instruction and its task type, and determine an instance generation strategy corresponding to each given instruction based on its task type, and perform data augmentation on the corresponding given instruction based on the instance generation strategy corresponding to each given instruction, and obtain an instruction dataset based on the augmentation result.
[0106] Furthermore, in some embodiments, the augmentation module 200 is used to: if the task type is a generation task, use the input-first method as the instance generation strategy; if the task type is a classification task, use the output-first method as the instance generation strategy.
[0107] Further, in some embodiments, the fine-tuning module 300 is used to: input the instruction dataset into a preset ChatGLM3-6B model to obtain output text vectors; calculate the loss between the output text vectors and the target text vectors, and based on the calculation results, fine-tune the pre-trained language model using a preset low-rank adaptive strategy, and determine whether the fine-tuned pre-trained language model meets the preset performance conditions; if the fine-tuned pre-trained language model meets the preset performance conditions, then the fine-tuned pre-trained language model is used as the target language model; otherwise, the parameters of the preset low-rank adaptive strategy are adjusted, and the step of fine-tuning the pre-trained language model using the preset low-rank adaptive strategy is re-executed until the new fine-tuned pre-trained language model meets the preset performance conditions, thereby obtaining the target language model.
[0108] Furthermore, in some embodiments, before inputting the instruction dataset into the preset ChatGLM3-6B model, the fine-tuning module 300 is also used to: preprocess the instruction dataset to obtain an instruction dataset that meets the preset ChatGLM3-6B model question-and-answer text format.
[0109] Furthermore, in some embodiments, the construction module 100 is configured to: obtain identifiers of multiple tasks, names of each task, task instructions of each task, task types of each task, and a list of multiple task instances; and construct a traffic meteorological seed dataset based on the identifiers of multiple tasks, names of each task, task instructions of each task, task types of each task, and the list of multiple task instances.
[0110] It should be noted that the explanation of the aforementioned embodiment of the large model fine-tuning method driven by adaptive command generation for the meteorological field also applies to the large model fine-tuning device driven by adaptive command generation for the meteorological field in this embodiment, and will not be repeated here.
[0111] According to an embodiment of the present invention, an adaptive instruction generation-driven large model fine-tuning device for the meteorological field, based on a preset adaptive instruction fine-tuning strategy, augments a traffic meteorological seed dataset to obtain an instruction dataset. It then uses a preset low-rank adaptive strategy to fine-tune a pre-trained language model until the fine-tuned pre-trained language model reaches preset performance conditions, thus obtaining a target language model for meteorological forecasting and / or providing meteorological information that meets preset customized requirements. This solves the problems of difficulty in creating large-scale instruction datasets and high resource consumption in related technologies, enabling fine-tuning of large models using automatically generated meteorological instruction data, thereby improving the model's performance in meteorological tasks.
[0112] Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. The electronic device may include:
[0113] The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.
[0114] When processor 502 executes the program, it implements the large model fine-tuning method for adaptive instruction generation driven by meteorology provided in the above embodiments.
[0115] Furthermore, electronic devices also include:
[0116] Communication interface 503 is used for communication between memory 501 and processor 502.
[0117] The memory 501 is used to store computer programs that can run on the processor 502.
[0118] Memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.
[0119] If the memory 501, processor 502, and communication interface 503 are implemented independently, then the communication interface 503, memory 501, and processor 502 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0120] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.
[0121] Processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.
[0122] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described adaptive instruction generation-driven large model fine-tuning method for the meteorological field.
[0123] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0124] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.
[0125] Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.
[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.
Claims
1. A large-scale model fine-tuning method driven by adaptive instruction generation for the meteorological field, characterized in that: include: Construct a traffic meteorology seed dataset; Based on a preset adaptive instruction fine-tuning strategy, the traffic meteorological seed dataset is augmented to obtain an instruction dataset. Based on the preset ChatGLM3-6B model and the instruction dataset, a preset low-rank adaptive strategy is used to fine-tune the pre-trained language model until the fine-tuned pre-trained language model reaches the preset performance conditions, thereby obtaining the target language model. This target language model is then used for weather forecasting and / or to provide weather information that meets preset customized requirements. The feature is that, based on the preset adaptive instruction fine-tuning strategy, data augmentation is performed on the traffic weather seed dataset to obtain the instruction dataset. This includes: determining at least one instruction contained in each instruction data to be generated; inputting each instruction and its corresponding input instance into a preset model to obtain the corresponding output instance, and obtaining multiple given instructions based on each instruction and its corresponding output instance; determining each given instruction and its task type, and determining the instance generation strategy corresponding to each given instruction based on its task type, and performing data augmentation on the corresponding given instruction based on the instance generation strategy, and obtaining the instruction dataset based on the augmentation results.
2. The large model fine-tuning method driven by adaptive instruction generation for the meteorological field according to claim 1, characterized in that, Determining the instance generation strategy corresponding to each given instruction based on the task type of each given instruction includes: if the task type is a generation task, then the input priority method is used as the instance generation strategy; if the task type is a classification task, then the output priority method is used as the instance generation strategy.
3. The large model fine-tuning method driven by adaptive instruction generation for the meteorological field according to claim 1, characterized in that, Based on a preset ChatGLM3-6B model and the instruction dataset, a preset low-rank adaptive strategy is used to fine-tune the pre-trained language model until the fine-tuned pre-trained language model reaches the preset performance conditions to obtain the target language model. This includes: inputting the instruction dataset into the preset ChatGLM3-6B model to obtain output text vectors; calculating the loss between the output text vectors and the target text vectors, and based on the calculation result, fine-tuning the pre-trained language model using the preset low-rank adaptive strategy, and determining whether the fine-tuned pre-trained language model reaches the preset performance conditions; if the fine-tuned pre-trained language model reaches the preset performance conditions, then the fine-tuned pre-trained language model is used as the target language model; otherwise, the parameters of the preset low-rank adaptive strategy are adjusted, and the step of fine-tuning the pre-trained language model using the preset low-rank adaptive strategy is repeated until the new fine-tuned pre-trained language model reaches the preset performance conditions to obtain the target language model.
4. The large model fine-tuning method driven by adaptive instruction generation for the meteorological field according to claim 3, characterized in that, Before inputting the instruction dataset into the preset ChatGLM3-6B model, the method further includes: preprocessing the instruction dataset to obtain an instruction dataset that meets the preset ChatGLM3-6B model question-and-answer text format.
5. The large model fine-tuning method driven by adaptive instruction generation for the meteorological field according to claim 1, characterized in that, Constructing a traffic meteorological seed dataset includes: obtaining identifiers for multiple tasks, the name of each task, the task instructions for each task, the task type for each task, and a list of multiple task instances; and constructing the traffic meteorological seed dataset based on the identifiers for the multiple tasks, the name of each task, the task instructions for each task, the task type for each task, and the list of multiple task instances.
6. A large model fine-tuning device driven by adaptive command generation for the meteorological field, characterized in that, include: The building block is used to construct the traffic meteorology seed dataset; An augmentation module is used to augment the traffic meteorological seed dataset based on a preset adaptive instruction fine-tuning strategy to obtain an instruction dataset. The fine-tuning module is used to fine-tune the pre-trained language model based on the preset ChatGLM3-6B model and the instruction dataset using a preset low-rank adaptive strategy until the fine-tuned pre-trained language model reaches the preset performance conditions, thereby obtaining the target language model. This target language model is used for weather forecasting and / or to provide weather information that meets preset customized requirements. The augmentation module is also used to execute the following instructions: determine at least one instruction contained in each instruction data to be generated; input each instruction and its corresponding input instance into the preset model to obtain the corresponding output instance, and obtain multiple given instructions based on each instruction and its corresponding output instance; determine each given instruction and its task type, determine the instance generation strategy corresponding to each given instruction based on the task type, and perform data augmentation on the corresponding given instruction based on the instance generation strategy, and obtain the instruction dataset based on the augmentation results.
7. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, the processor executing the program to implement the steps of the large model fine-tuning method for adaptive instruction generation driven by any one of claims 1-5 in the meteorological field.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the steps of the large model fine-tuning method driven by adaptive instruction generation for the meteorological field as described in any one of claims 1-5.