Training method and device for big language model in climate field

Abstract

The invention provides a method and a device for training a big language model in the climate field. The method comprises the following steps: acquiring interdisciplinary original data of meteorological changes; establishing a comprehensive climatic change interdisciplinary data system according to the meteorological change interdisciplinary original data; then, utilizing the comprehensive climatic change interdisciplinary data system, the general code data, the general mathematical problem data and the general encyclopedia data to further pre-train a base large language model to obtain a large language model after professional subject knowledge is injected; and finally, performing fine tuning on parameters in the big language model after the professional subject knowledge is injected by adopting a supervised fine tuning method to obtain a climate field big language model. Wherein after the climate field big language model receives the climate field related problem of the user, the complex climate change problem can be effectively replied.

Description

Technical Field

[0001] The present application relates to the technical field of climate change, and in particular to a method and device for training a large language model in the climate field. Background Art

[0002] Against the backdrop of global warming, extreme weather and climate events are increasing. At present, various scientific models and socio-economic models, represented by meteorological models, earth system models and integrated assessment models (IAMs), tend to focus on scientific analysis in a single professional field, and their main research issues are mainly concentrated in the field of climate mitigation. They pay particularly little attention to the field of climate adaptation, and are unable to respond to the policy needs, technical needs and international governance needs faced by complex climate change issues. Summary of the invention

[0003] In view of this, the present application provides a method and device for training a large language model in the climate field. The large language model in the climate field obtained through training can effectively respond to complex climate change issues.

[0004] The first aspect of the present application provides a method for training a large language model in the climate field, comprising:

[0005] Obtain interdisciplinary raw data on meteorological changes;

[0006] Based on the interdisciplinary raw data of meteorological changes, establish a comprehensive interdisciplinary data system of climate change;

[0007] Further pre-training the base large language model using the comprehensive climate change interdisciplinary data system, general code data, general mathematical problem data and general encyclopedia data to obtain a large language model infused with professional subject knowledge;

[0008] A supervised fine-tuning method is used to fine-tune the parameters of the large language model after injecting professional subject knowledge to obtain a large language model in the climate field; wherein, after receiving questions from users related to the climate field, the large language model in the climate field outputs corresponding answers.

[0009] Optionally, the obtaining of interdisciplinary raw data on meteorological changes includes:

[0010] Obtaining material data in the field of climate change by crawling; wherein the material data includes video data, audio data and document data;

[0011] Performing speech recognition on the video data and the audio data to obtain a speech recognition result;

[0012] Performing document parsing on the document data to obtain a document parsing result;

[0013] The speech recognition results and document parsing results are used as interdisciplinary raw data of meteorological changes.

[0014] Optionally, the establishment of a comprehensive interdisciplinary climate change data system based on the interdisciplinary raw data of meteorological changes includes:

[0015] Performing quality identification on the interdisciplinary raw data of meteorological changes to obtain interdisciplinary seed data of meteorological changes;

[0016] Inputting the interdisciplinary seed data of meteorological changes into a refined model, and outputting a climate field dataset;

[0017] The data in the climate field dataset are numerically expressed to obtain a comprehensive interdisciplinary data system for climate change.

[0018] Optionally, the use of the comprehensive climate change interdisciplinary data system, general code data, general mathematical problem data and general encyclopedia data to further pre-train the base large language model to obtain a large language model injected with professional subject knowledge includes:

[0019] Combine the comprehensive climate change interdisciplinary data system, general code data, general mathematical problem data and general encyclopedia data to obtain further pre-training data;

[0020] The large language model is further pre-trained using the further pre-training data and the training framework of the large language model to obtain a large language model that is infused with professional subject knowledge.

[0021] Optionally, the method of using supervised fine-tuning to fine-tune the parameters of the large language model after injecting professional subject knowledge to obtain a large language model in the climate field includes:

[0022] Acquire a pre-constructed instruction data set; wherein the pre-constructed instruction data set includes at least one instruction data sample; the instruction data sample includes an instruction and a response; the instruction includes n tags; the response includes k tags; n and k are positive integers;

[0023] In the process of supervised fine-tuning, the instructions and corresponding log-likelihoods given all the instruction data samples are maximized to regressively train the large language model injected with professional subject knowledge to obtain a large language model in the climate field.

[0024] The second aspect of the present application provides a training device for a large language model in the climate field, comprising:

[0025] The first acquisition unit is used to obtain interdisciplinary raw data on meteorological changes;

[0026] Establishing a unit for establishing a comprehensive interdisciplinary data system of climate change based on the interdisciplinary raw data of meteorological changes;

[0027] A pre-training unit, used to further pre-train the base large language model using the comprehensive climate change interdisciplinary data system, general code data, general mathematical problem data and general encyclopedia data, to obtain a large language model injected with professional subject knowledge;

[0028] The fine-tuning unit is used to fine-tune the parameters of the large language model after injecting professional subject knowledge by using a supervised fine-tuning method to obtain a large language model in the climate field; wherein, after receiving the user's questions related to the climate field, the large language model in the climate field outputs corresponding answers.

[0029] Optionally, the first acquiring unit includes:

[0030] A crawling unit, used for obtaining material data in the field of climate change by crawling; wherein the material data includes video data, audio data and document data;

[0031] A speech recognition unit, used to perform speech recognition on the video data and the audio data to obtain a speech recognition result;

[0032] A document parsing unit, used to parse the document data to obtain a document parsing result;

[0033] The determination unit is used to use the speech recognition result and the document parsing result as interdisciplinary raw data of meteorological changes.

[0034] Optionally, the establishing unit includes:

[0035] A quality identification unit, used for performing quality identification on the interdisciplinary raw data of meteorological changes to obtain interdisciplinary seed data of meteorological changes;

[0036] An input unit, used for inputting the interdisciplinary seed data of meteorological changes into the refined model, and outputting a climate field data set;

[0037] The numerical expression unit is used to numerically express the data in the climate field data set to obtain a comprehensive interdisciplinary data system for climate change.

[0038] Optionally, the pre-training unit includes:

[0039] A data matching unit is used to match the comprehensive climate change interdisciplinary data system, general code data, general mathematical problem data and general encyclopedia data to obtain further pre-training data;

[0040] The pre-training sub-unit is used to further pre-train the large language model using the further pre-training data and the training framework of the large language model to obtain the large language model after infusing professional subject knowledge.

[0041] Optionally, the fine-tuning unit includes:

[0042] A second acquisition unit is used to acquire a pre-constructed instruction data set; wherein the pre-constructed instruction data set includes at least one instruction data sample; the instruction data sample includes an instruction and a response; the instruction includes n tags; the response includes k tags; n and k are positive integers;

[0043] The fine-tuning subunit is used to maximize the instructions and corresponding log-likelihood given all the instruction data samples during the supervised fine-tuning process to regressively train the large language model injected with professional subject knowledge to obtain a large language model in the climate field.

[0044] A third aspect of the present application provides an electronic device, including:

[0045] one or more processors;

[0046] a storage device having one or more programs stored thereon;

[0047] When the one or more programs are executed by the one or more processors, the one or more processors implement the training method of the large language model in the climate field as described in any one of the first aspects.

[0048] A fourth aspect of the present application provides a computer storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for training a large language model in the climate field as described in any one of the first aspects is implemented.

[0049] It can be seen from the above scheme that the present application provides a training method and device for a large language model in the field of climate, which obtains interdisciplinary raw data on meteorological changes; establishes a comprehensive interdisciplinary data system for climate change based on the interdisciplinary raw data on meteorological changes; then, further pre-trains the base large language model using the comprehensive interdisciplinary data system for climate change, general code data, general mathematical problem data, and general encyclopedia data to obtain a large language model injected with professional subject knowledge; finally, fine-tunes the parameters in the large language model injected with professional subject knowledge using a supervised fine-tuning method to obtain a large language model in the field of climate; wherein, after receiving questions from users about the climate field, the large language model in the field of climate can effectively respond to complex climate change issues. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0051] Figure 1 A specific flow chart of a method for training a large language model in the climate field provided in an embodiment of the present application;

[0052] Figure 2 A specific flow chart of a method for obtaining interdisciplinary raw data of meteorological changes provided in another embodiment of the present application;

[0053] Figure 3 A specific flow chart of a method for establishing a comprehensive interdisciplinary climate change data system provided in another embodiment of the present application;

[0054] Figure 4 A specific flow chart of a method for further pre-training a large language model of a base provided in another embodiment of the present application;

[0055] Figure 5 A specific flow chart of a method for fine-tuning parameters in a large language model after injecting professional subject knowledge, provided in another embodiment of the present application;

[0056] Figure 6 A schematic diagram of supervised fine-tuning provided in another embodiment of the present application;

[0057] Figure 7 A schematic diagram of a training device for a large language model in the climate field provided in another embodiment of the present application. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0059] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0060] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0061] It should be noted that the concepts such as "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0062] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0063] The present application embodiment provides a method for training a large language model in the climate field, such as Figure 1 As shown, the specific steps include:

[0064] S101. Obtain interdisciplinary raw data on meteorological changes.

[0065] Optionally, in another embodiment of the present application, an implementation of step S101 is as follows: Figure 2 As shown, including:

[0066] S201. Obtain material data in the field of climate change by crawling.

[0067] The material data includes video data, audio data and document data. The document data includes but is not limited to web page texts, professional documents, etc. related to the field of climate change, which are not limited here.

[0068] S202: Perform speech recognition on the video data and the audio data to obtain a speech recognition result.

[0069] In the specific implementation process of the present application, the speech recognition method may adopt but is not limited to the automatic speech recognition (Automatic Speech Recognition, ASR) technology, which is not limited here.

[0070] S203: Perform document parsing on the document data to obtain a document parsing result.

[0071] In the specific implementation process of the present application, the document parsing method may adopt but is not limited to optical character recognition (OCR) technology, which is not limited here.

[0072] S204. Using the speech recognition results and the document parsing results as interdisciplinary raw data of meteorological changes.

[0073] S102. Establish a comprehensive interdisciplinary data system on climate change based on interdisciplinary raw data on meteorological changes.

[0074] Optionally, in another embodiment of the present application, an implementation of step S102 is as follows: Figure 3 As shown, including:

[0075] S301. Perform quality identification on interdisciplinary original data on meteorological changes to obtain interdisciplinary seed data on meteorological changes.

[0076] In the specific implementation process of this application, before performing quality identification on interdisciplinary raw data on meteorological changes, multiple rounds of data deduplication, data cleaning and other operations can be performed to reduce the workload of subsequent quality identification.

[0077] In the specific implementation process of this application, the quality identification of interdisciplinary raw data on meteorological change includes but is not limited to selecting all selected data for climate expert data, such as IPCC reports, without screening. For mixed expert data, such as books, papers, and patents, a keyword screening model screening method is used. Specifically, for books, only climate-related books are extracted as forecasts; for patents, relevant patent numbers are extracted according to the "China Green and Low-Carbon Patent Classification System" and used as the basis for screening; for papers, due to the large amount of data, the first round of preliminary screening is first carried out based on the relevance of the title to the climate, and then a small classifier is trained using the preliminary screened articles to classify all data to screen the required forecasts. For data crawled from the web. By analyzing the credibility and climate similarity of the website, the data can be screened to see if it can be used as corpus. The screened data is used as interdisciplinary seed data for meteorological change.

[0078] S302, inputting interdisciplinary seed data of meteorological changes into the refined model, and outputting a climate field dataset.

[0079] In the specific implementation process of this application, the refinement model can be but not limited to the FASTtext model, which is not limited here. The FASText model is an open source fast text classification and representation learning tool developed by Facebook. It combines the most successful concepts in natural language processing and machine learning, such as the bag-of-words model, n-gram and subword information, and shares information between categories through hidden representation, providing a simple and efficient text classification and representation learning method.

[0080] Specifically, a part of the interdisciplinary seed data on meteorological changes can be used as training sample data to train the FASTtext model to obtain a refined model, and then the refined model obtained by training can be used to refine the full amount of interdisciplinary seed data on meteorological changes, thereby obtaining a climate field data set that is helpful to the climate field and easy for large language models to learn, thereby achieving climate field enhancement of the data.

[0081] S303. Numerically express the data in the climate field dataset to obtain a comprehensive interdisciplinary data system on climate change.

[0082] In the specific implementation process of this application, the data in the climate field dataset can be numerically expressed through, but not limited to, text extraction, data vectorization, feature extraction, etc., and finally a comprehensive climate change interdisciplinary data system including agronomy, forestry, energy science, water conservancy engineering, environmental science, management, economics, political science, law and other related disciplines is formed.

[0083] S103. Use the comprehensive climate change interdisciplinary data system, general code data, general mathematical problem data and general encyclopedia data to further pre-train the base large language model to obtain a large language model that is infused with professional subject knowledge.

[0084] The base large language model can be but is not limited to INF-34B, which is not limited here. INF-34B has powerful language understanding and generation capabilities. It contains 34 billion parameters and can capture complex language features. Its training data comes from a selected 3.5T English-Chinese and bilingual corpus. In addition to its outstanding performance in the evaluation of public datasets, it also shows great potential in the scientific field.

[0085] At the same time, this application also uses the same word segmenter as INF-34B. That is, in the pre-word segmentation process: for numbers, follow the GPT series to divide them into 1-3 digits, with each word segment having a maximum of three digits and a minimum of one digit; for English, copy the first 65,536 word segments directly from the vocabulary of GPT4 and keep them in the language of interest: for Chinese, a word segmenter with a vocabulary of 30,000 trained on a 25GB Chinese corpus is used to extract Chinese word segments; for other symbols, common single-code characters are added from network data. The resulting word segmenter has a total of 96,536 word segments.

[0086] Optionally, in the specific implementation process of the present application, an implementation method of step S103 is as follows: Figure 4 As shown, including:

[0087] S401. Match the comprehensive climate change interdisciplinary data system, general code data, general mathematical problem data and general encyclopedia data to obtain further pre-training data.

[0088] Among them, general math problem data includes OpenWebMath, algebraic stack, artificially constructed math instruction data, etc., which are not limited here; general code data includes code data crawled from GitHub, artificially constructed code instruction data, etc., which are not limited here; general encyclopedia data includes Wikipedia, fineweb edu, etc., which are not limited here. In terms of climate data, professional data is mainly used and network data is supplemented. Professional data includes IPCC reports, papers, related patents and books, etc., which are not limited here. Network data includes climate-related data crawled from various websites, such as climate cc, which are not limited here. As shown in Table 1, a data ratio is provided for an embodiment of the present application.

[0089]

[0090]

[0091] Table 1

[0092] S402. Further pre-train the large language model using further pre-training data and the training framework of the large language model to obtain a large language model that is infused with professional subject knowledge.

[0093] In the specific implementation process of this application, the training framework of the base large language model INF-34B is similar to Megatron-lm. In the process of further pre-training INF-34B, in addition to adopting the basic strategies: tensor parallelism, pipeline parallelism, and using Zero's shared data parallelism, the gradients and parameters are overlapped at the end and the first step of gradient accumulation. The uneven bucket partitioning of parameters is also implemented, further improving the overlap ratio between communication and calculation. The rotation embedding matrix is ​​pre-calculated, and the key-value query operation of the rotation matrix is ​​written to further improve the training speed. The training cluster consists of 8 NVIDIA HGX H800 nodes, each of which consists of 8 GPUs with 80G video memory. The initial training learning rate is 1e-5, and it is gradually reduced to 1e-6. The tensor parallelism is set to 4, the pipeline parallelism is set to 8, and the context length is limited to 1024 to ensure the stability of the training. The training consumed a total of 25 billion tokens of data and was completed after 144 hours of training.

[0094] S104. Use a supervised fine-tuning method to fine-tune the parameters of the large language model after injecting professional subject knowledge to obtain a large language model in the climate field.

[0095] Among them, the climate field large language model outputs corresponding answers after receiving users' questions about the climate field.

[0096] It should be noted that supervised fine-tuning (SFT) refers to the use of data in the form of natural language to fine-tune the parameters of a large language model that has been injected with professional subject knowledge. The goal is to further train the large language model using artificially constructed instruction data, thereby enhancing the ability of the large language model in a specific field. Supervised fine-tuning can avoid training the model from scratch, greatly reducing the demand for computing resources, and is one of the most important methods to improve downstream task performance and domain-specific adaptation. It is also an important means to enhance the large model's instruction compliance and task solving capabilities, as well as cross-language migration capabilities.

[0097] Among them, data in natural language form, that is, text data in the form of questions and answers. For example: {"Please give me a brief introduction to the relevant knowledge of NLP.", "Natural Language Processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language."}, {"What are the main factors affecting climate change?", "Gas changes, ozone changes, radiation changes, etc."}, artificially constructed instruction data includes but is not limited to dolly, alpaca, firefly, etc., which are not limited here.

[0098] In the specific implementation process of this application, a full-parameter fine-tuning method is adopted. The principle of full-parameter fine-tuning is similar to model pre-training. The difference is that all parameters already have a good initial value, that is, less data is used to continue training the model based on the initial value to update the parameters.

[0099] Optionally, in another embodiment of the present application, an implementation of step S104 is as follows: Figure 5 As shown, including:

[0100] S501: Obtain a pre-built instruction data set.

[0101] The pre-constructed instruction data set includes at least one instruction data sample; the instruction data sample includes an instruction and a response; the instruction includes n tags; the response includes k tags; n and k are positive integers.

[0102] In the specific implementation process of this application, the pre-built instruction data set was pre-annotated by the transfer, including but not limited to designing more than 90,000 original question-answer pairs on the theme of climate change, and then asking experts to evaluate each question-answer pair 1) whether the question is a valuable climate question and score it with a score of 1-5; 2) whether the answer satisfies the question and the answer, and the completeness of the answer is scored with a score of 1-5. Based on the expert scoring, the question-answer pairs with a score of more than 4 points for both the question and the answer were selected as the pre-built instruction data set.

[0103] S502. During the supervised fine-tuning process, the instructions and corresponding log-likelihoods of all given instruction data samples are maximized to regressively train the large language model injected with professional subject knowledge, thereby obtaining a large language model in the climate field.

[0104] Specifically, for a given pre-built instruction dataset D containing M samples SFT , for each sample in the pre-built instruction dataset, it should contain instruction x m and the response y m , where each instruction and response contains n and k tokens, respectively. In general, the SFT process is performed by maximizing the x given all instruction samples. m Time m The log-likelihood of comes from regressively training a large language model p. This process can be expressed as:

[0105]

[0106] In the specific implementation process of this application, the learning rate during supervised fine-tuning is 1e-5, the batch size is 32, the weight decay is 0.1, and the number of training rounds is 1. The supervised fine-tuning schematic diagram can be shown as follows Figure 6 As shown, this application uses the SFT process to further improve the capabilities of the trained model in the field of climate change, and significantly enhances the command and task solving capabilities in the field of climate change.

[0107] It is understandable that after the model is trained, the model can also be evaluated. The evaluation methods used in this application include two types: evaluating the benchmark data set and evaluating the trained model.

[0108] When evaluating the benchmark dataset, C-eval, Pira 2.0, CDP-QA, etc. can be used for evaluation, but not limited to. C-eval can be used for questions from relevant subjects such as the Chinese college entrance examination and university final exams, and the correct answer needs to be selected from four answers given a question. Pira 2.0 can be constructed by compiling scientific abstracts and United Nations reports on oceans and climate change. This benchmark requires selecting the correct answer from five candidate answers given a question. The candidate answers are carefully designed to have great vocabulary similarities with the reference text and are very close to the correct answer. This intentional design increases the complexity of the task and requires a deeper understanding of the problem. CDP-QA is a dataset constructed through a questionnaire for a carbon emission project. It is a question-answering task in the field of environmental information, covering the disclosure of environmental information by enterprises, cities, governments, etc. The goal is to predict whether a given answer meets the corresponding question.

[0109] When evaluating the model, a unified prompt word template was used to evaluate the performance of the model of this application and the following baseline models on these three evaluation benchmarks: qwen2-7b, llama2-7b, and inf-34b-base.

[0110] In particular, in order to allow the base model to output the final results in the same format, the same five examples are provided during all model reasoning. The final experimental results are shown in Table 2:

[0111]

[0112]

[0113] Table 2

[0114] It can be seen from the above scheme that the present application provides a training method for a large language model in the field of climate, which is achieved by obtaining interdisciplinary raw data on meteorological changes; establishing a comprehensive interdisciplinary data system for climate change based on the interdisciplinary raw data on meteorological changes; then, further pre-training the base large language model using the comprehensive interdisciplinary data system for climate change, general code data, general mathematical problem data, and general encyclopedia data to obtain a large language model injected with professional subject knowledge; finally, fine-tuning the parameters in the large language model injected with professional subject knowledge using a supervised fine-tuning method to obtain a large language model in the field of climate; wherein, after receiving questions from users about the climate field, the large language model in the field of climate can effectively respond to complex climate change issues.

[0115] The present application embodiment provides a training device for a large language model in the climate field, such as Figure 7 As shown, specifically including:

[0116] The first acquisition unit 701 is used to acquire interdisciplinary raw data of meteorological changes.

[0117] Optionally, in another embodiment of the present application, an implementation of the first acquisition unit 701 includes:

[0118] The crawling unit is used to obtain material data in the field of climate change through crawling.

[0119] The material data includes video data, audio data and document data.

[0120] The speech recognition unit is used to perform speech recognition on the video data and the audio data to obtain a speech recognition result.

[0121] The document parsing unit is used to parse the document data to obtain a document parsing result.

[0122] A determination unit is used to use the speech recognition results and document parsing results as interdisciplinary raw data of meteorological changes.

[0123] For the specific working process of the units disclosed in the above embodiments of the present application, please refer to the corresponding method embodiments, such as Figure 2 As shown, no further description is given here.

[0124] The establishment unit 702 is used to establish a comprehensive interdisciplinary data system of climate change based on interdisciplinary raw data of meteorological changes.

[0125] Optionally, in another embodiment of the present application, an implementation of establishing unit 702 includes:

[0126] The quality identification unit is used to perform quality identification on interdisciplinary raw data of meteorological changes to obtain interdisciplinary seed data of meteorological changes.

[0127] The input unit is used to input interdisciplinary seed data of meteorological changes into the refined model, and the output is the climate field dataset.

[0128] The numerical expression unit is used to numerically express the data in the climate field dataset to obtain a comprehensive interdisciplinary data system on climate change.

[0129] For the specific working process of the units disclosed in the above embodiments of the present application, please refer to the corresponding method embodiments, such as Figure 3 As shown, no further description is given here.

[0130] The pre-training unit 703 is used to further pre-train the base large language model using the comprehensive climate change interdisciplinary data system, general code data, general mathematical problem data and general encyclopedia data to obtain a large language model injected with professional subject knowledge.

[0131] Optionally, in another embodiment of the present application, an implementation of the pre-training unit 703 includes:

[0132] The data matching unit is used to match the comprehensive climate change interdisciplinary data system, general code data, general mathematical problem data and general encyclopedia data to obtain further pre-training data.

[0133] The pre-training sub-unit is used to further pre-train the large language model using further pre-training data and the training framework of the large language model to obtain a large language model that is infused with professional subject knowledge.

[0134] For the specific working process of the units disclosed in the above embodiments of the present application, please refer to the corresponding method embodiments, such as Figure 4 As shown, no further description is given here.

[0135] The fine-tuning unit 704 is used to fine-tune the parameters of the large language model after the professional subject knowledge is injected using a supervised fine-tuning method to obtain a large language model in the climate field.

[0136] Among them, the climate field large language model outputs corresponding answers after receiving users' questions about the climate field.

[0137] For the specific working process of the units disclosed in the above embodiments of the present application, please refer to the corresponding method embodiments, such as Figure 1 As shown, no further description is given here.

[0138] Optionally, in another embodiment of the present application, an implementation of the fine-tuning unit 704 includes:

[0139] The second acquisition unit is used to acquire a pre-built instruction data set.

[0140] The pre-constructed instruction data set includes at least one instruction data sample; the instruction data sample includes an instruction and a response; the instruction includes n tags; the response includes k tags; n and k are positive integers.

[0141] The fine-tuning subunit is used to maximize the instructions and corresponding log-likelihood of all given instruction data samples during the supervised fine-tuning process to regressively train the large language model injected with professional subject knowledge, so as to obtain the large language model in the climate field.

[0142] For the specific working process of the units disclosed in the above embodiments of the present application, please refer to the corresponding method embodiments, such as Figure 5 As shown, no further description is given here.

[0143] It can be seen from the above scheme that the present application provides a training device for a large language model in the field of climate, which is achieved by obtaining interdisciplinary original data of meteorological changes in a first acquisition unit 701; the establishment unit 702 establishes a comprehensive interdisciplinary data system for climate change based on the interdisciplinary original data of meteorological changes; then, the pre-training unit 703 further pre-trains the base large language model using the comprehensive interdisciplinary data system for climate change, general code data, general mathematical problem data, and general encyclopedia data to obtain a large language model after injecting professional subject knowledge; finally, the fine-tuning unit 704 uses a supervised fine-tuning method to fine-tune the parameters in the large language model after injecting professional subject knowledge to obtain a large language model in the field of climate; wherein, after receiving questions from users about the climate field, the large language model in the field of climate can effectively respond to complex climate change issues.

[0144] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0145] Another embodiment of the present application provides an electronic device, including:

[0146] One or more processors.

[0147] A storage device having one or more programs stored thereon.

[0148] When the one or more programs are executed by the one or more processors, the one or more processors implement the training method of the large language model in the climate field as described in any one of the above embodiments.

[0149] Another embodiment of the present application provides a computer storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for training a large language model in the climate field as described in any one of the above embodiments is implemented.

[0150] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0151] It should be noted that the computer-readable medium mentioned above in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0152] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0153] Another embodiment of the present application provides a computer program product, which, when executed, is used to execute any of the above-mentioned methods for training a large language model in the climate field.

[0154] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, the above-mentioned functions defined in the method of the embodiment of the present application are executed.

[0155] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in this application is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing this application.

[0156] Although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present application. Certain features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination.

[0157] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of application involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above application concept. For example, the above features are replaced with (but not limited to) technical features with similar functions applied in the present application.

Claims

1. A method for training a large language model in the climate field, characterized in that: include: Obtain interdisciplinary raw data on meteorological changes; Based on the interdisciplinary raw data of meteorological changes, establish a comprehensive interdisciplinary data system of climate change; Further pre-training the base large language model using the comprehensive climate change interdisciplinary data system, general code data, general mathematical problem data and general encyclopedia data to obtain a large language model infused with professional subject knowledge; A supervised fine-tuning method is used to fine-tune the parameters of the large language model after injecting professional subject knowledge to obtain a large language model in the climate field; wherein, after receiving questions from users related to the climate field, the large language model in the climate field outputs corresponding answers.

2. The method for training a large language model in the climate field according to claim 1, characterized in that: The acquisition of interdisciplinary raw data on meteorological changes includes: Obtaining material data in the field of climate change by crawling; wherein the material data includes video data, audio data and document data; Performing speech recognition on the video data and the audio data to obtain a speech recognition result; Performing document parsing on the document data to obtain a document parsing result; The speech recognition results and document parsing results are used as interdisciplinary raw data of meteorological changes.

3. The method for training a large language model in the climate field according to claim 1, characterized in that: The above-mentioned interdisciplinary data system of climate change is established based on the interdisciplinary raw data of meteorological changes, including: Performing quality identification on the interdisciplinary raw data of meteorological changes to obtain interdisciplinary seed data of meteorological changes; Inputting the interdisciplinary seed data of meteorological changes into a refined model, and outputting a climate field dataset; The data in the climate field dataset are numerically expressed to obtain a comprehensive interdisciplinary data system for climate change.

4. The method for training a large language model in the climate field according to claim 1, characterized in that: The method further pre-trains the base large language model using the comprehensive climate change interdisciplinary data system, general code data, general mathematical problem data and general encyclopedia data to obtain a large language model injected with professional subject knowledge, including: Combine the comprehensive climate change interdisciplinary data system, general code data, general mathematical problem data and general encyclopedia data to obtain further pre-training data; The large language model is further pre-trained using the further pre-training data and the training framework of the large language model to obtain a large language model that is infused with professional subject knowledge.

5. The method for training a large language model in the climate field according to claim 1, characterized in that: The supervised fine-tuning method is used to fine-tune the parameters of the large language model after the professional subject knowledge is injected to obtain the climate field large language model, including: Acquire a pre-constructed instruction data set; wherein the pre-constructed instruction data set includes at least one instruction data sample; the instruction data sample includes an instruction and a response; the instruction includes n tags; the response includes k tags; n and k are positive integers; In the process of supervised fine-tuning, the instructions and corresponding log-likelihoods given all the instruction data samples are maximized to regressively train the large language model injected with professional subject knowledge to obtain a large language model in the climate field.

6. A training device for a large language model in the climate field, characterized in that: include: The first acquisition unit is used to obtain interdisciplinary raw data on meteorological changes; Establishing a unit for establishing a comprehensive interdisciplinary data system of climate change based on the interdisciplinary raw data of meteorological changes; A pre-training unit, used to further pre-train the base large language model using the comprehensive climate change interdisciplinary data system, general code data, general mathematical problem data and general encyclopedia data, to obtain a large language model injected with professional subject knowledge; The fine-tuning unit is used to fine-tune the parameters of the large language model after injecting professional subject knowledge by using a supervised fine-tuning method to obtain a large language model in the climate field; wherein, after receiving the user's questions related to the climate field, the large language model in the climate field outputs corresponding answers.

7. The training device for a large language model in the climate field according to claim 6, characterized in that: The first acquisition unit includes: A crawling unit, used for obtaining material data in the field of climate change by crawling; wherein the material data includes video data, audio data and document data; A speech recognition unit, used to perform speech recognition on the video data and the audio data to obtain a speech recognition result; A document parsing unit, used to parse the document data to obtain a document parsing result; The determination unit is used to use the speech recognition result and the document parsing result as interdisciplinary raw data of meteorological changes.

8. The training device for a large language model in the climate field according to claim 6, characterized in that: The establishing unit comprises: A quality identification unit, used for performing quality identification on the interdisciplinary raw data of meteorological changes to obtain interdisciplinary seed data of meteorological changes; An input unit, used for inputting the interdisciplinary seed data of meteorological changes into the refined model, and outputting a climate field data set; The numerical expression unit is used to numerically express the data in the climate field data set to obtain a comprehensive interdisciplinary data system for climate change.

9. The training device for a large language model in the climate field according to claim 6, characterized in that: The pre-training unit comprises: A data matching unit is used to match the comprehensive climate change interdisciplinary data system, general code data, general mathematical problem data and general encyclopedia data to obtain further pre-training data; The pre-training sub-unit is used to further pre-train the large language model using the further pre-training data and the training framework of the large language model to obtain the large language model after infusing professional subject knowledge.

10. The training device for a large language model in the climate field according to claim 6, characterized in that: The fine-tuning unit comprises: A second acquisition unit is used to acquire a pre-constructed instruction data set; wherein the pre-constructed instruction data set includes at least one instruction data sample; the instruction data sample includes an instruction and a response; the instruction includes n tags; the response includes k tags; n and k are positive integers; The fine-tuning subunit is used to maximize the instructions and corresponding log-likelihood given all the instruction data samples during the supervised fine-tuning process to regressively train the large language model injected with professional subject knowledge to obtain a large language model in the climate field.

Images