A geoscience question and answer large model construction method, system, device and medium based on enhanced low-rank adaptation

Abstract

A geoscience language question and answer large model construction method, system, device and medium based on enhanced low-rank adaptation, the method comprising: collecting and preprocessing geoscience data to obtain a geoscience dataset; classifying the geoscience dataset to obtain a geoscience text dataset and a geoscience question and answer dataset; inputting the geoscience text dataset into a corresponding large language model for training to generate a geoscience large language model; inputting the geoscience question and answer dataset into the geoscience large language model for fine-tuning, setting an enhanced low-rank adaptation technology, decomposing multiple low-rank matrices and giving different learning rates to optimize parameter adjustment, and finally adding a DeepSpeed acceleration framework to generate a geoscience language question and answer large model; score evaluating the geoscience language question and answer large model generated by training and continuously optimizing to obtain a weight file with the best score; the system, device and medium are used to implement the method; the present application greatly reduces the resources required for calculation, improves the accuracy and generalization ability of the model.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing technology, specifically relating to a method, system, device, and medium for constructing a large-scale geoscientific language question-answering model based on enhanced low-rank adaptive modeling. Background Technology

[0002] Earth science, through the observation of natural phenomena, discovers the natural laws inherent in the observational data, thereby studying the various processes, changes, and interactions of the Earth system at multiple scales. With the development of earth science, research is exhibiting new characteristics of interdisciplinary, cross-regional, and collaborative approaches. Earth science involves numerous disciplines such as atmosphere, ecology, hydrology, soil, oceanography, and geology, and is also closely related to physics, chemistry, and information science. Earth science big data possesses the "4Vs" common to traditional big data: large volume, diverse types, high velocity, and low value density. Effectively utilizing this earth science knowledge can help deepen our understanding of the Earth's critical zones, global environmental change, interactions between Earth's spheres, and the impacts of human activities.

[0003] In today's information-saturated era, while traditional search engines and web query methods can provide a wealth of information, they are often inefficient because the results they return typically contain a large amount of irrelevant or redundant data. Users need to invest significant time and effort in manual screening and filtering to extract valuable information. This process is not only time-consuming and labor-intensive but also easily limited by individual experience and knowledge, leading to incomplete or biased information selection. Furthermore, although mainstream question-answering systems such as ChatGPT and Kimi can quickly generate answers using artificial intelligence technology, the accuracy and reliability of these answers are not always guaranteed. These systems may have limitations in understanding complex problems, handling technical terms, or providing in-depth analysis, especially in specialized fields such as geoscience research, where these limitations may be more pronounced. Therefore, when dealing with geoscience-related issues, the use of specialized geoscience models is particularly urgent and important.

[0004] Patent application CN118193708A discloses a mineral knowledge question-answering method and system based on a large language model. This method uses web crawling technology to collect mineral data from professional platforms, which is then cleaned and divided into knowledge documents and question-answer pairs. The XGBoost algorithm is used to select the most relevant documents to assist the large language model in generating answers. Simultaneously, LoRA technology is used to fine-tune the model, and prompts are designed to guide the generation of professional mineral content, thereby improving the professionalism of the mineral question-answering system's answers. However, this method has some limitations: First, it mainly targets the subfield of mineral knowledge within geographical science, limiting its application scope. Second, the XGBoost algorithm suffers from high time and space complexity, which not only increases the resource consumption required to generate answers but also results in less than ideal performance when processing high-dimensional sparse data, further restricting its potential for wider application.

[0005] Patent application CN117892711A discloses a method for obtaining text relevance based on a large model. This method uses transformer technology to construct an LLPM (Limited Language Processing) probabilistic model, collects relevant training knowledge points, and transmits them to the large model after encoding and word segmentation. After training, the large model generates a model with probability calculation outputs. By calculating the relevance between new knowledge points and existing knowledge points, the accuracy of text relevance calculation is improved. However, this method has some limitations: First, it mainly targets the specific area of ​​large model training and probability calculation, and its application scope is relatively limited. Second, the transformer technology and LLPM large model used require a large amount of computing resources and storage space, increasing system complexity and resource consumption, further restricting its potential for application in a wider range of fields.

[0006] Patent application CN118171732A discloses a method and apparatus for extracting super-relational knowledge based on a fine-tuned large-scale model. This method acquires text data, constructs super-relational knowledge extraction instructions based on preset fine-tuning parameters, and utilizes the fine-tuned large-scale language model to build a super-relational knowledge extraction model. During training, the model employs a learning rate warm-up and decay strategy, using a training dataset to train it, ultimately obtaining a trained super-relational knowledge extraction model. This model can effectively extract super-relational knowledge and obtain relational data from the text to be parsed. By designing targeted instructions, entities and their related multi-dimensional relations can be extracted from the model output. However, this method has some limitations: First, it is primarily designed for the specific task of super-relational knowledge extraction, limiting its application scope. While it performs well in super-relational knowledge extraction, its applicability and effectiveness may be limited in other types of knowledge extraction tasks or natural language processing tasks. Summary of the Invention

[0007] To overcome the shortcomings of the existing technology, the present invention aims to provide a method, system, device, and medium for constructing a large-scale geoscientific language question-answering model based on enhanced low-rank adaptive technology. By employing specific web crawling techniques, it achieves efficient collection of geoscientific knowledge data, enabling the acquisition of large amounts of accurate geoscientific data, improving the model's professionalism, and allowing it to flexibly adapt to various questions and provide the most accurate answers. Through the use of enhanced low-rank adaptive technology, the generated large-scale geoscientific language model can be fine-tuned using a small number of parameters, resulting in a professional large-scale geoscientific language question-answering model. The present invention significantly reduces the computational resources required while improving the model's accuracy and generalization ability.

[0008] To achieve the above objectives, the technical solution of the present invention includes:

[0009] A method for constructing a large-scale geoscientific language question-answering model based on enhanced low-rank adaptive methods includes:

[0010] Step 1: Collect geoscientific data, including but not limited to China Geographic Network, geoscientific books, and geoscientific research papers;

[0011] Step 2: Preprocess the geoscientific data collected in Step 1 to obtain a geoscientific dataset;

[0012] Step 3: Classify the geoscience dataset obtained in Step 2 to obtain a geoscience text dataset and a geoscience question-and-answer dataset;

[0013] Step 4: Input the geoscience text dataset obtained in Step 3 into the corresponding large language model and perform self-supervised training through federated learning. Utilizing the isomorphic architecture of federated learning, the large language model is trained in a distributed manner on multiple devices. Each device independently processes its local data. After all training epochs are completed, each device sends the parameters of the large language model back to the central server. The central server aggregates the parameters of the large language model to generate the Geographic Sciences Large Language Model (GSLLM).

[0014] Step 5: Input the geoscience question-answering dataset obtained in Step 3 into the Geographic Sciences Large Language Model (GSLLM) obtained in Step 4 for fine-tuning, and then set up an enhanced low-rank adaptive technique, that is, by decomposing multiple low-rank matrices and assigning different learning rates to optimize parameter adjustment. Finally, add the DeepSpeed ​​acceleration framework to the Geographic Sciences Large Language Model (GSLLM) to generate a large geoscience language question-answering model.

[0015] Step 6: Evaluate the large-scale geoscientific language question-answering model generated in Step 5, use the scores to continuously optimize the large-scale geoscientific language question-answering model, and save the weight file of the best score that the large-scale geoscientific language question-answering model can achieve in the evaluation.

[0016] The specific method for step 2 is as follows:

[0017] Step 2.1: Filter and screen the geoscientific data collected in Step 1, and use the features of geoscientific knowledge as keywords to assist the crawling technology in crawling relevant local knowledge;

[0018] Step 2.2 utilizes multithreading and asynchronous processing methods to enable the crawling of multiple targets as described in Step 2.1.

[0019] Step 2.3: Using data cleaning and format conversion tools, remove the parts of the data obtained in Step 2.2 that are not related to geoscience, and format the data obtained in Step 2.2.

[0020] Step 2.4 involves reviewing and correcting errors in the data obtained in Step 2.3, updating outdated knowledge points, and conducting multiple data verifications and comparative analyses to obtain the geoscience dataset.

[0021] The specific method for step 3 is as follows:

[0022] Step 3.1: Classify the geoscience dataset obtained in Step 2. Convert the geoscience data collected from China Geographic Network and geoscience-related scientific research papers in the geoscience dataset obtained in Step 2 into the format of the geoscience text dataset used for pre-training, and obtain the geoscience text dataset.

[0023] Step 3.2 involves converting and fine-tuning the format of the geoscience question-and-answer dataset used in the geoscience dataset obtained in Step 2 from geoscience professional books, to obtain the geoscience question-and-answer dataset.

[0024] The large language model in step 4 uses a Transformer Decoder-only network architecture. The geoscientific text dataset obtained in step 3 is input into the large language model, and word embeddings are added. Word embeddings convert words or tokens in the text into fixed-dimensional vector representations. After word embeddings, Rotated Position Embedding (RoPE) is used to provide positional information for each word in the sequence without adding additional parameters. The large language model employs Global Query Attention (GQA) and the SwiGLU activation function. Root Mean Square Layer Normalization (RMSNorm) is used to normalize the root mean square of each sample in the input tensor of the large language model. The weighted average update formula is as follows:

[0025]

[0026] Among them, w t+1 This represents the overall weight update result at time t+1, where K indicates that there are a total of K parts, and n k Let represent the number of samples in the k-th part, and n represent the total number of samples. This represents the weight update result of the k-th part at time t+1.

[0027] The large language model in step 4 specifically includes:

[0028] Step 4.1: Set up the root mean square layer normalization (RMSNorm) in the large language model structure. Use the root mean square layer normalization (RMSNorm) in the large language model structure to input the geoscientific text dataset generated in Step 3. Calculate the root mean square value (RMS) of all elements for each input sample, and then normalize it. The formula for the root mean square value (RMS) is as follows:

[0029]

[0030] Where, x i Let N represent the value of the i-th data point, and N represent the total number of data points.

[0031] Step 4.2, set the SwiGLU activation function. The formula for the SwiGLU activation function is as follows:

[0032]

[0033] Where x represents the input vector, W represents the weight matrix, and b represents the bias term;

[0034] Step 4.3: Set the position embedding in the Transformer using Rotation Position Embedding (RoPE). This position embedding method encodes the absolute position using a rotation matrix. First, initialize the rotation angle, create the rotation matrix, and then perform the rotation matrix operation. The angle calculation and rotation vector formula for Rotation Position Embedding (RoPE) are as follows:

[0035] Angle calculation:

[0036] Rotation vector:

[0037] Where, θ i The rotation angle represents the i-th dimension and is used to define the rotation matrix; k is a hyperparameter used to control the magnitude of the rotation angle; d represents the dimension of the vector.

[0038] Step 4.4: Set up Global Query Attention (GQA). First, project the input sequence into query, key, and value vectors. Then, reduce the number of query vectors to one or a few. Depending on your model, you can start with a small number of query vectors and gradually increase them, observing the changes in model performance until you find a suitable balance. Then, use the shared query vectors to perform a dot product with all key vectors to obtain the attention score. Finally, use the attention score to perform a weighted sum of the value vectors to obtain the output vector.

[0039] The specific process of fine-tuning the Geographic Sciences Large Language Model (GSLLM) in step 5 is as follows:

[0040] Step 5.1: During fine-tuning, the weights of the Geographic Sciences Large Language Model (GSLLM) are decomposed into multiple low-rank matrices using a low-rank matrix decomposition. The original weight matrix W is decomposed into low-rank matrices A and B, i.e., W≈A*B. Multiple low-rank matrices A and B are combined to form a hierarchical structure, with each layer having a different learning rate (lr-1, lr-2, ..., lr-n). The learning rate of all low-rank matrices is set, and an adaptive optimization algorithm is used to dynamically adjust the learning rate. The step size is updated by adaptively adjusting the parameters based on the feedback during the training process.

[0041] Step 5.2: During the training process, the Geographic Sciences Large Language Model (GSLLM) is accelerated by DeepSpeed, ultimately generating a large-scale geospatial language question-answering model based on enhanced low-rank adaptation.

[0042] The specific method for step 6 is as follows:

[0043] Step 6.1: Evaluate the geoscientific language question-answering model generated in Step 5 using four evaluation methods: BLEU4, ROUGE-1, ROUGE-2, and ROUGE-L. The formulas for BLEU4, ROUGE-1, ROUGE-2, and ROUGE-L are as follows:

[0044]

[0045] Where exp represents the natural exponential function, w represents the weight, and BP (Brevity Penalty) is the penalty for overly short translations, calculated as follows:

[0046]

[0047] Where c represents the number of n-grams in the candidate translation (machine translation); r represents the number of n-grams in the reference translation;

[0048]

[0049] Where max means to take the maximum value, count means to count the number of words, gram means an n-gram, that is, a sequence of n consecutive words in the text, Candidate means machine-generated text, Reference means manually generated text or known high-quality text, ROUGE-1 calculates the overlap of a single word, and ROUGE-2 calculates the overlap of two consecutive words.

[0050]

[0051] Step 6.2: Based on the evaluation score obtained in 6.1, change the hyperparameters when inputting the geoscience language question answering model, continuously optimize the weight file of the training model, and finally retain the weight file with the best score achieved.

[0052] This invention also provides a large-scale geoscientific language question-answering model construction system based on enhanced low-rank adaptive modeling, comprising:

[0053] The geoscience data collection module is used to collect geoscience data, including but not limited to the China Geographic website, geoscience books, and geoscience-related research papers.

[0054] The geoscientific data preprocessing module is used to preprocess the collected geoscientific data to obtain geoscientific datasets;

[0055] The geoscience dataset classification module is used to classify geoscience datasets to obtain geoscience text datasets and geoscience question-and-answer datasets;

[0056] The Geographic Sciences Large Language Model (GSLLM) generation module is used to input geoscientific text datasets into the corresponding large language models. Self-supervised training is performed through federated learning. Utilizing the isomorphic architecture of federated learning, the large language models are trained in a distributed manner on multiple devices. Each device independently processes its local data. After all training epochs are completed, each device sends the parameters of the large language model back to the central server. The central server aggregates the parameters of the large language model to generate the Geographic Sciences Large Language Model (GSLLM).

[0057] The module for generating a large-scale geoscience language question-answering model is used to input geoscience question-answering datasets into the Geographic Sciences Large Language Model (GSLLM) for fine-tuning. Then, an enhanced low-rank adaptive technique is set up, which optimizes parameter adjustment by decomposing multiple low-rank matrices and assigning different learning rates. Finally, the DeepSpeed ​​acceleration framework is added to the Geographic Sciences Large Language Model (GSLLM) to generate a large-scale geoscience language question-answering model.

[0058] The optimal weight file acquisition module is used to evaluate the scores of the trained geoscientific and linguistic question-answering model, continuously optimize the model using the scores, and save the weight file of the best score that the geoscientific and linguistic question-answering model can achieve in the evaluation.

[0059] This invention also provides a device for constructing a large geoscientific language question-answering model based on enhanced low-rank adaptive methods, comprising:

[0060] Memory: A computer-readable device that stores the computer program of the above-mentioned method for constructing a large geoscientific language question-answering model based on enhanced low-rank adaptive Memory;

[0061] Processor: Used to implement the method for constructing a large geoscientific language question-answering model based on enhanced low-rank adaptive methods when executing the computer program.

[0062] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, enables the implementation of the aforementioned method for constructing a large geoscientific language question-answering model based on enhanced low-rank adaptive methods.

[0063] Compared with the prior art, the present invention has the following advantages:

[0064] First, a dedicated dataset was created specifically for the field of geographical science, employing a method that combines automated error correction with manual screening. Specific steps included data cleaning, format conversion, and expert review. In step 2, specific web crawling techniques were used to efficiently collect geoscientific knowledge data, enabling the acquisition of a large amount of accurate geoscientific data and improving the model's professionalism. During data processing, the noise and inconsistencies of the large amounts of data were overcome, ensuring the accuracy and reliability of the data.

[0065] Second, addressing the complexity of the geosciences domain, step 4 employs federated learning and distributed training across multiple nodes to generate a large-scale geosciences language question-answering model with 8 billion parameters. This massive number of parameters enables the model to more accurately capture the grammar, semantics, and contextual information of the language, thereby generating more natural and coherent text. Furthermore, the large-scale geosciences language question-answering model exhibits greater robustness in the face of noisy and inconsistent data. Even with incomplete or noisy data, it can still generate reasonable outputs while considering a broader context, thus ensuring the coherence and consistency of the text.

[0066] Third, for the field of geographical science, step 4 generates the Geographic Sciences Large Language Model (GSLLM), which fills the gap in the current pre-trained large geoscience models. Based on this, the downstream tasks are further refined, and a large number of question-and-answer geoscience datasets are used to generate a more professional geoscience question-and-answer large model, ensuring its accuracy and efficiency in solving various complex geospatial problems, while realizing rapid data parsing and deep knowledge mining.

[0067] Fourth, considering the large number of parameters in the Geographic Sciences Large Language Model (GSLLM) which may lead to excessive resource consumption, step 5 proposes an enhanced low-rank adaptive technique. This technique captures the main features of the data by decomposing the original high-dimensional data matrix into a product of low-rank matrices, and sets a low-rank matrix that is adaptively adjusted according to the structure of the Geographic Sciences Large Language Model (GSLLM). This significantly reduces the computational resources required while improving the accuracy and generalization ability of the Geographic Sciences Large Language Model (GSLLM).

[0068] In summary, by creating professional geoscientific text datasets and geoscientific question-answering datasets, this invention improves the accuracy and reliability of the geoscientific language question-answering model in geosciences, while considering a broader context to ensure text coherence and consistency. Furthermore, by utilizing an enhanced low-rank adaptive method, this invention can achieve training results comparable to full training while significantly reducing the computational resources required. Attached Figure Description

[0069] Figure 1 This is a flowchart illustrating the implementation of the present invention.

[0070] Figure 2This describes the specific process for generating data in this invention.

[0071] Figure 3 This is a schematic diagram of the low-rank adaptive principle of the present invention.

[0072] Figure 4 This is an iterative graph showing the evaluation score of the model in this invention. Detailed Implementation

[0073] The embodiments and effects of the present invention will be further described below with reference to the accompanying drawings.

[0074] The purpose of this invention is to address the shortcomings of existing technologies by employing specific web crawling techniques to achieve efficient collection of geoscientific knowledge data. It proposes a large-scale geoscientific language model based on enhanced low-rank adaptive modeling, a language model deeply integrated with geoscientific expertise. This model fine-tunes the generated GSLLM model using low-rank adaptive methods, resulting in a professional question-answering model capable of flexibly adapting to various questions and providing the most accurate answers. This innovation significantly reduces resource consumption, improves model accuracy, and the large number of parameters enhances its generalization ability in geoscientific applications, ensuring accuracy and efficiency in solving various complex geospatial problems. Simultaneously, it enables rapid data analysis and deep knowledge mining.

[0075] Reference Figure 1 A method for constructing a large-scale geoscientific language question-answering model based on enhanced low-rank adaptation includes the following steps:

[0076] Step 1: Collect geoscientific data, including but not limited to China Geographic Network, geoscientific books, and geoscientific research papers;

[0077] The specific method for step 1 is as follows:

[0078] Data was collected from multiple reliable sources, with a focus on Chinese geography. The collected data included the China Geographic Information Network (CNGIS), geoscience textbooks, and related research papers. CNGIS, as an authoritative geographic information platform, provides detailed geographic data and materials covering various aspects such as topography, climate, and ecology. Geoscience textbooks provide systematic and theoretical knowledge support for this invention, helping it to understand and organize geographic data. Related research papers offer the latest research results and academic perspectives, enabling this invention to acquire cutting-edge geographic knowledge.

[0079] Step 2: Preprocess the geoscientific data collected in Step 1 to obtain a more accurate geoscientific dataset;

[0080] The specific method for step 2 is as follows:

[0081] Step 2.1: Filter and screen the geoscientific data collected in Step 1, and use the features of geoscientific knowledge as keywords to assist the crawling technology in crawling relevant local knowledge;

[0082] Step 2.2 utilizes multithreading technology and asynchronous processing methods to enable the web crawling technology used in Step 2.1 to crawl multiple targets.

[0083] Step 2.3: Using data cleaning and format conversion tools, remove the parts of the data obtained in Step 2.2 that are not related to geoscience, and format the data obtained in Step 2.2.

[0084] Step 2.4 involves reviewing and correcting the data obtained in Step 2.3. This includes correcting erroneous information, updating outdated information, and conducting multiple data verifications and comparative analyses to ultimately obtain the geoscience dataset.

[0085] In step 2, the geoscientific data sources collected in step 1 are filtered and selected based on the characteristics of geoscientific knowledge, enabling the crawler script to automatically retrieve a large amount of geographical data from the required networks. During the crawling process, this invention employs multi-threading technology and asynchronous processing methods to improve data acquisition efficiency. Furthermore, this invention uses data cleaning and format conversion tools to ensure that the data retrieved from the network meets the dataset requirements of this invention and facilitates subsequent processing and analysis. To further improve data quality, this invention also incorporates the opinions of geographical experts to screen and verify the collected data. Geographical experts utilize their extensive professional knowledge and practical experience to comprehensively review and correct the data, eliminating inaccurate or outdated information to ensure the data's authority. Simultaneously, this invention conducts multiple rounds of data verification and comparative analysis to guarantee the internal consistency and external reliability of the data. Through this series of rigorous screening and verification processes, this invention ultimately obtains high-quality, comprehensive, and reliable geographical knowledge data.

[0086] Step 3: Classify the geoscience dataset obtained in Step 2 to obtain a geoscience text dataset and a geoscience question-and-answer dataset;

[0087] Step 3.1: Classify the geoscience dataset obtained in Step 2. Convert the geoscience data collected from China Geographic Network and geoscience-related scientific research papers in the geoscience dataset obtained in Step 2 into the format of the geoscience text dataset used for pre-training, and obtain the geoscience text dataset.

[0088] Step 3.2 involves converting and fine-tuning the format of the geoscience question-and-answer dataset used in the geoscience dataset obtained in Step 2 from geoscience professional books, to obtain the geoscience question-and-answer dataset.

[0089] Reference Figure 2 In step 3, the geoscientific dataset collected in step 2 is categorized. First, this invention integrates data collected from organized research papers related to the China Geographic Network and geosciences into a comprehensive geoscientific text dataset. This data includes the latest research findings, detailed descriptions of geographical phenomena, and regional geographical features, providing rich content for the pre-trained model of this invention. To ensure data diversity and coverage, this invention pays special attention to integrating data from different sources, ensuring that the dataset covers various geographical knowledge areas, such as landforms, climate, hydrology, and vegetation. Simultaneously, this invention transforms the content of geoscientific books into a question-and-answer format, generating the geoscientific question-and-answer dataset of this invention. That is, this invention extracts key knowledge points and common questions from geoscientific books and generates corresponding question-and-answer pairs using natural language processing technology. For example, from a chapter on climate change, this invention extracts the common question "What is the greenhouse effect?" and generates a detailed answer. These question-and-answer pairs are also reviewed and revised by geography experts.

[0090] Step 4: Input the geoscience text dataset obtained in Step 3 into the corresponding large language model and perform self-supervised training through federated learning. Utilizing the isomorphic architecture of federated learning, the large language model is trained in a distributed manner on multiple devices. Each device independently processes its local data. After all training epochs are completed, each device sends the parameters of the large language model back to the central server. The central server aggregates these parameters of the large language model to generate the Geographic Sciences Large Language Model (GSLLM) of this invention.

[0091] The large language model in step 4 uses a Transformer Decoder-only network architecture. First, the geoscientific text dataset obtained in step 3 is input into the large language model, and word embeddings are added. Word embeddings convert words or tokens in the text into fixed-dimensional vector representations. After word embeddings, Rotated Position Embedding (RoPE) is used to provide positional information for each word in the sequence without adding extra parameters, thus enabling the model to better understand the relative positional information in the sequence. Then, the large language model in this invention employs a Global Query Attention (GQA) method to reduce the number of parameters and computational complexity in attention calculations. Next, the large language model in this invention uses the SwiGLU activation function, which is an improved non-linear activation function. Finally, Root Mean Square Layer Normalization (RMSNorm) is used to normalize the root mean square of each sample in the input tensor of the large language model to maintain stable training and optimization results. The weighted average update formula is:

[0092]

[0093] Among them, w t+1 This represents the overall weight update result at time t+1, where K indicates that there are a total of K parts, and n k Let represent the number of samples in the k-th part, and n represent the total number of samples. This represents the weight update result of the k-th part at time t+1.

[0094] The large language model in step 4 specifically includes:

[0095] Step 4.1: Set up the root mean square layer normalization (RMSNorm) in the large language model structure. Use the root mean square layer normalization (RMSNorm) in the large language model structure to input the geoscientific text dataset generated in Step 3. Calculate the root mean square value (RMS) of all elements for each input sample, and then normalize it to eliminate scale differences in the input data. The formula for the root mean square value (RMS) is as follows:

[0096]

[0097] Where, x i Let N represent the value of the i-th data point, and N represent the total number of data points.

[0098] Step 4.2: Set the SwiGLU activation function. The SwiGLU activation function uses a gating mechanism to achieve dynamic information selection, improving the model's feature selection ability and performance. The formula for the SwiGLU activation function is as follows:

[0099]

[0100] Where x represents the input vector, W represents the weight matrix, and b represents the bias term;

[0101] Step 4.3: Set the position embedding in the Transformer using Rotation Position Embedding (RoPE). This position embedding method encodes the absolute position using a rotation matrix. First, initialize the rotation angle, create the rotation matrix, and then perform the rotation matrix operation. The angle calculation and rotation vector formula for Rotation Position Embedding (RoPE) are as follows:

[0102] Angle calculation:

[0103] Rotation vector:

[0104] Where, θ i The rotation angle represents the i-th dimension and is used to define the rotation matrix; k is a hyperparameter used to control the magnitude of the rotation angle; d represents the dimension of the vector.

[0105] Step 4.4: Set up Global Query Attention (GQA). First, project the input sequence into query, key, and value vectors. Then, reduce the number of query vectors to one or a few. Depending on your model, you can start with a small number of query vectors and gradually increase them, observing the changes in model performance until you find a suitable balance. Then, use the shared query vectors to perform a dot product with all key vectors to obtain the attention score. Finally, use the attention score to perform a weighted sum of the value vectors to obtain the output vector.

[0106] Reference Figure 3 Step 5: Input the geoscience question-answering dataset obtained in Step 3 into the Geographic Sciences Large Language Model (GSLLM) obtained in Step 4 for fine-tuning. Then, set up an enhanced low-rank adaptive technique, that is, decompose multiple low-rank matrices and assign different learning rates to optimize parameter adjustment. Finally, add the DeepSpeed ​​acceleration framework to the Geographic Sciences Large Language Model (GSLLM) to improve computational efficiency and model performance, so as to achieve efficient processing and accurate answers to geoscience question-answering data; generate a large geoscience language question-answering model.

[0107] The specific process of fine-tuning the Geographic Sciences Large Language Model (GSLLM) in step 5 is as follows:

[0108] Step 5.1: During fine-tuning, the weights of the Geographic Sciences Large Language Model (GSLLM) are decomposed into low-rank matrices, specifically, the original weight matrix W is decomposed into low-rank matrices A and B, i.e., W≈A*B. However, unlike the original low-rank principle, this invention uses multiple low-rank matrices A and B combined together to form a hierarchical structure. Each layer of matrices has a different learning rate (lr-1, lr-2, ..., lr-n) to adapt to the characteristics and training requirements of each matrix. For example, the first layer of low-rank matrices A and B are set with a learning rate of lr-1, and for the next layer of low-rank matrices, the learning rate is set to lr-2, and so on, until the learning rates of all low-rank matrices are set. After setting the learning rates of all low-rank matrices, an adaptive optimization algorithm is used to dynamically adjust the learning rates, and the step size is updated adaptively based on the feedback during the training process.

[0109] Step 5.2: During training, this invention incorporates the DeepSpeed ​​acceleration framework into the Geographic Sciences Large Language Model (GSLLM). This technology reduces memory consumption by utilizing the ZeRO (Zero Redundancy Optimizer) technique within the DeepSpeed ​​framework for distributed data parallel optimization. Specifically, this invention employs stage 0 of zero_optimization; ultimately generating a large-scale geospatial language question-answering model based on enhanced low-rank adaptation.

[0110] Reference Figure 4 Step 6: Evaluate the geoscientific language question-answering model generated in Step 5 by setting four evaluation methods: BLEU4, ROUGE-1, ROUGE-2 and ROUGE-L. Continuously optimize the geoscientific language question-answering model using the scores and save the weight file of the best score that the geoscientific language question-answering model can achieve in the evaluation.

[0111] The specific method for step 6 is as follows:

[0112] Step 6.1: After training, the geospatial language question-answering model generated in Step 5 is evaluated. This invention uses four evaluation methods: BLEU4, ROUGE-1, ROUGE-2, and ROUGE-L to evaluate the model's score. The formulas for BLEU4, ROUGE-1, ROUGE-2, and ROUGE-L are as follows:

[0113]

[0114] Where exp represents the natural exponential function, w represents the weight, and BP (Brevity Penalty) is the penalty for overly short translations, calculated as follows:

[0115]

[0116] Where c represents the number of n-grams in the candidate translation (machine translation), and r represents the number of n-grams in the reference translation.

[0117]

[0118] Where `max` represents the maximum value, `count` represents the number of occurrences, `gram` represents an n-gram, i.e., a sequence of n consecutive words in the text, `Candidate` represents machine-generated text, and `Reference` represents manually generated text or known high-quality text. `ROUGE-1` calculates the overlap of a single word, and `ROUGE-2` calculates the overlap of two consecutive words.

[0119]

[0120] Step 6.2: Based on the evaluation score obtained in 6.1, change the hyperparameters of the geoscientific language question answering model when inputting it to continuously optimize the weight file of the training model, and finally retain the weight file with the best score that it can achieve.

[0121] The effects of the present invention will be further explained below with reference to simulation experiments.

[0122] 1. Simulation experimental conditions:

[0123] The hardware testing platform for the simulation experiment of this invention is: CPU 48 cores 192G; graphics card is Ascend 910 Premium A32G.

[0124] The software platform for the simulation experiment of this invention is as follows: Operating system: EulerOS 2.0 (SP8); Online platform: ModelArts; Notebook integrated development environment, Python 3.10.14; Hardware platform: Ascend 910A

[0125] 2. Simulation content and result analysis:

[0126] Table 1 provides detailed information about the geoscience dataset;

[0127] Table 2 shows a comparison of training evaluation index scores.

[0128] The simulation experiments of this invention were conducted using a self-collected and organized geoscience dataset, which fills the gap in current question-and-answer datasets in the geoscience field. Detailed parameters of the data used in this invention are shown in Table 1.

[0129] Table 1. Detailed information on geoscience datasets

[0130]

[0131]

[0132] The simulation experiments of this invention were conducted on a self-created dataset, and the evaluation scores of the original model are compared in Table 2. The experiments clearly show that the performance of the model of this invention is significantly better than that of the original model.

[0133] Table 2 Comparison of Training Evaluation Index Scores of the Invention

[0134]

[0135] The final results show that the present invention scored 89.71 on BLEU-4, 91.26 on ROUGE-1, 89.14 on ROUGE-2, and 90.83 on ROUGE-L.

[0136] The data above demonstrates that this invention saves training costs while ensuring that the geospatial language question-answering model performs well on current mainstream evaluation metrics.

[0137] This invention also provides a large-scale geoscientific language question-answering model construction system based on enhanced low-rank adaptive modeling, comprising:

[0138] The geoscience data collection module is used to collect geoscience data in step 1, including but not limited to the China Geographic website, geoscience books and geoscience-related research papers.

[0139] The geoscientific data preprocessing module is used to preprocess the geoscientific data collected in step 1 in step 2 to obtain a geoscientific dataset.

[0140] The geoscience dataset classification module is used to classify the geoscience dataset obtained in step 2 in step 3, resulting in a geoscience text dataset and a geoscience question-and-answer dataset.

[0141] The Geographic Sciences Large Language Model (GSLLM) generation module is used to input the geographic text dataset obtained in step 3 into the corresponding large language model in step 4. Self-supervised training is performed through federated learning. Utilizing the isomorphic architecture of federated learning, the large language model is trained in a distributed manner on multiple devices. Each device independently processes its local data. After all training epochs are completed, each device sends the parameters of the large language model back to the central server. The central server aggregates the parameters of the large language model to generate the Geographic Sciences Large Language Model (GSLLM).

[0142] The large-scale geoscience language question-answering model generation module is used to input the geoscience question-answering dataset obtained in step 3 into the geoscience large language model (GSLLM) obtained in step 4 for fine-tuning in step 5, and then set up an enhanced low-rank adaptive technique, that is, to optimize parameter adjustment by decomposing multiple low-rank matrices and assigning different learning rates. Finally, the DeepSpeed ​​acceleration framework is added to the geoscience large language model (GSLLM) to generate the large-scale geoscience language question-answering model.

[0143] The optimal weight file acquisition module is used to evaluate the score of the geoscientific and linguistic question-answering model generated in step 5 in step 6, continuously optimize the geoscientific and linguistic question-answering model using the score, and save the weight file of the best score that the geoscientific and linguistic question-answering model can achieve in the evaluation.

[0144] This invention also provides a device for constructing a large geoscientific language question-answering model based on enhanced low-rank adaptive methods, comprising:

[0145] Memory: A computer-readable device that stores the computer program of the above-mentioned method for constructing a large geoscientific language question-answering model based on enhanced low-rank adaptive Memory;

[0146] Processor: Used to implement the method for constructing a large geoscientific language question-answering model based on enhanced low-rank adaptive methods when executing the computer program.

[0147] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, enables the implementation of the aforementioned method for constructing a large geoscientific language question-answering model based on enhanced low-rank adaptive methods.

Claims

1. A method for constructing a large-scale geoscience question and answer model based on enhanced low-rank adaptation, characterized in that, Comprise: Step 1, collect geoscience data, including but not limited to China geographic network, geoscience professional books and geoscience professional related scientific research papers; Step 2, pre-processing the geoscience data collected in step 1 to obtain a geoscience dataset; Step 3, classifying the geoscience dataset obtained in step 2 to obtain a geoscience text dataset and a geoscience question and answer dataset; Step 4, inputting the geoscience text dataset obtained in step 3 into a corresponding large language model, performing self-supervised training through federated learning, utilizing the isomorphic architecture of federated learning, the large language model is trained in a distributed manner on multiple devices, each device independently processes its local data, after all training rounds (epochs) are completed, each device sends the parameters of the large language model back to the central server, the central server aggregates the parameters of the large language model to generate a geoscience large language model (Geographic Sciences Large language Model, GSLLM); The large language model specifically comprises: Step 4.1, setting the root mean square layer normalization (RMSNorm) in the large language model structure, using the root mean square layer normalization (RMSNorm) in the large language model structure to input the geoscience text dataset generated in step 3, calculating the root mean square value (RMS) of all elements of each input sample, then normalizing, the root mean square value (RMS) formula is as follows: where x i represents the value of the i-th data point, and N represents the total number of data points; Step 4.2, setting the SwiGLU activation function, the SwiGLU activation function formula is as follows: Where x represents the input vector, W represents the weight matrix, and b represents the bias term; Step 4.3, using the rotation position embedding (RoPE) to set the position embedding in the Transformer, this position embedding method encodes the absolute position using a rotation matrix, first initialize the rotation angle, create a rotation matrix, then rotate the position embedding (RoPE) angle calculation and rotation vector formula as follows: Angle calculation: Rotational vector: where θ i represents the rotation angle of the i-th dimension, used to define the rotation matrix; k is a hyperparameter used to control the size of the rotation angle; d represents the dimension of the vector; Step 4.4, setting global query attention (GQA), first project the input sequence into query, key and value vectors, then reduce the number of query vectors to one or several, according to the model, you can start with a smaller number of query vectors, gradually increase, observe the change of model performance, until a suitable balance point is found, then use the shared query vector to calculate the dot product with all key vectors to get the attention score, finally use the attention score to weight sum the value vector to get the output vector; Step 5, input the geographic Q&A data set obtained in step 3 into the geographic large language model (GSLLM) obtained in step 4 for fine-tuning, and set up an enhanced low-rank adaptive technology, that is, by decomposing multiple low-rank matrices and assigning different learning rates to optimize parameter adjustment, finally add DeepSpeed acceleration framework to the geographic large language model (GSLLM) to generate a large geographic language Q&A model; Step 6, score evaluation is performed on the large geographic language Q&A model generated by training in step 5, and the large geographic language Q&A model is continuously optimized using the score, and the weight file of the best score that the large geographic language Q&A model can achieve in evaluation is saved.

2. The method according to claim 1, wherein, The specific method of step 2 is: Step 2.1, filtering and screening the geographic data collected in step 1, using the characteristics of geographic knowledge as feature words to assist the crawler technology to crawl related local knowledge; Step 2.2, use multi-threading technology and asynchronous processing method to make the crawler technology in step 2.1 crawl multiple targets; Step 2.3, use data cleaning and format conversion tools to remove the part of the data obtained in step 2.2 that does not belong to geography, and format the data obtained in step 2.2; Step 2.4, review and correction, correct the incorrect knowledge in the data obtained in step 2.3, update the outdated knowledge points, and perform multiple data verification and comparison analysis on the data to obtain a geographic data set.

3. The method of claim 1, wherein the method is characterized by, The specific method of step 3 is: Step 3.1, classify the geographic data set obtained in step 2, convert the geographic data collected from the Chinese geographic network and the scientific research papers related to the professional research of geography in the geographic data set obtained in step 2 into the format of the pre-training geographic text data set, and obtain a geographic text data set; Step 3.2, convert the geographic data collected from the books related to the professional research of geography in the geographic data set obtained in step 2 into the format of the fine-tuning geographic Q&A data set, and obtain a geographic Q&A data set.

4. The method of claim 1, wherein the method is characterized by, The large language model of step 4 uses the Decoder-only network architecture of Transformer, inputs the geographic text data set obtained in step 3 into the large language model, and adds word embedding, which converts words or tokens in the text into fixed-dimensional vector representations, after word embedding, rotation position embedding (RoPE) is used to provide position information of each word in the sequence without increasing additional parameters, the large language model adopts a global query attention (GQA), uses SwiGLU activation function, and adopts root mean square layer normalization (RMSNorm) to normalize the root mean square of each sample in the input tensor of the large language model, and the weight weighted average update formula is: wherein w t+1 represents the overall weight update result at t+1, K represents a total of K parts, n k represents the number of samples in the kth part, n represents the total number of samples, represents the weight update result of the kth part at t+1.

5. The method of claim 1, wherein the method is characterized by, The specific process of fine-tuning the Geographic Sciences Large language Model (GSLLM) in step 5 is as follows: Step 5.1, during fine-tuning, the weights of the Geographic Sciences Large language Model (GSLLM) are decomposed into multiple low-rank matrices; the original weight matrix W is decomposed into low-rank matrices A and B, i.e. W ≈ A*B, a hierarchical structure is formed by combining multiple low-rank matrices A and B together, each layer of matrix has different learning rate (lr-1, lr-2,..., lr-n), set the learning rate of all low-rank matrices, then use adaptive optimization algorithm to dynamically adjust the learning rate, and adjust the parameter update step according to the feedback in the training process; Step 5.2, during the training process, the DeepSpeed acceleration framework is selected to add to the Geographic Sciences Large language Model (GSLLM), and finally a large geographic language question and answer model based on enhanced low-rank adaptive is generated.

6. The method of claim 1, wherein the method is characterized by, The specific method of step 6 is as follows: Step 6.1, score evaluation is performed on the large geographic language question and answer model generated in step 5, and four evaluation methods of BLEU4, ROUGE-1, ROUGE-2 and ROUGE-L are used to evaluate the score of the large geographic language question and answer model; the formulas of BLEU4, ROUGE-1, ROUGE-2 and ROUGE-L are as follows: Where exp represents the natural exponential function, w represents the weight, BP (Brevity Penalty): penalty for short translation, the calculation formula is: Where c represents the number of n-grams in the candidate translation (machine translation); r represents the number of n-grams in the reference translation; Where max represents the maximum value, count represents the number of statistics, gram: represents an n-gram, i.e. a sequence of n consecutive words in the text, Candidate represents the machine-generated text, Reference represents the manually generated text or known high-quality text, ROUGE-1 calculates the overlap of single words, and ROUGE-2 calculates the overlap of two consecutive words; Step 6.2, according to the evaluation score obtained in 6.1, change the hyperparameters when inputting the large geographic language question and answer model, optimize the weight file of the training model, and finally retain the weight file with the best score.

7. A large-scale geoscience language question and answer model construction system based on enhanced low-rank self-adaption according to any one of claims 1 to 6, characterized in that, Including: Geoscience data collection module, used for collecting geoscience data, including but not limited to China geographic network, geoscience professional books and geoscience professional related scientific research papers; Geoscience data preprocessing module, used for preprocessing the collected geoscience data to obtain geoscience dataset; Geoscience dataset classification module, used for classifying the geoscience dataset to obtain geoscience text dataset and geoscience question and answer dataset; The Geographic Sciences Large language Model (GSLLM) generation module is configured to input a geographic text dataset into a corresponding large language model, perform self-supervised training through federated learning, utilize the homogeneous architecture of federated learning, and perform distributed training of the large language model on multiple devices, with each device independently processing local data. After all training epochs are completed, each device sends the parameters of the large language model back to a central server, and the central server aggregates the parameters of the large language model to generate a Geographic Sciences Large language Model (GSLLM). The Geographic Sciences Language Question and Answer Large Model generation module is configured to input a geographic question and answer dataset into the Geographic Sciences Large language Model (GSLLM) for fine-tuning, set up an enhanced low-rank adaptive technology, i.e., decompose multiple low-rank matrices and assign different learning rates to optimize parameter adjustment, and finally add a DeepSpeed acceleration framework to the Geographic Sciences Large language Model (GSLLM) to generate a Geographic Sciences Language Question and Answer Large Model. The optimal weight file acquisition module is configured to score evaluate the generated Geographic Sciences Language Question and Answer Large Model, continuously optimize the Geographic Sciences Language Question and Answer Large Model using the score, and save the weight file of the best score that the Geographic Sciences Language Question and Answer Large Model can achieve in the evaluation.

8. A device for constructing a geoscience language question and answer large model based on enhanced low-rank adaptation, characterized in that, The computer program for constructing a Geographic Sciences Language Question and Answer Large Model based on enhanced low-rank adaptation according to any one of claims 1-6 is stored in the memory and is readable by the computer. The processor is configured to execute the computer program to implement the method for constructing a Geographic Sciences Language Question and Answer Large Model based on enhanced low-rank adaptation according to any one of claims 1-6. The computer program for constructing a Geographic Sciences Language Question and Answer Large Model based on enhanced low-rank adaptation according to any one of claims 1-6 is stored in the computer-readable storage medium and is executable by the processor.

9. A computer-readable storage medium, characterized in that, ​

Images