Enhanced low-rank self-adaption-based geoscience language question and answer large model construction method, system and equipment and medium

Abstract

The invention discloses a geoscience language question and answer large model construction method, system and device based on enhanced low-rank self-adaption and a medium, and the method comprises the steps: collecting geoscience data, and carrying out the preprocessing of the geoscience data, and obtaining a geoscience data set; classifying the geoscience data set to obtain a geoscience text data set and a geoscience question and answer data set; inputting the geoscience text data set into a corresponding large language model for training to generate a geoscience large language model; inputting the geoscience question and answer data set into a geoscience large language model for fine tuning, setting an enhanced low-rank adaptive technology, decomposing a plurality of low-rank matrixes and endowing different learning rates to optimize parameter adjustment, and finally adding a DeepSpeed acceleration framework to generate a geoscience language question and answer large model; performing score evaluation and continuous optimization on the geoscience language question and answer large model generated by training to obtain a weight file of an optimal score; the system, the equipment and the medium are used for implementing the method. According to the method, the accuracy and generalization ability of the model are improved while resources required by calculation are greatly reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and specifically relates to a method, system, device and medium for constructing a large model of geoscience language question and answer based on enhanced low-rank adaptation. Background Art

[0002] Geology is to discover the natural laws contained in the observation data through the observation of natural phenomena, so as to study the various processes, changes and interaction laws of the earth system at multiple scales. With the development of geoscience, geoscience research is showing new characteristics of interdisciplinary, cross-regional and collaborative. Geology involves many scientific materials such as atmosphere, ecology, hydrology, soil, ocean, geology, etc., and is also closely related to physics, chemistry and information science. Geological big data has the "4V" common characteristics of traditional big data, such as large data volume (volume), various types (variety), high speed and high efficiency (velocity) and low value density (value). Using these geological knowledge well can help people deepen their understanding of the earth's critical zones, global environmental changes, the interaction between the earth's spheres and the impact of human activities.

[0003] In the current era of information explosion, although traditional search engines and web query methods can provide a large amount of information, these methods are often inefficient because the results they return usually contain a large amount of irrelevant or redundant data. Users need to invest a lot of time and energy in manual screening and filtering to extract valuable information. This process is not only time-consuming and labor-intensive, but also easily limited by personal experience and knowledge level, resulting in incomplete or biased information screening. In addition, although the current mainstream question-answering systems, such as ChatGPT and Kimi, can quickly generate answers through artificial intelligence technology, the accuracy and reliability of these answers cannot always be guaranteed. These systems may have limitations in understanding complex problems, handling professional terms, or providing in-depth analysis, especially in professional fields such as geoscience research, these limitations may be more obvious. Therefore, when dealing with geoscience-related problems, it is particularly urgent and important to use professional geoscience models.

[0004] The patent application document with publication number CN118193708A discloses a mineral knowledge question-and-answer method and system based on a large language model. The method uses crawler technology to collect mineral data from a professional platform, and divides it into knowledge documents and question-and-answer pairs after cleaning. The XGBoost algorithm is used to screen the most relevant documents to assist the large language model in generating answers. At the same time, the model is fine-tuned through LoRA technology, and prompt words are designed to guide the generation of professional mineral content, thereby improving the professional answers of the mineral question-and-answer system. However, this method has some limitations: first, it is mainly aimed at mineral knowledge, a subdivision of geographical science, and its application scope is relatively limited. Secondly, the XGBoost algorithm used faces the problem of high time and space complexity, which not only increases the resource consumption required to generate answers, but also has less than ideal performance when processing high-dimensional sparse data, further restricting its application potential in a wider range of fields.

[0005] The patent application document with publication number CN117892711A discloses a method for obtaining text relevance based on a large model. The method uses transformer technology to build an LLPM large language band probability model, collects relevant training knowledge points, and passes them to the large model after encoding and word segmentation. After the large model is trained, a model with probability calculation output is generated. By calculating the correlation between new knowledge points and existing knowledge points, the accuracy of text relevance calculation is improved. However, this method has some limitations: First, it is mainly aimed at the sub-field of large model training and probability calculation, and its application scope is relatively limited. Secondly, the transformer technology and LLPM large model used require a lot of computing resources and storage space, which increases the system complexity and resource consumption, and further restricts its application potential in a wider range of fields.

[0006] The patent application document with publication number CN118171732A discloses a method and device for extracting super-relational knowledge based on a fine-tuned large model. The method obtains text data, constructs super-relational knowledge extraction instructions based on preset fine-tuning parameters, and constructs a super-relational knowledge extraction model using a fine-tuned large language model. The model adopts a learning rate warm-up and decay strategy during the training process, and is trained using a training data set to finally obtain a trained super-relational knowledge extraction model. The model can effectively extract super-relational knowledge from the text to be parsed and obtain relational data. By designing targeted instructions, entities and their related multivariate relations can be extracted from the model output. However, this method has some limitations: first, it is mainly aimed at the specific task of super-relational knowledge extraction, and its application scope is relatively limited. Although it has a good performance in super-relational knowledge extraction, its applicability and effectiveness may be limited in other types of knowledge extraction tasks or natural language processing tasks. Summary of the invention

[0007] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a method, system, device and medium for constructing a large model of geological language question and answer based on enhanced low-rank adaptation. By adopting specific crawler technology to achieve efficient collection of geological knowledge data, a large amount of accurate geological data can be obtained, and the professionalism of the model can be improved, so that it can flexibly adapt to various problems and make the most correct answers; by using enhanced low-rank adaptation, a small part of parameters can be used to fine-tune the generated large geological language model to obtain a professional large geological language question and answer model; the present invention improves the accuracy and generalization ability of the model while greatly reducing the resources required for calculation.

[0008] To achieve the above object, the technical solution of the present invention includes:

[0009] A method for constructing a large geoscience language question answering model based on enhanced low-rank adaptation, comprising:

[0010] Step 1: Collect geoscience data, including but not limited to China Geography Network, geoscience books and geoscience-related scientific research papers;

[0011] Step 2, preprocessing the geoscience data collected in step 1 to obtain a geoscience data set;

[0012] Step 3, classify the geoscience dataset obtained in step 2 to obtain a geoscience text dataset and a geoscience question-answering dataset;

[0013] Step 4: Input the geoscience text dataset obtained in step 3 into the corresponding large language model, and perform self-supervised training through federated learning. Using the homogeneous architecture of federated learning, the large language model is distributedly trained on multiple devices, and each device processes its local data independently. After all training rounds (epochs), each device sends the parameters of the large language model back to the central server, and the central server aggregates the parameters of the large language model to generate a Geographic Sciences Large Language Model (GSLLM).

[0014] Step 5: Input the geoscience question-answering dataset obtained in step 3 into the Geographic Sciences Large Language Model (GSLLM) obtained in step 4 for fine-tuning, and then set the enhanced low-rank adaptive technology, that is, by decomposing multiple low-rank matrices and assigning different learning rates to optimize parameter adjustment, and finally add the DeepSpeed ​​acceleration framework to the Geographic Sciences Large Language Model (GSLLM) to generate a large model of geoscience language question-answering;

[0015] Step 6, perform score evaluation on the geoscience language question and answer big model generated by the training in step 5, use the score to continuously optimize the geoscience language question and answer big model, and save the weight file of the best score that the geoscience language question and answer big model can achieve in the evaluation.

[0016] The specific method of step 2 is:

[0017] Step 2.1, filter and screen the geoscience data collected in step 1, and use the characteristics of geoscience knowledge as feature words to assist crawler technology in crawling relevant local knowledge;

[0018] Step 2.2, using multi-threading technology and asynchronous processing methods, so that the crawler technology used in step 2.1 crawls multiple targets;

[0019] Step 2.3, using data cleaning and format conversion tools, remove the non-geological parts of the data obtained in step 2.2, and format the data obtained in step 2.2;

[0020] Step 2.4, review and correct, correct the erroneous knowledge in the data obtained in step 2.3, update the outdated knowledge points, and perform multiple data verification and comparative analysis on the data to obtain a geoscience data set.

[0021] The specific method of step 3 is:

[0022] Step 3.1, classify the geoscience data set obtained in step 2, convert the geoscience data collected from the China Geography Network and geoscience related scientific research papers in the geoscience data set obtained in step 2 into the format of the geoscience text data set used for pre-training, and obtain the geoscience text data set;

[0023] Step 3.2: convert the geoscience data collected from geoscience professional books in the geoscience dataset obtained in step 2 into the format of the geoscience question-answering dataset used for fine-tuning, thereby obtaining the geoscience question-answering dataset.

[0024] The large language model in step 4 uses the Transformer Decoder-only network architecture, inputs the geoscientific text dataset obtained in step 3 into the large language model, and adds word embedding. The word embedding converts the words or tags in the text into vector representations of fixed dimensions. After the word embedding, the rotation position embedding (RoPE) is used to provide the position information of each word in the sequence without adding additional parameters. The large language model adopts a global query attention (GQA), and uses the SwiGLU activation function, and uses the root mean square layer normalization (RMSNorm) to normalize the root mean square of each sample in the large language model input tensor. The weighted average update formula is:

[0025]

[0026] Among them, w t+1 represents the overall weight update result at time t+1, K represents a total of K parts, n k represents the number of samples in the kth part, n represents the total number of samples, Represents the weight update result of the kth part at time t+1.

[0027] The large language model in step 4 specifically includes:

[0028] Step 4.1, set the root mean square layer normalization (RMSNorm) in the large language model structure, use the root mean square layer normalization (RMSNorm) in the large language model structure to input the geoscientific text dataset generated in step 3, calculate the root mean square value (RMS) of all elements of each input sample, and then normalize it. The root mean square value (RMS) formula is as follows:

[0029]

[0030] Among them, x i represents the value of the i-th data point, and N represents the total number of data points;

[0031] Step 4.2, set the SwiGLU activation function. The SwiGLU activation function formula is as follows:

[0032]

[0033] Among them, x represents the input vector, W represents the weight matrix, and b represents the bias term;

[0034] Step 4.3, use rotation position embedding (RoPE) to set the position embedding in Transformer. This position embedding method encodes the absolute position with a rotation matrix. First, initialize the rotation angle, create a rotation matrix, and then rotate the matrix. The angle calculation and rotation vector formula of rotation position embedding (RoPE) are as follows:

[0035] Angle calculation:

[0036] Rotate vector:

[0037] Among them, θ i represents the rotation angle of the i-th dimension, which is used to define the rotation matrix; k is a hyperparameter used to control the size of the rotation angle; d represents the dimension of the vector;

[0038] Step 4.4, set up global query attention (GQA), first project the input sequence into query, key, and value vectors, then reduce the number of query vectors to one or several. According to your own model, you can start with a smaller number of query vectors and gradually increase it to observe the changes in model performance until you find a suitable balance point. Then use the shared query vector to perform dot product calculations with all key vectors to obtain the attention score. Finally, use the attention score to perform weighted summation on the value vector to obtain the output vector.

[0039] The specific process of fine-tuning the Geographic Sciences Large Language Model (GSLLM) in step 5 is as follows:

[0040] Step 5.1, during fine-tuning, perform low-rank matrix decomposition on the weights of the Geographic Sciences Largelanguage Model (GSLLM) into multiple low-rank matrices; decompose the original weight matrix W into low-rank matrices A and B, that is, W≈A*B, and combine multiple low-rank matrices A and B to form a hierarchical structure. Each layer of the matrix has a different learning rate (lr-1, lr-2,..., lr-n). Set the learning rate of all low-rank matrices, and then use the adaptive optimization algorithm to dynamically adjust the learning rate. Adaptively adjust the parameter update step size according to the feedback during the training process;

[0041] Step 5.2: During the training process, the Geographic Sciences Largelanguage Model (GSLLM) is selected to join the DeepSpeed ​​acceleration framework, and finally a large model of Geographic Sciences language question answering based on enhanced low-rank adaptation is generated.

[0042] The specific method of step 6 is:

[0043] Step 6.1: Evaluate the scores of the geoscience language question answering model generated in step 5. Four evaluation methods, BLEU4, ROUGE-1, ROUGE-2, and ROUGE-L, are used to evaluate the scores of the geoscience language question answering model. The formulas for BLEU4, ROUGE-1, ROUGE-2, and ROUGE-L are as follows:

[0044]

[0045] Among them, exp represents the natural exponential function, w represents the weight, BP (Brevity Penalty): penalizes translations that are too short, and the calculation formula is:

[0046]

[0047] Where c represents the number of n-grams in the candidate translation (machine translation); r represents the number of n-grams in the reference translation;

[0048]

[0049] Among them, max means taking the maximum value, count means counting the number, gram: represents an n-gram, that is, a sequence of n consecutive words in the text, Candidate represents machine-generated text, Reference represents artificially generated text or known high-quality text, ROUGE-1 calculates the overlap of a single word, and ROUGE-2 calculates the overlap of two consecutive words;

[0050]

[0051] The score is 6.2. Based on the evaluation score obtained in 6.1, the hyperparameters of the large model input for geoscience language question answering are changed, and the weight file of the training model is continuously optimized. Finally, the weight file with the best score is retained.

[0052] The present invention also provides a geoscience language question-answering large model construction system based on enhanced low-rank adaptation, comprising:

[0053] The geoscience data collection module is used to collect geoscience data, including but not limited to China Geography Network, geoscience books and geoscience-related scientific research papers;

[0054] A geoscience data preprocessing module is used to preprocess the collected geoscience data to obtain a geoscience data set;

[0055] The geoscience dataset classification module is used to classify the geoscience dataset to obtain the geoscience text dataset and the geoscience question-answering dataset;

[0056] The Geographic Sciences Large Language Model (GSLLM) generation module is used to input the geoscience text dataset into the corresponding large language model, and perform self-supervised training through federated learning. Using the homogeneous architecture of federated learning, the large language model is distributedly trained on multiple devices, and each device processes its local data independently. After all training rounds (epochs), each device sends the parameters of the large language model back to the central server, and the central server aggregates the parameters of the large language model to generate the Geographic Sciences Large Language Model (GSLLM);

[0057] The module for generating a large model for geography language question answering is used to input the geography question answering dataset into the Geographic Sciences Large Language Model (GSLLM) for fine-tuning, and then set up enhanced low-rank adaptation technology, that is, by decomposing multiple low-rank matrices and assigning different learning rates to optimize parameter adjustment, and finally adding the DeepSpeed ​​acceleration framework to the Geographic Sciences Large Language Model (GSLLM) to generate a large model for geography language question answering;

[0058] The best weight file acquisition module is used to evaluate the scores of the geoscience language question and answer big model generated by training, use the scores to continuously optimize the geoscience language question and answer big model, and save the weight file of the best score that the geoscience language question and answer big model can achieve in the evaluation.

[0059] The present invention also provides a device for constructing a large model of geoscience language question answering based on enhanced low-rank adaptation, comprising:

[0060] Memory: a computer program storing the above-mentioned method for building a large model of geoscience language question answering based on enhanced low-rank adaptation, which is a computer-readable device;

[0061] Processor: used to implement the method for constructing a large model of geoscience language question and answer based on enhanced low-rank adaptation when executing the computer program.

[0062] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the method for constructing a large model of geological language question and answer based on enhanced low-rank adaptation.

[0063] Compared with the prior art, the present invention has the following advantages:

[0064] First, we created a dedicated data set for the field of geographic science, and introduced a method that combines automated error correction and manual screening. The specific steps include data cleaning, format conversion, and expert review. In step 2, we used specific crawler technology to efficiently collect geoscientific knowledge data, which enabled us to obtain a large amount of accurate geoscientific data and improve the professionalism of the model. In the data processing process, we overcame the noise and inconsistency of a large amount of data, ensuring the accuracy and reliability of the data.

[0065] Second, in view of the complexity of the geoscience field, step 4 uses federated learning and multi-node distributed training to generate a geoscience language question-answering model with 8 billion parameters. This huge number of parameters enables the model to more accurately capture the grammar, semantics and contextual information of the language, thereby generating more natural and coherent text. In addition, the geoscience language question-answering model is more robust in the face of noisy and inconsistent data. Even when the data is incomplete or noisy, it can still generate reasonable outputs while taking into account a wider range of contexts to ensure the coherence and consistency of the text.

[0066] Third, for the field of geographical sciences, step 4 generates a Geographic Sciences Large Language Model (GSLLM), which fills the gap in the current large pre-trained models of geosciences. On this basis, the downstream tasks are further refined, and a large number of geoscience question-and-answer datasets are used to generate a more professional geoscience question-and-answer large model, ensuring its accuracy and efficiency in solving various complex geographic spatial problems, while achieving rapid data analysis and in-depth knowledge mining.

[0067] Fourth, in view of the huge number of parameters of the Geographic Sciences Large language Model (GSLLM), which may lead to excessive resource consumption, step 5 proposes an enhanced low-rank adaptive technology, which captures the main features of the data by decomposing the original high-dimensional data matrix into the product of matrices with lower rank, and sets a low-rank matrix that is adaptively adjusted according to the structure of the Geographic Sciences Large language Model (GSLLM). While greatly reducing the resources required for calculation, it improves the accuracy and generalization ability of the Geographic Sciences Large language Model (GSLLM).

[0068] In summary, the present invention improves the accuracy and reliability of the geological language question and answer model of the present invention in the geological field by producing professional geological text datasets and geological question and answer datasets, while taking into account a wider range of contexts to ensure the coherence and consistency of the text; and by utilizing the enhanced low-rank adaptive method, the present invention can achieve a training effect comparable to full training while significantly reducing the resources required for calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 It is a flow chart for implementing the present invention.

[0070] Figure 2The specific process of data generation for the present invention.

[0071] Figure 3 This is a diagram of the low-rank adaptation principle of the present invention.

[0072] Figure 4 Evaluation score iteration graph for the proposed model. DETAILED DESCRIPTION

[0073] The embodiments and effects of the present invention are further described below in conjunction with the accompanying drawings.

[0074] The purpose of the present invention is to address the deficiencies of the above-mentioned existing technologies, use specific crawler technology to achieve efficient collection of geoscientific knowledge data, and propose a geoscientific language model based on enhanced low-rank adaptation, a language model that deeply integrates geoscientific professional knowledge, which fine-tunes the generated GSLLM model by applying low-rank adaptation to obtain a professional question-answering model that can flexibly adapt to various questions and give the most correct answers. This innovation will significantly reduce resource consumption and improve the accuracy of the model, and a large number of parameters can improve the generalization ability in geoscientific applications, ensuring its accuracy and efficiency in solving various complex geographic spatial problems, while achieving rapid analysis of data and deep knowledge mining.

[0075] Reference Figure 1 , a method for constructing a large geoscience language question answering model based on enhanced low-rank adaptation, comprising the following steps:

[0076] Step 1: Collect geoscience data, including but not limited to China Geography Network, geoscience books and geoscience-related scientific research papers;

[0077] The specific method of step 1 is:

[0078] Data is collected from multiple reliable sources. The present invention focuses on Chinese geography, and the collected data include China Geography Network, geography professional books and geography related scientific research papers. As an authoritative geographic information platform, China Geography Network provides detailed geographic data and information, covering information on topography, climate, ecology and other aspects; geography professional books provide systematic and theoretical knowledge support for the present invention, helping the present invention to understand and organize geographic data; related scientific research papers provide the latest research results and academic viewpoints, enabling the present invention to obtain the most cutting-edge geographical knowledge.

[0079] Step 2, preprocessing the geoscience data collected in step 1 to obtain a more accurate geoscience data set;

[0080] The specific method of step 2 is:

[0081] Step 2.1, filter and screen the geoscience data collected in step 1, and use the characteristics of geoscience knowledge as feature words to assist crawler technology in crawling relevant local knowledge;

[0082] Step 2.2, using multi-threading technology and asynchronous processing methods, so that the crawler technology used in step 2.1 can crawl multiple targets;

[0083] Step 2.3, using data cleaning and format conversion tools, remove the non-geological parts of the data obtained in step 2.2, and format the data obtained in step 2.2;

[0084] In step 2.4, geographical experts conduct review and correction, correct the erroneous knowledge in the data obtained in step 2.3, update the outdated knowledge points, and perform multiple data verification and comparative analysis on the data to finally obtain the geological data set.

[0085] In the step 2, the source of the geoscience data collected in step 1 is filtered and screened by using the characteristics of geoscience knowledge as the key part, so that the crawler script can automatically crawl a large amount of geography data from the required network. In the crawler process, the present invention adopts multi-threading technology and asynchronous processing methods to improve the efficiency of data collection. In addition, the present invention also uses data cleaning and format conversion tools to ensure that the data crawled from the network meets the data set requirements of the present invention and is convenient for subsequent processing and analysis. In order to further improve the data quality, the present invention also combines the opinions of geography experts to screen and verify the collected data. Geography experts use rich professional knowledge and practical experience to conduct a comprehensive review and correction of the data, eliminate inaccurate or outdated information, and ensure the authority of the data. At the same time, the present invention also conducts multiple rounds of data verification and comparative analysis to ensure the internal consistency and external reliability of the data. Through this series of strict screening and verification processes, the present invention finally obtains a high-quality, comprehensive and reliable geography knowledge data.

[0086] Step 3, classify the geoscience dataset obtained in step 2 to obtain a geoscience text dataset and a geoscience question-answering dataset;

[0087] Step 3.1, classify the geoscience data set obtained in step 2, convert the geoscience data collected from the China Geography Network and geoscience related scientific research papers in the geoscience data set obtained in step 2 into the format of the geoscience text data set used for pre-training, and obtain the geoscience text data set;

[0088] Step 3.2: convert the geoscience data collected from geoscience professional books in the geoscience dataset obtained in step 2 into the format of the geoscience question-answering dataset used for fine-tuning, thereby obtaining the geoscience question-answering dataset.

[0089] Reference Figure 2 , in the step 3, the geoscience data set collected in step 2 is classified; first, the present invention integrates the data collected from the scientific research papers related to the China Geography Network and geoscience majors that have been sorted into a comprehensive geoscience text data set; these data include the latest research results, detailed descriptions of geographical phenomena, regional geographical features, etc., which can provide rich content for the pre-training model of the present invention. In order to ensure the diversity and coverage of data, the present invention pays special attention to the integration of data from different sources to ensure that the data set covers various geographical knowledge fields, such as landforms, climate, hydrology, vegetation, etc. At the same time, the present invention converts the content of geoscience professional books into the form of questions and answers to generate the geoscience question and answer data set of the present invention; that is, the present invention extracts key knowledge points and common questions from geoscience professional books, and generates corresponding question and answer pairs through natural language processing technology. For example, from a chapter on climate change, the present invention extracts common questions such as "What is the greenhouse effect?" and generates detailed answers. These question and answer pairs will also be reviewed and revised by geography experts.

[0090] Step 4, input the geological text data set obtained in step 3 into the corresponding large language model, perform self-supervised training through federated learning, and use the homogeneous architecture of federated learning to perform distributed training on multiple devices. Each device processes its local data independently. After all training epochs, each device sends the parameters of the large language model back to the central server, and the central server aggregates the parameters of these large language models to generate the Geographic Sciences Large Language Model (GSLLM) of this invention.

[0091] The large language model of step 4 uses the decoder-only network architecture of Transformer. First, the geological text dataset obtained in step 3 is input into the large language model, and word embedding is added. The word embedding converts the words or tags in the text into vector representations of fixed dimensions. After the word embedding, the rotation position embedding (RoPE) is used to provide the position information of each word in the sequence without adding additional parameters, so that the model can better understand the relative position information in the sequence. Then, the large language model in this invention adopts a global query attention (GQA) to reduce the number of parameters and computational complexity in the attention calculation. Then, the large language model in this invention uses the SwiGLU activation function, which is an improved nonlinear activation function. Finally, the root mean square layer normalization (RMSNorm) is used to normalize the root mean square of each sample in the input tensor of the large language model to maintain a stable training and optimization effect. The weighted average update formula is:

[0092]

[0093] Among them, w t+1 represents the overall weight update result at time t+1, K represents a total of K parts, n k represents the number of samples in the kth part, n represents the total number of samples, Represents the weight update result of the kth part at time t+1.

[0094] The large language model in step 4 specifically includes:

[0095] Step 4.1, set the root mean square layer normalization (RMSNorm) in the large language model structure, use the root mean square layer normalization (RMSNorm) in the large language model structure to input the geoscientific text dataset generated in step 3, calculate the root mean square value (RMS) of all elements of each input sample, and then normalize it to eliminate the scale difference of the input data. The root mean square value (RMS) formula is as follows:

[0096]

[0097] Among them, x i represents the value of the i-th data point, and N represents the total number of data points;

[0098] Step 4.2, set the SwiGLU activation function. The SwiGLU activation function realizes dynamic information selection through the gating mechanism, improves the feature selection ability and performance of the model, and the SwiGLU activation function formula is as follows:

[0099]

[0100] Among them, x represents the input vector, W represents the weight matrix, and b represents the bias term;

[0101] Step 4.3, use rotation position embedding (RoPE) to set the position embedding in Transformer. This position embedding method encodes the absolute position with a rotation matrix. First, initialize the rotation angle, create a rotation matrix, and then rotate the matrix. The angle calculation and rotation vector formula of rotation position embedding (RoPE) are as follows:

[0102] Angle calculation:

[0103] Rotate vector:

[0104] Among them, θ i represents the rotation angle of the i-th dimension, which is used to define the rotation matrix; k is a hyperparameter used to control the size of the rotation angle; d represents the dimension of the vector;

[0105] Step 4.4, set up global query attention (GQA), first project the input sequence into query, key, and value vectors, then reduce the number of query vectors to one or several. According to your own model, you can start with a smaller number of query vectors and gradually increase it to observe the changes in model performance until you find a suitable balance point. Then use the shared query vector to perform dot product calculations with all key vectors to obtain the attention score. Finally, use the attention score to perform weighted summation on the value vector to obtain the output vector.

[0106] Reference Figure 3 , step 5, input the geoscience question-answering dataset obtained in step 3 into the geoscience large language model (GSLLM) obtained in step 4 for fine-tuning, and then set the enhanced low-rank adaptive technology, that is, by decomposing multiple low-rank matrices and assigning different learning rates to optimize parameter adjustment, and finally add the DeepSpeed ​​acceleration framework to the geoscience large language model (GSLLM) to improve computing efficiency and model performance, so as to achieve efficient processing and accurate answers to geoscience question-answering data; generate a large model of geoscience language question-answering;

[0107] The specific process of fine-tuning the Geographic Sciences Large Language Model (GSLLM) in step 5 is as follows:

[0108] Step 5.1, when fine-tuning, perform low-rank matrix decomposition on the weights of the Geographic Sciences Largelanguage Model (GSLLM) and decompose them into multiple low-rank matrices; specifically, decompose the original weight matrix W into low-rank matrices A and B, that is, W≈A*B, but unlike the original low-rank principle, the present invention combines multiple low-rank matrices A and B to form a hierarchical structure, and each layer of matrices has a different learning rate (lr-1, lr-2,..., lr-n) to adapt to the characteristics and training requirements of each matrix. For example, the first layer of low-rank matrix A and low-rank matrix B are respectively set to a learning rate of lr-1, and for the next layer of low-rank matrix, the learning rate is set to lr-2, and so on, until the learning rates of all low-rank matrices are set, set the learning rates of all low-rank matrices, and then use the adaptive optimization algorithm to dynamically adjust the learning rate, and adaptively adjust the parameter update step size according to the feedback during the training process;

[0109] Step 5.2, during the training process, the present invention selects to add the DeepSpeed ​​acceleration framework to the Geographic Sciences Largelanguage Model (GSLLM), which reduces memory consumption by using the ZeRO (Zero Redundancy Optimizer) technology of the DeepSpeed ​​acceleration framework to perform distributed data parallel optimization. Specifically, stage 0 in zero_optimization is used in this invention; and finally a large model of geography language question answering based on enhanced low-rank adaptation is generated.

[0110] Reference Figure 4 , Step 6, perform score evaluation on the geoscience language question answering model generated by training in step 5, set four score evaluation methods, namely BLEU4, ROUGE-1, ROUGE-2 and ROUGE-L, use the scores to continuously optimize the geoscience language question answering model, and save the weight file of the best score that the geoscience language question answering model can achieve in the evaluation.

[0111] The specific method of step 6 is:

[0112] Step 6.1, after the training is completed, the score of the geo-language question-answering model generated in step 5 is evaluated. This invention uses four evaluation methods, BLEU4, ROUGE-1, ROUGE-2 and ROUGE-L, to evaluate the model score; the formulas of BLEU4, ROUGE-1, ROUGE-2 and ROUGE-L are as follows:

[0113]

[0114] Among them, exp represents the natural exponential function, w represents the weight, BP (Brevity Penalty): penalizes translations that are too short, and the calculation formula is:

[0115]

[0116] Where c represents the number of n-grams in the candidate translation (machine translation) and r represents the number of n-grams in the reference translation.

[0117]

[0118] Among them, max means taking the maximum value, count means counting the number, gram: represents an n-gram, that is, a sequence of n consecutive words in the text, Candidate represents machine-generated text, Reference represents artificially generated text or known high-quality text, ROUGE-1 calculates the overlap of a single word, and ROUGE-2 calculates the overlap of two consecutive words.

[0119]

[0120] The score is 6.2. Based on the evaluation score obtained in 6.1, we change the hyperparameters of the large model input for geoscience language question answering to continuously optimize the weight file of the training model, and finally retain the weight file with the best score that can be achieved.

[0121] The effect of the present invention is further described below in conjunction with simulation experiments.

[0122] 1. Simulation experiment conditions:

[0123] The hardware test platform of the simulation experiment of the present invention is: CPU 48 cores 192G; graphics card is Ascend 910Premium A32G

[0124] The software platform of the simulation experiment of the present invention is: operating system: EulerOS2.0 (SP8); online platform: ModelArts; notebook integrated development environment, Python 3.10.14; hardware platform: Ascend 910A

[0125] 2. Simulation content and result analysis:

[0126] Table 1 provides detailed information on the geoscience dataset;

[0127] Table 2 is a comparison of training evaluation index scores.

[0128] The simulation experiment of the present invention is conducted on a self-collected geoscience data set, which makes up for the lack of question-answering data sets in the geoscience field. The detailed parameters of the data used in the present invention are shown in Table 1.

[0129] Table 1 Detailed information of geoscience datasets

[0130]

[0131]

[0132] The simulation experiment of the present invention is conducted on a self-created data set, and the evaluation scores of the original model are compared in Table 2. It can be seen from the experiment that the effect of the model of the present invention is significantly better than that of the original model.

[0133] Table 2 Comparison of training evaluation index scores of the present invention

[0134]

[0135] The final results show that the score of the present invention is 89.71 on BLEU-4, 91.26 on ROUGE-1, 89.14 on ROUGE-2, and 90.83 on ROUGE-L.

[0136] The above data show that the present invention not only saves training costs, but also ensures that the geoscience language question-answering model has a good effect on the current mainstream evaluation indicators.

[0137] The present invention also provides a geoscience language question-answering large model construction system based on enhanced low-rank adaptation, comprising:

[0138] A geoscience data collection module is used to collect geoscience data in step 1, including but not limited to China Geography Network, geoscience professional books and geoscience related scientific research papers;

[0139] A geoscience data preprocessing module is used to implement the preprocessing of the geoscience data collected in step 1 in step 2 to obtain a geoscience data set;

[0140] A geoscience data set classification module is used to implement the classification of the geoscience data set obtained in step 2 in step 3 to obtain a geoscience text data set and a geoscience question-answering data set;

[0141] A Geographic Sciences Large Language Model (GSLLM) generation module is used to implement the input of the geological text dataset obtained in step 3 into the corresponding large language model in step 4, and perform self-supervised training through federated learning. Using the homogeneous architecture of federated learning, the large language model is distributedly trained on multiple devices, and each device processes its local data independently. After all training rounds (epochs) are completed, each device sends the parameters of the large language model back to the central server, and the central server aggregates the parameters of the large language model to generate a Geographic Sciences Large Language Model (GSLLM);

[0142] A large model generation module for geography language question answering is used to implement the step 5 of inputting the geography question answering dataset obtained in step 3 into the geography language model (GSLLM) obtained in step 4 for fine-tuning, and then setting an enhanced low-rank adaptive technology, that is, by decomposing multiple low-rank matrices and assigning different learning rates to optimize parameter adjustment, and finally adding the DeepSpeed ​​acceleration framework to the geography language model (GSLLM) to generate a large model for geography language question answering;

[0143] The best weight file acquisition module is used to implement the score evaluation of the geoscience language question and answer big model generated by step 5 training in step 6, use the score to continuously optimize the geoscience language question and answer big model, and save the weight file of the best score that the geoscience language question and answer big model can achieve in the evaluation.

[0144] The present invention also provides a device for constructing a large model of geoscience language question answering based on enhanced low-rank adaptation, comprising:

[0145] Memory: a computer program storing the above-mentioned method for building a large model of geoscience language question answering based on enhanced low-rank adaptation, which is a computer-readable device;

[0146] Processor: used to implement the method for constructing a large model of geoscience language question and answer based on enhanced low-rank adaptation when executing the computer program.

[0147] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the method for constructing a large model of geological language question and answer based on enhanced low-rank adaptation.

Claims

1. A method for constructing a large model of geoscience language question answering based on enhanced low-rank adaptation, characterized in that: include: Step 1: Collect geoscience data, including but not limited to China Geography Network, geoscience books and geoscience-related scientific research papers; Step 2, preprocessing the geoscience data collected in step 1 to obtain a geoscience data set; Step 3, classify the geoscience dataset obtained in step 2 to obtain a geoscience text dataset and a geoscience question-answering dataset; Step 4: Input the geoscience text dataset obtained in step 3 into the corresponding large language model, and perform self-supervised training through federated learning. Using the homogeneous architecture of federated learning, the large language model is distributedly trained on multiple devices, and each device processes its local data independently. After all training rounds (epochs), each device sends the parameters of the large language model back to the central server, and the central server aggregates the parameters of the large language model to generate a Geographic Sciences Large Language Model (GSLLM). Step 5: Input the geoscience question-answering dataset obtained in step 3 into the Geographic Sciences Large Language Model (GSLLM) obtained in step 4 for fine-tuning, and then set the enhanced low-rank adaptive technology, that is, by decomposing multiple low-rank matrices and assigning different learning rates to optimize parameter adjustment, and finally add the DeepSpeed ​​acceleration framework to the Geographic Sciences Large Language Model (GSLLM) to generate a large model of geoscience language question-answering; Step 6, perform score evaluation on the geoscience language question and answer big model generated by the training in step 5, use the score to continuously optimize the geoscience language question and answer big model, and save the weight file of the best score that the geoscience language question and answer big model can achieve in the evaluation.

2. According to claim 1, a method for constructing a large model of geoscience language question answering based on enhanced low-rank adaptation is characterized in that: The specific method of step 2 is: Step 2.1, filter and screen the geoscience data collected in step 1, and use the characteristics of geoscience knowledge as feature words to assist crawler technology in crawling relevant local knowledge; Step 2.2, using multi-threading technology and asynchronous processing methods, so that the crawler technology used in step 2.1 crawls multiple targets; Step 2.3, using data cleaning and format conversion tools, remove the non-geological parts of the data obtained in step 2.2, and format the data obtained in step 2.2; Step 2.4, review and correct, correct the erroneous knowledge in the data obtained in step 2.3, update the outdated knowledge points, and perform multiple data verification and comparative analysis on the data to obtain a geoscience data set.

3. The method for constructing a large geoscience language question-answering model based on enhanced low-rank adaptation according to claim 1 is characterized in that: The specific method of step 3 is: Step 3.1, classify the geoscience data set obtained in step 2, convert the geoscience data collected from the China Geography Network and geoscience related scientific research papers in the geoscience data set obtained in step 2 into the format of the geoscience text data set used for pre-training, and obtain the geoscience text data set; Step 3.2: convert the geoscience data collected from geoscience professional books in the geoscience dataset obtained in step 2 into the format of the geoscience question-answering dataset used for fine-tuning, thereby obtaining the geoscience question-answering dataset.

4. The method for constructing a large geoscience language question-answering model based on enhanced low-rank adaptation according to claim 1 is characterized in that: The large language model in step 4 uses the Transformer Decoder-only network architecture, inputs the geoscientific text dataset obtained in step 3 into the large language model, and adds word embedding. The word embedding converts the words or tags in the text into vector representations of fixed dimensions. After the word embedding, the rotation position embedding (RoPE) is used to provide the position information of each word in the sequence without adding additional parameters. The large language model adopts a global query attention (GQA), and uses the SwiGLU activation function, and uses the root mean square layer normalization (RMSNorm) to normalize the root mean square of each sample in the large language model input tensor. The weighted average update formula is: Among them, w t+1 represents the overall weight update result at time t+1, K represents a total of K parts, n k represents the number of samples in the kth part, n represents the total number of samples, Represents the weight update result of the kth part at time t+1.

5. The method for constructing a large geoscience language question-answering model based on enhanced low-rank adaptation according to claim 1 or 4, characterized in that: The large language model in step 4 specifically includes: Step 4.1, set the root mean square layer normalization (RMSNorm) in the large language model structure, use the root mean square layer normalization (RMSNorm) in the large language model structure to input the geoscientific text dataset generated in step 3, calculate the root mean square value (RMS) of all elements of each input sample, and then normalize it. The root mean square value (RMS) formula is as follows: Among them, x i represents the value of the i-th data point, and N represents the total number of data points; Step 4.2, set the SwiGLU activation function. The SwiGLU activation function formula is as follows: Among them, x represents the input vector, W represents the weight matrix, and b represents the bias term; Step 4.3, use rotation position embedding (RoPE) to set the position embedding in Transformer. This position embedding method encodes the absolute position with a rotation matrix. First, initialize the rotation angle, create a rotation matrix, and then rotate the matrix. The angle calculation and rotation vector formula of rotation position embedding (RoPE) are as follows: Angle calculation: Rotate vector: Among them, θ i represents the rotation angle of the i-th dimension, which is used to define the rotation matrix; k is a hyperparameter used to control the size of the rotation angle; d represents the dimension of the vector; Step 4.4, set up global query attention (GQA), first project the input sequence into query, key, and value vectors, then reduce the number of query vectors to one or several. According to your own model, you can start with a smaller number of query vectors and gradually increase it to observe the changes in model performance until you find a suitable balance point. Then use the shared query vector to perform dot product calculations with all key vectors to obtain the attention score. Finally, use the attention score to perform weighted summation on the value vector to obtain the output vector.

6. The method for constructing a large geoscience language question-answering model based on enhanced low-rank adaptation according to claim 1, characterized in that: The specific process of fine-tuning the Geographic Sciences Large Language Model (GSLLM) in step 5 is as follows: Step 5.1, during fine-tuning, perform low-rank matrix decomposition on the weights of the Geographic Sciences Large language Model (GSLLM) into multiple low-rank matrices; decompose the original weight matrix W into low-rank matrices A and B, that is, W≈A*B, and combine multiple low-rank matrices A and B to form a hierarchical structure. Each layer of the matrix has a different learning rate (lr-1, lr-2,..., lr-n). Set the learning rate of all low-rank matrices, and then use the adaptive optimization algorithm to dynamically adjust the learning rate. Adaptively adjust the parameter update step size according to the feedback during the training process; Step 5.2: During the training process, the Geographic Sciences Large Language Model (GSLLM) is selected to join the DeepSpeed ​​acceleration framework, and finally a large model of Geographic Sciences language question and answering based on enhanced low-rank adaptation is generated.

7. The method for constructing a large geoscience language question-answering model based on enhanced low-rank adaptation according to claim 1, characterized in that: The specific method of step 6 is: Step 6.1: Evaluate the scores of the geoscience language question answering model generated in step 5. Four evaluation methods, BLEU4, ROUGE-1, ROUGE-2, and ROUGE-L, are used to evaluate the scores of the geoscience language question answering model. The formulas for BLEU4, ROUGE-1, ROUGE-2, and ROUGE-L are as follows: Among them, exp represents the natural exponential function, w represents the weight, BP (Brevity Penalty): penalizes translations that are too short, and the calculation formula is: Where c represents the number of n-grams in the candidate translation (machine translation); r represents the number of n-grams in the reference translation; Among them, max means taking the maximum value, count means counting the number, gram: represents an n-gram, that is, a sequence of n consecutive words in the text, Candidate represents machine-generated text, Reference represents artificially generated text or known high-quality text, ROUGE-1 calculates the overlap of a single word, and ROUGE-2 calculates the overlap of two consecutive words; The score is 6.

2. Based on the evaluation score obtained in 6.1, the hyperparameters of the large model input for geoscience language question answering are changed, and the weight file of the training model is continuously optimized. Finally, the weight file with the best score is retained.

8. A system for building a large model of geoscience language question answering based on enhanced low-rank adaptation based on the method according to any one of claims 1 to 7, characterized in that: include: The geoscience data collection module is used to collect geoscience data, including but not limited to China Geography Network, geoscience books and geoscience-related scientific research papers; A geoscience data preprocessing module is used to preprocess the collected geoscience data to obtain a geoscience data set; The geoscience dataset classification module is used to classify the geoscience dataset to obtain the geoscience text dataset and the geoscience question-answering dataset; The Geographic Sciences Large Language Model (GSLLM) generation module is used to input the geoscience text dataset into the corresponding large language model, and perform self-supervised training through federated learning. Using the homogeneous architecture of federated learning, the large language model is distributedly trained on multiple devices, and each device processes its local data independently. After all training rounds (epochs), each device sends the parameters of the large language model back to the central server, and the central server aggregates the parameters of the large language model to generate the Geographic Sciences Large Language Model (GSLLM); The module for generating a large model for geography language question answering is used to input the geography question answering dataset into the Geographic Sciences Large Language Model (GSLLM) for fine-tuning, and then set up enhanced low-rank adaptation technology, that is, by decomposing multiple low-rank matrices and assigning different learning rates to optimize parameter adjustment, and finally adding the DeepSpeed ​​acceleration framework to the Geographic Sciences Large Language Model (GSLLM) to generate a large model for geography language question answering; The best weight file acquisition module is used to evaluate the scores of the geoscience language question and answer big model generated by training, use the scores to continuously optimize the geoscience language question and answer big model, and save the weight file of the best score that the geoscience language question and answer big model can achieve in the evaluation.

9. A device for constructing a large model of geoscience language question answering based on enhanced low-rank adaptation, characterized in that: include: Memory: a computer program storing a method for constructing a large model of geoscience language question answering based on enhanced low-rank adaptation as described in any one of claims 1 to 7, which is a computer-readable device; Processor: used to implement the method for constructing a large model of geoscience language question and answer based on enhanced low-rank adaptation as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, can implement a method for constructing a large model of geoscience language question and answer based on enhanced low-rank adaptation as described in any one of claims 1 to 7.

Images