Vertical large language model training method and system in carbon neutralization field

Abstract

The invention discloses a vertical large language model training method and system in the carbon neutralization field, and the method comprises the following steps: collecting data of the carbon neutralization field, carrying out the data preprocessing, constructing a carbon neutralization field knowledge base, and updating the carbon neutralization field knowledge base through a dynamic updating mechanism; performing dynamic semantic partitioning and vectorization coding on the text of the carbon neutralization domain knowledge base, and storing the text into a vector database; performing staged fine tuning on the pre-trained large language model based on a low-rank adaptation technology, wherein the fine tuning comprises general instruction fine tuning and carbon neutralization field professional fine tuning; a retrieval enhancement generation mechanism is adopted, knowledge fragments related to user query are retrieved through a vector database, and a large language model is input to generate answers. Compared with the prior art, the method has the advantages that the answer reliability is improved through conflict detection and source tracing, so that the large language model can more accurately adapt to knowledge requirements in the carbon neutralization field.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence and natural language processing technology, and more specifically, to a vertical large language model training method and system in the field of carbon neutrality. Background Art

[0002] As global climate change becomes increasingly severe, achieving carbon neutrality has become a core goal for governments, research institutions, and businesses worldwide. The concept of carbon neutrality encompasses a range of technical measures and policy strategies aimed at reducing greenhouse gas emissions and achieving net-zero emissions through various means, including carbon capture, utilization, and storage (CCUS), carbon trading markets, carbon footprint assessments, and low-carbon technology research and development. This field involves multiple interdisciplinary fields, including but not limited to energy science, environmental engineering, climate economics, and policies and regulations. Due to the inherent complexity and specialization of carbon neutrality research, the efficient integration and application of interdisciplinary knowledge to assist in practical issues such as carbon management, policy decision-making, and carbon emission optimization has become a major challenge in current technological development.

[0003] In recent years, artificial intelligence (AI) technology, especially large language models (LLMs) based on deep learning, has made breakthrough progress in the field of natural language processing (NLP). Large language models, with their excellent text understanding and generation capabilities, are widely used in tasks such as intelligent question answering, text summarization, machine translation, and content generation. Mainstream large language models, such as GPT-4 developed by OpenAI, Google's Gemini, and Meta's LLaMA series, are all trained based on large-scale Internet corpora. They have strong generalization capabilities and can demonstrate reasoning and language expression capabilities close to humans in multiple tasks. However, although these general large models perform relatively well in open fields, their application in a multidisciplinary vertical field such as carbon neutrality still faces many challenges. Since the training data for general large models mainly comes from social media, news, Wikipedia, open source papers, etc., and the professional knowledge related to carbon neutrality is mostly contained in closed research papers, policy documents, industry reports, regulatory standards and other documents, general large models often face problems such as lack of professional knowledge, inaccurate conceptual understanding, and limited reasoning ability when handling carbon neutrality-related tasks. In addition, existing large language models are prone to hallucination when generating answers, that is, generating seemingly reasonable but actually wrong information in the absence of authoritative knowledge support, affecting the accuracy and credibility of decision-making. Therefore, how to optimize large language models for the specific field of carbon neutrality so that they have stronger professional knowledge understanding capabilities, enhance their reasoning accuracy, and improve the reliability of output content has become a technical problem that needs to be solved urgently.

[0004] Existing technologies for intelligent applications in the field of carbon neutrality rely primarily on two technical approaches: one utilizes general-purpose large language models for task optimization, and the other relies on rule-matching or knowledge-based retrieval and question-answering systems. In the first approach, existing general-purpose large language models such as GPT-4, Gemini, and LLaMA are widely used for text processing and intelligent question answering. However, due to their lack of domain-specific fine-tuning, they have limited expertise in areas such as carbon management, carbon trading markets, and negative emissions technologies, and often provide inaccurate or incomplete answers to complex carbon neutrality questions. Furthermore, general-purpose large models often lack access to dynamic information such as the latest carbon emission regulations, policy updates, and corporate carbon accounting standards, limiting their practical application. The second approach involves knowledge-based question-answering systems, such as those that utilize rule matching, knowledge graphs, and database retrieval to provide standardized carbon neutrality information query services. While these approaches offer advantages in terms of authoritative data and high traceability, they also suffer from limited flexibility, difficulty handling open-ended question-answering scenarios, and difficulty performing complex reasoning. For example, FAQ systems based on rule matching can only answer predefined questions. While knowledge graph-based approaches can perform semantic reasoning to a certain extent, they are expensive to update, difficult to maintain, and struggle to adapt to the rapid changes in new policies and regulations. Furthermore, constructing knowledge graphs requires extensive manual annotation, hindering large-scale automated expansion.

[0005] To address these issues, Retrieval-Augmented Generation (RAG) technology has emerged in recent years as an important tool for enhancing the expertise of large language models. RAG combines the capabilities of traditional knowledge retrieval and generative large language models. After a user enters a question, vector retrieval methods are used to retrieve the most relevant text fragments. The retrieved text is then fed into the large model as additional context to generate an answer, thereby improving the accuracy and expertise of the response. Compared to purely generative models, this approach offers greater flexibility while providing more reliable knowledge support. However, existing RAG methods still have limitations. For one thing, the quality of retrieved text is difficult to guarantee and may contain redundant, contradictory, or erroneous information, affecting the model's final output. Furthermore, when processing large knowledge bases, existing vector retrieval methods (such as FAISS) still have room for improvement in balancing query efficiency and accuracy. Ensuring efficient retrieval remains an unresolved issue, particularly in scenarios requiring real-time access to large volumes of literature and policy documents. In addition, when the retrieved external knowledge conflicts with the model's existing internal knowledge, how to fuse the information and resolve the potential contradictions is also a major challenge when the current RAG technology is applied to high-reliability tasks.

[0006] In order to further improve the adaptability of large language models in the field of carbon neutrality, researchers have tried to use parameter-efficient fine-tuning (PEFT) technology to optimize the model in recent years to make it more accurate in specific fields. Full parameter fine-tuning requires adjusting all parameters of the model, which consumes a lot of computing resources and is difficult to promote on a large scale in practical applications. Low-Rank Adaptation (LoRA), as an optimization solution for PEFT, adds low-rank parameters to the weight matrix of the existing model, allowing the model to learn new domain knowledge without affecting the original parameters. This method has achieved good results in professional fields such as medicine, law, and finance, but its application in the field of carbon neutrality is still relatively small, especially in how to efficiently construct high-quality training data sets and how to combine retrieval enhancement methods to supplement knowledge. There is still a lot of room for optimization.

[0007] Existing technologies still have limitations in carbon neutrality intelligent question-answering, knowledge management, and policy analysis. These limitations are primarily reflected in the insufficient coverage of specialized knowledge within general-purpose large language models, the limited flexibility of knowledge-based question-answering systems, room for improvement in data quality control and retrieval efficiency within existing RAG solutions, and the need for further improvement in cross-disciplinary knowledge integration using existing fine-tuning techniques. Therefore, a new technical solution is urgently needed that combines methods such as retrieval-enhanced generation and efficient parameter fine-tuning to optimize the knowledge acquisition capabilities of large language models in the carbon neutrality domain, improve their reasoning accuracy, and enhance their applicability to tasks such as intelligent question-answering, policy interpretation, and carbon emission management.

[0008] To this end, based on the above requirements and the shortcomings of the existing technology, this application proposes a vertical large language model training method and system in the field of carbon neutrality. Summary of the Invention

[0009] The present invention provides a vertical large language model training method and system in the field of carbon neutrality. By constructing a high-quality knowledge database, optimizing the semantic retrieval mechanism, improving the fine-tuning strategy, and improving the reliability of answers through conflict detection and source tracing, the large language model can more accurately adapt to the knowledge needs in the field of carbon neutrality, and improve its reliability and interpretability in applications such as intelligent question and answer, carbon emission calculation, and carbon trading market forecasting.

[0010] The primary purpose of the present invention is to solve the above technical problems, and the technical solutions of the present invention are as follows: A first aspect of the present invention provides a vertical large language model training method in the field of carbon neutrality, the method comprising the following steps: S1. Collect data in the field of carbon neutrality and perform data preprocessing, build a knowledge base in the field of carbon neutrality, and use a dynamic update mechanism to update the knowledge base in the field of carbon neutrality.

[0011] S2. Dynamically semantically segment and vectorize the text of the carbon neutrality domain knowledge base and store it in a vector database.

[0012] S3. Fine-tune the pre-trained large language model in stages based on low-rank adaptation technology, including general instruction fine-tuning and professional fine-tuning in the field of carbon neutrality.

[0013] S4. Adopting the retrieval enhancement generation mechanism, the knowledge fragments related to the user query are retrieved through the vector database, and the large language model is input to generate the answer.

[0014] Furthermore, the data preprocessing in step S1 includes the following steps: removing noise data from unstructured text, and using NLP semantic similarity calculation to remove redundant text; converting the text format into standardized text data, and combining OCR technology to perform high-precision recognition of scanned research documents; storing all data in a structured manner and adding summary labels to the data.

[0015] Furthermore, the specific process of dynamic semantic segmentation and vectorized encoding in step S2 is as follows: segment the document according to paragraph delimiters, and further segment paragraphs with more than 800 characters based on punctuation using regular expressions, while introducing an overlapping area of ​​150 characters in the segmentation process; generate sentence vectors for the segmented text using a lightweight cross-language semantic embedding model, implement semantic retrieval through the FAISS database, and standardize the generated vectors.

[0016] Furthermore, in the process of fine-tuning the pre-trained large language model in stages in step S3, the specific steps of fine-tuning the general instructions are: using a large-scale general instruction dataset to train the overall large language model, and the large-scale general instruction dataset includes: general question-answering instructions in the directions of knowledge question-answering, reading comprehension, logical reasoning and code generation.

[0017] Furthermore, the specific steps of the professional fine-tuning in the field of carbon neutrality are: inserting a trainable low-rank matrix adaptation layer into the attention layer of the large language model, and using professional data sets to train the low-rank matrix adaptation layer based on the general instruction fine-tuning. The professional data sets include: professional Q&A data, policy and regulation interpretation Q&A, carbon emission calculation and modeling Q&A, and corporate sustainability reports and carbon accounting data.

[0018] Furthermore, after completing the general instruction fine-tuning and carbon neutrality field professional fine-tuning processes, a hybrid data enhancement training process is required. The large-scale general instruction dataset and professional dataset are mixed and the low-rank matrix adaptation layer is trained using synthetic question-answer pairs, knowledge distillation and adversarial training.

[0019] Furthermore, the retrieval enhancement generation mechanism described in step S4 adopts a multi-index weighted fusion retrieval method to perform vector database retrieval. When the user enters a query question, the natural language question entered by the user is first converted into a high-dimensional semantic vector, and then searched in the vector database to obtain the knowledge fragments most relevant to the query; after receiving the knowledge fragments returned by the preliminary retrieval, they are sorted according to semantic similarity, and then logical reasoning is performed using the natural language inference model to calculate the implication probability between the query text and the candidate knowledge fragments, and the top N high-quality knowledge fragments are screened out; after the knowledge retrieval and sorting are completed, the highly relevant knowledge fragments are spliced ​​and integrated with the user's original query, and input into the large language model to generate the final answer.

[0020] Furthermore, the specific process of the multi-index weighted fusion retrieval is: converting the user query text into a vector and normalizing it, using the FAISS vector database to perform similarity calculation, and searching for the knowledge fragment that best matches the query semantics in the core document index and the additional document index respectively.

[0021] The second aspect of the present invention provides a vertical large language model training system in the field of carbon neutrality. The system is used for the vertical large language model training method in the field of carbon neutrality, and includes: a carbon neutrality field data acquisition module, a vectorized encoding module, a low-rank adaptation module and a retrieval enhancement module.

[0022] The carbon neutrality field data acquisition module collects data in the carbon neutrality field and performs data preprocessing, constructs a carbon neutrality field knowledge base, and uses a dynamic update mechanism to update the carbon neutrality field knowledge base; the vectorization encoding module dynamically semantically segments and vectorizes the text of the carbon neutrality field knowledge base and stores it in the vector database; the low-rank adaptation module fine-tunes the pre-trained large language model in stages based on low-rank adaptation technology, including general instruction fine-tuning and carbon neutrality field professional fine-tuning; the retrieval enhancement module uses a retrieval enhancement generation mechanism to retrieve knowledge fragments related to user queries through the vector database, and inputs them into the large language model to generate answers.

[0023] Compared with the prior art, the beneficial effects of the technical solution of the present invention are: The present invention provides a vertical large language model training method and system in the field of carbon neutrality. By constructing a knowledge base in the field of carbon neutrality, it can significantly improve the professionalism and accuracy of the model in tasks such as carbon emission accounting and policy interpretation; LoRA phased fine-tuning is used to achieve domain adaptation at low cost to avoid overfitting; weighted retrieval and logical reasoning ensure that high-authority knowledge is prioritized, reduce noise interference, and improve the professionalism and credibility of answers; dynamic mechanisms ensure the real-time and consistency of the knowledge base, and adapt to the actual needs of frequent policy changes in the field of carbon neutrality; hybrid data enhances the model's cross-domain generalization ability, and at the same time deeply adapts to carbon neutrality sub-domain tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a flowchart of a vertical large language model training method in the field of carbon neutrality according to the present invention.

[0025] Figure 2 Schematic diagram of the semantically enhanced retrieval and ranking mechanism in one embodiment of the present invention.

[0026] Figure 3 This is a schematic diagram of a vertical large language model training system in the field of carbon neutrality in the present invention. DETAILED DESCRIPTION

[0027] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features therein can be combined with each other.

[0028] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0029] Example 1 like Figure 1 As shown, the present invention provides a vertical large language model training method in the field of carbon neutrality, and the method includes the following steps: S1. Collect data in the field of carbon neutrality and perform data preprocessing, build a knowledge base in the field of carbon neutrality, and use a dynamic update mechanism to update the knowledge base in the field of carbon neutrality.

[0030] S2. Dynamically semantically segment and vectorize the text of the carbon neutrality domain knowledge base and store it in a vector database.

[0031] S3. Fine-tune the pre-trained large language model in stages based on low-rank adaptation technology, including general instruction fine-tuning and professional fine-tuning in the field of carbon neutrality.

[0032] S4. Adopting the retrieval enhancement generation mechanism, the knowledge fragments related to the user query are retrieved through the vector database, and the large language model is input to generate the answer.

[0033] The data preprocessing described in step S1 includes the following steps: removing noise data from unstructured text and using NLP semantic similarity calculation to remove redundant text; converting the text format into standardized text data and combining OCR technology to perform high-precision recognition of scanned research documents; storing all data in a structured manner and adding summary labels to the data.

[0034] In a specific embodiment, as shown in Table 1, the data sources include but are not limited to: (1) national and regional policy documents on carbon emissions, such as the EU ETS regulations, the Paris Agreement text, and national carbon peak and carbon neutrality action plans; (2) the climate assessment report released by the Intergovernmental Panel on Climate Change (IPCC), covering global carbon reduction targets, climate change impacts and response strategies; (3) several carbon neutrality-related research papers from academic databases such as China National Knowledge Infrastructure (CNKI) and Web of Science (WoS), covering carbon capture, carbon trading, carbon accounting, CCUS engineering practice, negative emission technology, etc.; (4) corporate ESG (environment, society, governance) reports and carbon emission annual reports, providing global corporate carbon emissions, carbon neutrality commitments and low-carbon technology applications; (5) patent databases, including technical patents in the fields of CCUS, carbon sequestration, carbon footprint calculation models, carbon credit systems, etc.; (6) a database containing approximately 60,000 (7) General instruction fine-tuning data (GPT4-Alpaca 50,000 data) to ensure that the model has basic language understanding capabilities to avoid professional fine-tuning causing a decrease in the model's generalization ability; (8) Professional instruction fine-tuning data (UltraChat 5,382 carbon neutrality-related questions and answers) to improve the model's accuracy in tasks such as carbon emission management, carbon credit trading, and policy interpretation.

[0035] Table 1 Knowledge base data sources and quantity statistics

[0036] In addition, all text data must go through the following key steps before entering the knowledge base: (1) Data cleaning: remove noise data, irrelevant content, page numbers, special characters, etc. in unstructured text, and use NLP semantic similarity calculation to remove redundant text to prevent duplicate data from affecting model learning; (2) Format conversion: convert HTML, PDF, scanned documents and other formats into standardized text data, and combine OCR technology to perform high-precision recognition of scanned research documents to ensure text integrity; (3) Hierarchical classification: all data are structured and stored according to the macro policy layer (national regulations, carbon market trading rules, international agreements), technical solution layer (CCUS, negative emission technology, renewable energy), and market economy layer (carbon credit system, carbon accounting standards, carbon trading market), so that the knowledge retrieval system can quickly locate the required information and improve the RAG recall rate; (4) Semantic label annotation: use the BERT-based NLP model to automatically add labels to the data, such as [carbon market] [carbon emissions] [carbon trading policy] [carbon capture technology], so as to improve the model's context matching ability in the question and answer generation process.

[0037] In terms of knowledge base updates, in order to cope with the rapid changes in policies, regulations, and technical research in the field of carbon neutrality, the present invention designs a dynamic knowledge update mechanism to ensure that the model always performs question-answering reasoning based on the latest industry information. This mechanism includes: (1) Periodic knowledge synchronization: The system can automatically crawl the latest policies, regulations, industry reports, and research papers every 6 months or 1 year, and automatically update the database through timestamp analysis; (2) Knowledge failure detection: Using NLP to compare the provisions of new and old regulations, automatically detect and mark policies that have been abolished or adjusted, such as the annual revision of carbon emission standards; (3) Expert review mechanism: For key policy and regulatory data, a manual review link is introduced to ensure the accuracy of data such as legal provisions and carbon market regulatory rules; (4) Semantic fusion algorithm: When there is a knowledge conflict between new and old regulations, the system uses a Transformer-based weight adjustment mechanism to automatically optimize the knowledge authority to ensure that the content returned by the question-answering system always complies with the latest policies.

[0038] The specific process of dynamic semantic segmentation and vectorized encoding described in step S2 is as follows: the document is segmented according to paragraph delimiters, and paragraphs with more than 800 characters are further segmented based on punctuation using regular expressions. At the same time, an overlapping area of ​​150 characters is introduced in the segmentation process; a lightweight cross-language semantic embedding model is used to generate sentence vectors for the segmented text, and semantic retrieval is implemented through the FAISS database, and the generated vectors are normalized.

[0039] Based on these technical features, text data is converted into high-dimensional semantic vectors, enabling efficient query and knowledge retrieval based on semantic similarity. This paper also optimizes the dynamic text segmentation strategy, vectorization method, index construction process, and multi-index weighted fusion retrieval mechanism to ensure high accuracy, stability, and efficiency in knowledge base retrieval results.

[0040] In a specific embodiment, during the text vectorization process, the present invention adopts a lightweight, cross-domain compatible semantic embedding model, namely paraphrase-multilingual-MiniLM-L12-v2, to generate high-dimensional semantic vectors for the segmented text segments. This model has been optimized for multiple languages ​​and has the advantages of low computational cost and fast retrieval speed, making it suitable for large-scale semantic retrieval tasks in the field of carbon neutrality. The text blocks are batch-inputted into paraphrase-multilingual-MiniLM-L12-v2 for encoding, and the generated embedding vectors are standardized using L2 normalization to improve the stability of similarity calculation in vector retrieval. Compared with traditional keyword matching-based retrieval methods, semantic embedding technology can capture deeper semantic associations and is particularly suitable for interdisciplinary knowledge retrieval needs in the field of carbon neutrality, such as the association analysis between carbon capture and use (CCUS) and carbon trading markets. In the process of storing embedded vectors, the present invention uses the Facebook AI Similarity Search (FAISS) vector database to support large-scale text semantic indexing and efficient retrieval. All generated embedding vectors are stored in the database according to the following process: Index initialization: The system first creates a FAISS index structure based on the inner product index (IndexFlatIP) to support efficient similarity calculation. The inner product index method has higher computational efficiency than the L2 Euclidean distance index on large corpora.

[0041] Incremental storage: The present invention adopts an incremental index construction strategy, that is, when new document data is imported, the vectorization module will automatically expand the existing index database to ensure that the knowledge base can be continuously updated without the need to rebuild the index each time.

[0042] Text index mapping: The system stores the mapping relationship between text blocks and embedded vectors in the FAISS vector database. That is, after retrieval, the system can quickly find the corresponding original text from the returned vector index to ensure data traceability.

[0043] The construction of the FAISS vector index library enables the model to perform real-time semantic matching in large-scale knowledge bases. Compared with traditional retrieval methods based on rule matching or SQL structured databases, this method has faster search speeds and stronger cross-semantic reasoning capabilities, and can support complex query tasks such as "The impact of carbon emission pricing on carbon market fluctuations" or "Evaluating the economic feasibility of negative emission technologies." When a user submits a query, the system will perform the following operations: Query vectorization: The user input text is converted into semantic vectors through paraphrase-multilingual-MiniLM-L12-v2.

[0044] Index search: Search the core document index and the additional document index respectively, and calculate the similarity score.

[0045] Multi-index weighted fusion: adjust the weights of search results from different index libraries, giving greater weight to higher-quality documents.

[0046] High-precision screening: Optimize the final returned results by combining BM25 retrieval confidence, topic relevance, and other methods, screen out the top K text fragments with the highest similarity, and provide them as knowledge-enhanced input to the generative large language model.

[0047] In the process of fine-tuning the pre-trained large language model in stages in step S3, the specific steps of the general instruction fine-tuning are: using a large-scale general instruction dataset to train the overall large language model, and the large-scale general instruction dataset includes: general question-answering instructions in the directions of knowledge question-answering, reading comprehension, logical reasoning and code generation.

[0048] Traditional full-parameter fine-tuning typically requires updating all model parameters, which not only consumes enormous computational resources but also easily leads to overfitting and catastrophic forgetting. Based on the above technical features, this invention achieves a low-cost, high-efficiency, and highly stable fine-tuning approach, significantly improving the model's performance in tasks such as carbon trading, carbon emissions accounting, and carbon neutrality policy analysis without compromising its generalization capabilities.

[0049] The specific steps of the professional fine-tuning in the field of carbon neutrality are: inserting a trainable low-rank matrix adaptation layer into the attention layer of the large language model, and using professional data sets to train the low-rank matrix adaptation layer based on the general instruction fine-tuning. The professional data sets include: professional Q&A data, policy and regulation interpretation Q&A, carbon emission calculation and modeling Q&A, and corporate sustainability reports and carbon accounting data.

[0050] After completing the general instruction fine-tuning and carbon neutrality field professional fine-tuning processes, a hybrid data enhancement training process is required. The large-scale general instruction dataset and professional dataset are mixed and the low-rank matrix adaptation layer is trained using synthetic question-answer pairs, knowledge distillation and adversarial training.

[0051] It's important to note that in the first phase, the system uses a large-scale dataset of general instructions for initial adaptation to ensure the model's basic language understanding capabilities and mitigate the risk of a decline in generalization during subsequent specialized fine-tuning. The GPT4-Alpaca dataset contains 50,000 general question-answering instructions, covering open-domain tasks such as knowledge question answering, reading comprehension, logical reasoning, and code generation. This phase of training equips the model with broad language interaction capabilities, ensuring that its understanding of fundamental language structures is not compromised by domain fine-tuning.

[0052] This method significantly reduces the computational resources required for training, enabling efficient domain adaptation training even on a single GPU or even on consumer-grade devices. Compared to traditional fine-tuning methods, the LoRA training method of this invention has the following technical advantages: Low computational cost: LoRA only needs to adjust a small part of the model's weights, reducing training parameters by 10-100 times, significantly reducing computing power requirements; Low overfitting risk: This avoids the problem of overfitting the model to a specific domain due to full parameter fine-tuning, ensuring that the model still has strong general reasoning capabilities. Strong knowledge retention: By introducing a low-rank matrix in the attention layer without modifying the original model weights, the model retains the original knowledge while acquiring new knowledge, effectively reducing the problem of catastrophic forgetting; Strong adaptability: Since the LoRA structure can be reused in different models, the training method of the present invention can quickly adapt to LLMs of different sizes and architectures (such as GLM-4-9B, ChatGLM, Llama2-13B, etc.) and is suitable for computing platforms of different sizes.

[0053] The retrieval enhancement generation mechanism described in step S4 uses a multi-index weighted fusion retrieval method to perform vector database retrieval. When the user enters a query question, the natural language question entered by the user is first converted into a high-dimensional semantic vector, and then searched in the vector database to obtain the knowledge fragments most relevant to the query; after receiving the knowledge fragments returned by the preliminary retrieval, they are sorted according to semantic similarity, and then logical reasoning is performed using the natural language inference model to calculate the implication probability between the query text and the candidate knowledge fragments, and the top N high-quality knowledge fragments are screened out; after the knowledge retrieval and sorting are completed, the highly relevant knowledge fragments are spliced ​​and integrated with the user's original query, and input into the large language model to generate the final answer.

[0054] Based on the above technical features, the present invention not only relies on the model's internally stored knowledge when generating answers, but also dynamically retrieves the latest information relevant to the query from external knowledge bases. This approach effectively alleviates the knowledge timeliness issues inherent in traditional large language models (LLMs) due to the static nature of training data, while also improving the factual accuracy, knowledge coverage, and credibility of the generated answers. Compared to existing LLMs that directly generate content, the RAG module of the present invention utilizes a multi-stage optimization strategy involving knowledge retrieval, semantic ranking, content fusion, and conflict detection, resulting in a model with greater professionalism, stability, and traceability for carbon neutrality question-answering tasks.

[0055] In a specific embodiment, Figure 2 As shown in the figure, when the user enters a query question, the RAG module first converts the natural language question entered by the user into a high-dimensional semantic vector and searches the vector database (FAISS) to obtain the knowledge fragments most relevant to the query. In order to ensure the efficiency and accuracy of the retrieval, the present invention uses sentence-transformers / msmarco-distilbert-base-v3 as the preliminary retrieval model. The model has been optimized to quickly convert the user query text into a vector and improve the retrieval stability through normalization. Subsequently, the system uses the FAISS vector database to calculate the similarity and searches for the knowledge fragments (top-k documents) that best match the query semantics in the core document index and the additional document index respectively. Among them, the core document index mainly stores highly authoritative data from government regulations, research papers, carbon trading policy documents, corporate ESG reports, etc., while the additional document index includes supplementary information sources such as industry news and Wikipedia. To improve retrieval accuracy, this paper further optimizes the retrieval fusion strategy based on FAISS retrieval, adopts cosine similarity for fast matching, and combines it with a multi-index weighted fusion mechanism to ensure that the retrieval weight of authoritative data is higher, making the knowledge fragments finally returned more in line with the user's query intent.

[0056] Because the initial search may return some knowledge fragments with high generalization or low relevance, this invention further introduces a refined ranking mechanism to ensure that the knowledge data ultimately used to generate answers possesses greater relevance and logical consistency. During this stage, the system employs a two-stage ranking optimization strategy: First, the top-k documents returned by the initial search are ranked based on semantic similarity. Subsequently, the system utilizes SciBERT-NLI (Natural Language Inference model) for logical reasoning, calculating the "entailment" probability (entailment score) between the query text and the candidate knowledge fragments, ultimately selecting the top N high-quality knowledge fragments. Compared to the traditional BM25 keyword matching method, this natural language inference (NLI)-based screening method provides a deeper understanding of the logical relationships between texts, ensuring that the retrieved knowledge content is more semantically and logically aligned with the user's query requirements.

[0057] After knowledge retrieval and ranking are completed, the system will splice and integrate highly relevant knowledge fragments with the user's original query and input them into a large language model (such as Mini-CarbonGPT) to generate the final answer. Unlike traditional LLMs that rely solely on internal knowledge to generate text, the RAG module of the present invention can dynamically integrate external knowledge and provide content based on the latest laws, policies, and academic research, thereby significantly improving the timeliness, professionalism, and accuracy of the answers. In addition, in order to solve the hallucination problem in the content generated by LLM, the present invention designs a knowledge conflict detection and credibility scoring mechanism to analyze whether the generated answers have logical conflicts with the known knowledge base content. The mechanism operates based on the following steps: (1) Segment alignment of the answers generated by LLM is performed to detect whether they reference the content retrieved from the knowledge base; (2) NLI reasoning + similarity matching is used to determine the consistency of the generated content with the knowledge base content; (3) For information that may be contradictory, confidence scoring and penalties are implemented to reduce its output probability, and reference source information is added to the final answer. This mechanism is particularly suitable for task scenarios that require extremely high accuracy, such as legal interpretation, carbon trading market analysis, and corporate carbon accounting.

[0058] The RAG module of the present invention has been further optimized in the system architecture to ensure the efficient execution of retrieval and generation tasks. First, to address the efficiency issues of large-scale corpus retrieval, the present invention adopts a distributed index structure that supports parallel query and batch calculation, so that the retrieval process can be completed in milliseconds. Secondly, in order to optimize the accuracy of query matching, the system introduces an incremental index update mechanism to ensure that the FAISS vector library can dynamically adapt to the latest regulations and research results, and automatically adjust the retrieval weight when new knowledge enters the system. In addition, the present invention optimizes the context fusion strategy in the generative large model reasoning process, improves the accuracy of the model in long text reasoning tasks through windowing processing, and enhances the adaptability of the model in the field of carbon neutrality through multi-task learning (MTL).

[0059] Compared to traditional LLM methods that directly generate content, the RAG solution of this invention significantly improves the reliability, professionalism, and knowledge coverage of question-answering systems, making it particularly suitable for specialized scenarios such as carbon emission management, policy interpretation, and carbon trading markets. The innovation of this patent lies in its integration of multiple key technologies, including efficient semantic retrieval, cross-index weighted fusion, and knowledge conflict detection, effectively enhancing the practicality and credibility of large language models in the field of carbon neutrality.

[0060] The specific process of the multi-index weighted fusion retrieval is as follows: converting the user query text into a vector and normalizing it, using the FAISS vector database to perform similarity calculation, and searching for the knowledge fragment that best matches the query semantics in the core document index and the additional document index respectively.

[0061] Based on the above technical features, the present invention has achieved substantial breakthroughs in multiple key technical links through systematic improvements to the large language model in the field of carbon neutrality, significantly improving the model's performance in terms of professionalism, accuracy, practicality, and adaptability, and overcoming multiple pain points and bottlenecks in the existing technology. Compared with traditional general-purpose large language models, the vertical field large model proposed by the present invention is not only more in-depth in terms of knowledge acquisition and expression, but also has stronger technical advantages in terms of system stability and evolution capabilities.

[0062] First, at the knowledge data level, the present invention effectively compensates for the problems of knowledge generalization, lack of industry information, and semantic understanding deviation in general large models by introducing an authoritative, structured, and widely covered professional knowledge base on carbon neutrality. This knowledge base is derived from a variety of authoritative channels, including national policies and regulations, reports from international organizations, academic research papers, corporate ESG reports, and carbon trading market data. After unified semantic segmentation, labeling, and vectorization processing, it forms a semantic database with high timeliness, high authority, and high professionalism. This database not only provides a rich source of knowledge for model training, but also provides real-time support through semantic retrieval in question and answer generation, enabling the model to have stronger knowledge and interpretation capabilities in tasks such as carbon emission calculation, policy interpretation, and carbon credit analysis.

[0063] Secondly, at the level of language generation mechanisms, this invention innovatively integrates a retrieval-augmented generation mechanism (RAG). By jointly inputting user queries and semantically related paragraphs from the knowledge base into a large language model, this enables dynamic knowledge content completion and fact verification during the answer generation process. At the same time, to further alleviate the "hallucination phenomenon" common in large models, that is, the risk of the model generating false or inaccurate information, this invention has designed a knowledge conflict detection and credibility scoring mechanism. This mechanism can promptly identify and adjust when there are deviations between the generated content and the knowledge base, ensuring that the output results are verifiable, logically consistent, and traceable, significantly improving the safety and professionalism of the model in tasks such as policy analysis and regulatory comparison.

[0064] Third, at the model training strategy level, this invention utilizes Low-Rank Adaptive Fine-tuning (LoRA) technology to achieve efficient domain adaptation of the base model with minimal computational resource overhead. By dividing the large language model into a general instruction pre-training phase and a carbon-neutral specialized fine-tuning phase, and inserting a trainable parameter matrix into the attention layer for localized adjustments, the model can quickly grasp the domain terminology, policy logic, and computational task knowledge required while maintaining general language capabilities. Compared to full parameter fine-tuning, the LoRA fine-tuning solution of this invention offers advantages such as flexible deployment, efficient computation, and strong adaptability, making it adaptable to a wide range of computing platforms and application scenarios.

[0065] Finally, at the level of knowledge updating and maintenance mechanism, the present invention constructs a dynamic knowledge synchronization and incremental database reconstruction system to solve the "outdated knowledge" problem of traditional models. This mechanism captures the latest information from platforms such as IPCC, Ministry of Ecology and Environment, and academic databases on a regular or on-demand basis, and combines timestamp management, knowledge failure detection, and semantic conflict identification technology to achieve automatic updating of the knowledge base, elimination of old knowledge, and weight adjustment. Combined with the incremental update structure of the vector database, the knowledge supplement process does not affect the existing retrieval structure. The system always maintains the ability to respond quickly to the latest policy changes and academic progress, and is highly adaptable in actual environments where policies change frequently.

[0066] In summary, this invention, through the technical collaboration of four core links: data construction, generation mechanism, training method, and knowledge updating, not only breaks through the limitations of existing technologies in terms of knowledge depth, output credibility, and model adaptability, but also realizes the intelligent processing capability of complex problems in the field of carbon neutrality. This technical solution can be widely used in many fields such as government carbon neutrality policy decision-making support systems, corporate carbon accounting and disclosure platforms, and intelligent knowledge service engines for scientific research institutions. It has extremely high engineering feasibility, commercial scalability, and technological innovation, and has important practical significance and industrial value in the context of promoting green and low-carbon transformation.

[0067] Example 2 Based on the above embodiment 1, combined Figure 3 This embodiment also provides a vertical large language model training system in the field of carbon neutrality. The system is used for the vertical large language model training method in the field of carbon neutrality, and includes: a carbon neutrality field data acquisition module, a vectorized encoding module, a low-rank adaptation module and a retrieval enhancement module.

[0068] The carbon neutrality field data acquisition module collects data in the carbon neutrality field and performs data preprocessing, constructs a carbon neutrality field knowledge base, and uses a dynamic update mechanism to update the carbon neutrality field knowledge base; the vectorization encoding module dynamically semantically segments and vectorizes the text of the carbon neutrality field knowledge base and stores it in the vector database; the low-rank adaptation module fine-tunes the pre-trained large language model in stages based on low-rank adaptation technology, including general instruction fine-tuning and carbon neutrality field professional fine-tuning; the retrieval enhancement module uses a retrieval enhancement generation mechanism to retrieve knowledge fragments related to user queries through the vector database, and inputs them into the large language model to generate answers.

[0069] Example 3 This embodiment will further illustrate the deployment environment and usage method of the large model described in the present invention.

[0070] In this example, the Mini-CarbonGPT system is deployed in a medium- to high-performance server environment to support large language model training, inference, and knowledge retrieval tasks. The hardware configuration includes dual-core Intel Xeon Platinum 8269CY CPUs (128 threads total), 512GB of RAM, and two NVIDIA RTX 2080Ti GPUs (22GB total video memory). High-speed NVMe solid-state drives are used to support large-scale corpus loading and indexing. The operating system is Ubuntu Server 20.04, and the Python runtime environment version is 3.10. The primary deep learning frameworks used are PyTorch 2.0 and Hugging Face Transformers. Model management and inference integrate the DeepSpeed ​​and PEFT frameworks to support efficient parameter fine-tuning and deployment.

[0071] The basic language model uses the open-source GLM-4-9B model as its foundation, which boasts strong Chinese comprehension and open-ended question-answering capabilities. This model is vertically adapted for the carbon neutrality sector through a two-stage fine-tuning process. To balance professionalism and deployment efficiency, this paper introduces an INT4 quantization strategy based on the GLM model, combined with LoRA structure embedding (Low-Rank Adaptation). This significantly reduces model memory usage during fine-tuning and deployment, ensuring stable and efficient operation even in a single-machine, multi-GPU environment.

[0072] To support semantic retrieval and vectorized storage, this example uses FAISS (Facebook AI Similarity Search) to build a vector database. The vectorization model uses paraphrase-multilingual-MiniLM-L12-v2, a lightweight and cross-lingual model capable of building high-quality sentence vectors in multilingual contexts. The system operates in a modular fashion, encompassing six core components: corpus construction, semantic indexing, fine-tuning training, RAG retrieval enhancement, inference generation, and result filtering. This supports end-to-end intelligent generation and semantic traceability for carbon-neutral question-answering tasks.

[0073] To enhance the expertise and accuracy of large language models in carbon neutrality tasks, this paper constructs a structured, dynamically updated carbon neutrality domain knowledge base. This knowledge base integrates multi-source information from government policies, scientific research literature, industry reports, and encyclopedia data, ensuring the model has a broad and in-depth understanding of regulations, emission standards, and carbon accounting principles.

[0074] In terms of data sources, the knowledge base content mainly consists of the following categories: Policy and regulatory texts: collected from the official websites of the Ministry of Ecology and Environment of China, the National Development and Reform Commission, and the United Nations IPCC website, including key policy documents such as the "Top-level Design Plan for Carbon Peaking and Carbon Neutrality", the "Action Plan for Carbon Peaking Before 2030", and the "Global Greenhouse Gas Accounting Standard".

[0075] Academic research papers: Keywords such as “carbon emissions,” “CCUS,” and “carbon footprint assessment” were searched from CNKI and Web of Science, and a total of 6,096 papers with high citations and highly relevant content in the past ten years were selected.

[0076] Enterprise and market data: including ESG reports of listed companies, carbon trading market white papers, regional carbon emission statistical yearbooks, etc., which constitute the supporting part of carbon market dynamic information.

[0077] Supplementing encyclopedia data: By crawling approximately 60,000 pages related to keywords such as "carbon neutrality", "carbon sink", and "negative emission technology" from Wikipedia, we use them as a general semantic support library to enhance the model's coverage of long-tail knowledge points.

[0078] All collected texts must undergo a rigorous preprocessing process before being stored in the database, including: paragraph-priority semantic segmentation: dividing the text into chunks of no more than 800 characters in length, with the overlapping area of ​​each chunk set to 150 characters (chunk_overlap) to maintain contextual coherence; text cleaning and grammatical error correction: using regular expressions to remove redundant content such as page numbers, garbled characters, and empty paragraphs, and combining language_tool_python to correct language standardization; duplication detection and deduplication optimization: through hash verification and Jaccard similarity analysis, highly repeated corpus paragraphs are removed to improve the quality of the knowledge base; semantic embedding encoding: the processed text segments are input into the paraphrase-MiniLM model to generate sentence vectors, and then L2 normalization is performed to standardize their index performance in the FAISS database.

[0079] Ultimately, the knowledge base was organized into a dual-index structure, divided into: a core literature index, which includes authoritative materials such as original policy texts, highly cited papers, and government annual reports; and an additional knowledge index, which includes supplementary content such as encyclopedia materials, general technical explanations, and secondary references. This structured corpus organization not only improves recall efficiency and semantic relevance during retrieval, but also supports a dynamic knowledge update mechanism. Through regular crawling and incremental encoding strategies, the system can automatically expand the knowledge base, de-rank or remove outdated content, and ensure that the model's question-answer generation and semantic reasoning are always based on the latest policies and research.

[0080] To further enhance the adaptability of the basic language model to tasks in the carbon neutrality field, this paper employs Low-Rank Adaptation (LoRA) fine-tuning technology. This allows for efficient parameter optimization and expertise transfer without changing the core weights of the basic model. This fine-tuning process is conducted in two phases, balancing the model's general language capabilities with its domain-specific expressive capabilities.

[0081] In the first phase of general instruction fine-tuning, the system used the openly available GPT4-Alpaca dataset, which contains 52,000 high-quality human instruction and response samples, covering multiple scenarios such as general question-answering, conversational dialogue, and multi-turn instruction understanding. The goal of this phase was to maintain the model's capabilities in basic language understanding, task generalization, and language generation, preventing subsequent fine-tuning in specialized areas from causing overfitting or a decrease in generalization capabilities. During fine-tuning, the following parameter configurations were used: The learning rate is set to 2e-5; Batch size is set to 64; LoRA rank value is 8; Dropout value is 0.05; Use the Early Stopping strategy to automatically terminate training when the validation set loss does not decrease significantly.

[0082] In the second phase of fine-tuning specialized instructions, the present invention constructed a dedicated carbon neutrality domain fine-tuning dataset, drawing on UltraChat professional conversation data, question-and-answer conversion of policy document content, explanatory question-and-answer on carbon emission calculation formulas, and structured conversational data from the CCUS technical knowledge graph, totaling 5,382 high-quality domain instruction data. At this stage, the model possessed general language capabilities, and fine-tuning shifted to tasks such as understanding domain terminology, reasoning about complex causal relationships, and ensuring the correctness of regulatory language. Due to the small size of the training data, the LoRA fine-tuning strategy significantly reduced the number of parameter updates, effectively preventing overfitting and accelerating training convergence.

[0083] In an environment with limited hardware resources (2080Ti GPUs x 2), this example significantly reduces the graphics memory overhead required for training thanks to the combination of LoRa and INT4 quantization technology. Peak graphics memory usage on a single graphics card is kept below 10.2GB, and all fine-tuning tasks can be completed on a single machine within 24 hours. The fine-tuned model demonstrates enhanced domain terminology understanding, contextual relevance, and factual accuracy in the carbon neutrality question-answering task, significantly outperforming both the original untuned model and mainstream open-source general-purpose models.

[0084] When a user enters a natural language question, the system invokes the paraphrase-multilingual-MiniLM-L12-v2 vectorization model to encode the input text into a 384-dimensional semantic vector. This vector is then used to match against the two-layer FAISS vector database, querying both the core knowledge index and the supplementary knowledge index, returning the top 10 similar text passages. To ensure authoritativeness, the system assigns different weights to results from different sources, giving higher priority to core policy documents, for example.

[0085] Secondly, based on the search results, this paper introduces a semantically refined ranking mechanism, using a trained natural language inference model (SciBERT-NLI) to model the logical relationship between candidate text segments and the query. The model output includes labels such as "entailment," "neutral," and "contradiction." The system ranks the results based on the confidence level of the "entailment" category, retaining only the top k most informative pieces of knowledge as supporting content.

[0086] Next, the system combines the user's original question with these high-confidence knowledge fragments to create a complete context, which is then fed into the fine-tuned Mini-CarbonGPT model as a prompt. During the generation process, the system performs formatting checks and redundancy reduction on the output content to ensure clear, logically closed, and well-worded responses.

[0087] Finally, to enhance the verifiability and credibility of model-generated content, this paper designs a knowledge conflict detection and confidence scoring mechanism. When the system detects that the model-generated content contains unsupported or inconsistent paragraphs with the knowledge base, it automatically annotates them and lowers their output priority. Furthermore, the source document number or link to the original corpus of the cited paragraph is included in the answer output, forming a traceable question-and-answer chain, significantly improving the reliability and regulatory compliance of the question-and-answer system.

[0088] In a specific embodiment, the carbon neutrality vertical large language model described in this invention is applied to the comprehensive research scenario of CCUS (Carbon Capture, Utilization, and Storage). Through a semantic retrieval and generation mechanism, it rapidly responds to researchers' queries on complex issues such as technical principles, policies and regulations, and economic feasibility, and outputs intelligent answers with clear structure, traceable knowledge, and authoritative content. Users enter a natural language question through the system front-end. For example, a question like "Please introduce the current mainstream carbon capture technology routes and their applicable scenarios, and identify key future development directions based on Chinese policies." The system first inputs the user's question into the vector encoding module. A 384-dimensional semantic vector is generated using the paraphrase-multilingual-MiniLM model. A top-k search is then performed on the "Core Knowledge Index" and "Additional Knowledge Index." The system then uses the SciBERT-NLI model to perform logical entailment analysis on these search results and the user's question, selecting and fusing high-confidence content segments encompassing the three elements of "capture methods + application scenarios + Chinese policies." The model then combines the user's question with the semantically filtered text segments to automatically generate the answer.

[0089] In terms of output capability and efficiency, this embodiment automatically extracts the three elements of "technical path + applicable scenarios + policy orientation", reflecting the ability of content understanding and structural planning; the model output is accompanied by a retrieval source number to enhance verifiability and avoid "generation illusion", which represents strong knowledge traceability; Mini-CarbonGPT, after fine-tuning based on LoRA, can accurately use professional terms such as "post-combustion" and "carbon capture and storage", and correctly cite policy context, reflecting the characteristics of excellent professional adaptation; the entire process from user input to result generation takes an average of 2.1 seconds, and its fast response technical effect is suitable for real-time application scenarios such as scientific research consultation and policy question answering.

[0090] Table 2 Comparison with existing models (GPT-4o, GLM-base)

[0091] This example fully demonstrates the practical application of this invention in comprehensive CCUS research scenarios. By integrating semantic retrieval, dynamic knowledge base updates, RAG generation, and conflict detection mechanisms, the model can quickly and accurately answer highly complex, cross-knowledge-dimensional technical and policy questions. It is particularly suitable for scenarios such as CCUS project consulting, policy development assistance, and research data analysis. As shown in Table 2, compared to traditional question-answering systems or general-purpose large models, this invention possesses key advantages such as strong professional adaptability, broad knowledge coverage, high traceability, and a clear output structure. It has significant engineering value and market application prospects in intelligent assistance applications within the carbon neutrality technical path.

[0092] In another specific embodiment, the application results of the present invention in low-carbon and zero-carbon technology scenarios are shown in Table 3. Table 3 Quantitative and qualitative evaluations with other models on the same task

[0093] This example demonstrates that the present invention can fully leverage its specialized knowledge base and semantic generation capabilities to address complex coordinated scheduling tasks in low-carbon and zero-carbon energy systems, providing high-quality answers that are logical, technical, and policy-oriented. This approach is particularly applicable in the following scenarios: power system operation planning; energy storage scheduling and economic analysis; policy consultation and energy structure assessment; and support for corporate carbon neutrality decision-making. Compared to traditional large language models, this invention offers significant advantages in structural understanding, precise terminology, and policy reference. It can be widely applied in scenarios such as smart grid platforms, energy storage system developers, power dispatch centers, and new energy planning and research institutions, enhancing the intelligence level of low-carbon development.

[0094] In a specific example, to evaluate the performance of various models in objective (multiple-choice) scenarios related to carbon neutrality, this study designed 700 multiple-choice questions across five core sub-areas: CCUS comprehensive research, negative emissions technologies, low-carbon and zero-carbon technologies, carbon neutrality strategy and policy planning, and carbon accounting and market mechanisms. Table 4 summarizes the accuracy rates of each model in different sub-areas.

[0095] Table 4 Objective question evaluation results (accuracy of different models in various fields)

[0096] Overall, Mini-CarbonGPT took first place with an average accuracy of 80.57%, followed by GPT-4o (79.43%) and GLM-RAG (77.14%). This demonstrates that Mini-CarbonGPT possesses a significant reasoning advantage in objective-based tasks, demonstrating that fine-tuning and retrieval enhancement are crucial for such knowledge-intensive tasks. Other commercial or research models (Gemini-1.5 Flash, Kimi, ERNIE Bot-3.5, and GLM-Finetune) achieved average scores between 72% and 76%, while the original, unfine-tuned GLM-4-9B model achieved only 70.00%. This suggests that basic large-scale models have significant knowledge gaps in the field of carbon neutrality.

[0097] There are significant differences in the accuracy of the various models across different sub-fields of carbon neutrality. In the CCUS integrated research field, Mini-CarbonGPT achieved an accuracy of 76.84%, slightly lower than GPT-4o (78.53%). This suggests that GPT-4o still possesses strong generalization capabilities in deep reasoning and language understanding tasks in the CCUS integrated research field. On the other hand, the average accuracy of the models in the CCUS field was 72.39%, while the initial model accuracy of GLM-4-9B was only 61.02%. This suggests that the initial model of GLM-4-9B may have been insufficiently pre-trained in the CCUS integrated field. However, Mini-CarbonGPT compensated for this lack of pre-training material through specialized fine-tuning and the RAG strategy, gaining reinforcement of interdisciplinary terminology and background knowledge, thereby significantly narrowing the gap.

[0098] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiments; and the aforementioned storage medium includes: various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks or optical disks.

[0099] Alternatively, if the above-mentioned embodiments of the present invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes a number of instructions for causing a computer device to execute all or part of the methods described in the various embodiments of the present invention. The aforementioned storage media include various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0100] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. The icons in the accompanying drawings that describe the structural positional relationships are only for illustrative purposes and are not to be construed as limiting the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A vertical large language model training method in the field of carbon neutrality, characterized in that: The following steps are involved: S1. Collect data in the field of carbon neutrality and perform data preprocessing to build a knowledge base in the field of carbon neutrality, and use a dynamic update mechanism to update the knowledge base in the field of carbon neutrality; S2. Dynamically semantically segment and vectorize the text of the carbon neutrality domain knowledge base and store it in a vector database; S3. Fine-tune the pre-trained large language model in stages based on low-rank adaptation technology, including general instruction fine-tuning and carbon neutrality-specific fine-tuning; S4. Adopting the retrieval enhancement generation mechanism, the knowledge fragments related to the user query are retrieved through the vector database, and the large language model is input to generate the answer.

2. A vertical large language model training method in the field of carbon neutrality according to claim 1, characterized in that The data preprocessing described in step S1 includes the following steps: removing noise data in unstructured text, and using NLP semantic similarity calculation to remove redundant text; converting the text format into standardized text data, and combining OCR technology to perform high-precision recognition of scanned research documents; all data are structured and stored, and summary labels are added to the data.

3. A vertical large language model training method in the field of carbon neutrality according to claim 2, characterized in that: The dynamic update mechanism is specifically as follows: according to the preset update cycle or when actively triggered by the user, the latest relevant policies, papers, reports and market data are automatically captured from the network data; NLP technology is used to compare the contents of old and new policies, and the abolished or adjusted policies are automatically detected and marked, and manual review is introduced for key policies; when there is conflicting content between the old and new policies, a weight adjustment mechanism is used to automatically optimize the authority of knowledge.

4. A vertical large language model training method in the field of carbon neutrality according to claim 1, characterized in that The specific process of dynamic semantic segmentation and vectorized encoding described in step S2 is as follows: the document is segmented according to paragraph delimiters, and paragraphs with more than 800 characters are further segmented based on punctuation using regular expressions. At the same time, an overlapping area of ​​150 characters is introduced in the segmentation process; a lightweight cross-language semantic embedding model is used to generate sentence vectors for the segmented text, and semantic retrieval is implemented through the FAISS vector database, and the generated vectors are standardized.

5. The vertical large language model training method in the field of carbon neutrality according to claim 1 is characterized in that: In the process of fine-tuning the pre-trained large language model in stages in step S3, the specific steps of the general instruction fine-tuning are: using a large-scale general instruction dataset to train the overall large language model, and the large-scale general instruction dataset includes: general question-answering instructions in the directions of knowledge question-answering, reading comprehension, logical reasoning and code generation.

6. A vertical large language model training method in the field of carbon neutrality according to claim 5, characterized in that: The specific steps of the professional fine-tuning in the field of carbon neutrality are: inserting a trainable low-rank matrix adaptation layer into the attention layer of the large language model, and using professional data sets to train the low-rank matrix adaptation layer based on the general instruction fine-tuning. The professional data sets include: professional Q&A data, policy and regulation interpretation Q&A, carbon emission calculation and modeling Q&A, and corporate sustainability reports and carbon accounting data.

7. A vertical large language model training method in the field of carbon neutrality according to claim 6, characterized in that: After completing the general instruction fine-tuning and carbon neutrality field professional fine-tuning processes, a hybrid data enhancement training process is required. The large-scale general instruction dataset and professional dataset are mixed and the low-rank matrix adaptation layer is trained using synthetic question-answer pairs, knowledge distillation and adversarial training.

8. The vertical large language model training method in the field of carbon neutrality according to claim 1 is characterized in that: The search enhancement generation mechanism described in step S4 uses a multi-index weighted fusion search method to search the vector database. When the user enters a query question, the natural language question entered by the user is first converted into a high-dimensional semantic vector, and then searched in the vector database to obtain the knowledge fragments most relevant to the query. After receiving the knowledge fragments returned by the preliminary search, they are sorted according to semantic similarity, and then logical reasoning is performed using the natural language inference model to calculate the implication probability between the query text and the candidate knowledge fragments, and the top N high-quality knowledge fragments are screened. After knowledge retrieval and ranking are completed, highly relevant knowledge fragments are spliced ​​and integrated with the user's original query, and input into the large language model to generate the final answer.

9. A vertical large language model training method in the field of carbon neutrality according to claim 8, characterized in that: The specific process of the multi-index weighted fusion retrieval is as follows: converting the user query text into a vector and normalizing it, using the FAISS vector database to perform similarity calculation, and searching for the knowledge fragment that best matches the query semantics in the core document index and the additional document index respectively.

10. A vertical large language model training system in the field of carbon neutrality, the system is used for a vertical large language model training method in the field of carbon neutrality according to any one of claims 1-9, characterized in that: It includes: carbon neutrality field data acquisition module, vectorized encoding module, low-rank adaptation module and retrieval enhancement module; The carbon neutrality field data acquisition module collects data in the carbon neutrality field and performs data preprocessing, constructs a carbon neutrality field knowledge base, and uses a dynamic update mechanism to update the carbon neutrality field knowledge base; the vectorized encoding module dynamically semantically segments and vectorizes the text of the carbon neutrality field knowledge base and stores it in a vector database; the low-rank adaptation module fine-tunes the pre-trained large language model in stages based on low-rank adaptation technology, including general instruction fine-tuning and carbon neutrality field professional fine-tuning; The retrieval enhancement module adopts a retrieval enhancement generation mechanism to retrieve knowledge fragments related to user queries through a vector database and input them into a large language model to generate answers.

Images