Power system adaptive evaluation system construction method based on LLM and analytic hierarchy process

Abstract

The invention discloses a power system adaptive evaluation system construction method based on LLM and analytic hierarchy process, comprising the following steps: step 1, collecting corpora, and constructing a knowledge base through multi-source data; step 2, knowledge enhancement and processing based on an RAG framework; step 3, professional field fine tuning and adaptive optimization of the large language model; step 4, generation of a self-adaptive power system evaluation system; step 5, subjective evaluation by human experts; and step 6, performing big language model evaluation and human expert comprehensive evaluation. The comprehensiveness, the accuracy and the adaptability of power system evaluation are remarkably improved, and the high-standard requirement in a complex evaluation environment is met.

Description

Technical Field

[0001] The present invention relates to the field of intelligent testing technology, and in particular to a method for constructing a power system adaptive evaluation system based on LLM and analytic hierarchy process, which integrates the large language model (LLM) and the judgment method of human experts, and can comprehensively and dynamically analyze and optimize the performance and applicability of the power system, thereby achieving a more scientific and comprehensive power system evaluation. Background Art

[0002] The performance evaluation of powertrains is of great significance in the automotive and industrial fields. Traditional evaluation methods mainly rely on the experience and manual operation of human experts. However, these methods have many limitations, such as the difficulty in systematically and comprehensively considering all evaluation indicators, and the evaluation process is limited by factors such as the breadth of experts' knowledge and time and energy. Especially in the face of diverse and complex evaluation scenarios, traditional methods often cannot fully and accurately reflect the true performance and applicability of powertrains.

[0003] In recent years, with the breakthroughs made by large language models (such as GLM-4.0) in the fields of natural language processing and knowledge reasoning, with their powerful learning and generation capabilities, a more comprehensive evaluation system for complex systems can be constructed. Large language models can process large amounts of heterogeneous data, learn and summarize cross-domain knowledge, which gives them greater breadth and depth when constructing an evaluation system for dynamical systems. Through automated knowledge extraction and analysis, large language models can cover more evaluation indicators, reduce human bias and omissions, and thus improve the objectivity and comprehensiveness of the evaluation system.

[0004] However, the application of large language models in specific products or scenarios still has certain limitations. Although the model has a broad knowledge base and strong reasoning ability, it may lack professional depth or practicality when faced with specific technical details or scenarios. Relatively speaking, human experts have rich field experience and intuitive judgment ability, and can provide more detailed and adaptive evaluations for specific products or environments. This feature makes expert opinions more practical in some cases, especially when it comes to optimization decisions in specific areas. Therefore, the present invention proposes a comprehensive evaluation method that integrates large language models and human experts, and constructs a more scientific and efficient power system test evaluation system by complementing the advantages of the two. Summary of the invention

[0005] The purpose of the present invention is to provide a method for constructing a power system adaptive evaluation system based on a large language model and a hierarchical analysis method to address the technical defects in the prior art. The method introduces an adaptive artificial intelligence evaluation strategy, combines the feedback from human experts, optimizes and improves the evaluation index system, and improves the accuracy and adaptability of the evaluation.

[0006] The technical solution adopted to achieve the purpose of the present invention is:

[0007] A method for constructing a power system adaptive evaluation system based on LLM and analytic hierarchy process includes the following steps:

[0008] Step 1: Collect corpus and build a knowledge base through multi-source data:

[0009] After preprocessing the corpus, the structured data in the corpus is represented by knowledge graph technology and stored in a relational database. The unstructured data in the corpus is processed and stored in a NoSQL database. The knowledge base is constructed using the relational database and the NoSQL database.

[0010] Step 2: Knowledge enhancement and processing based on the RAG framework:

[0011] The data in the relational database and NoSQL database use the BERT word vector embedding model to convert the text content into embedded knowledge vectors, store them in a unified test corpus, and interact with the large language model;

[0012] Step 3: Fine-tune and optimize the adaptability of large language models in specialized fields:

[0013] Fine-tune the selected large language model through continuous task-specific training sets and validation sets, and optimize the parameters of the large language model during the fine-tuning process. At runtime, the user's natural language needs are converted into query vectors through the BERT word vector embedding model. The query vector is similar to the knowledge vector in the vectorized knowledge base, and the knowledge vector with the highest similarity is selected. The vector is used to find the corresponding knowledge entry and passed to the large language model.

[0014] Step 4: Generation of an adaptive power system evaluation system:

[0015] Based on the guidance of user needs and human intervention prompts, the large language model extracts key evaluation indicators of the power system and generates a multi-level evaluation framework. The evaluation framework includes an evaluation hierarchy structure and the initial weights of the evaluation indicators at each level. Through multiple rounds of iterative optimization with user feedback, the evaluation hierarchy structure and the weights of each level are further adjusted to finally form an adaptive power system evaluation system.

[0016] Step 5, subjective evaluation by human experts:

[0017] Invite several experts in the field of power systems to obtain the evaluation framework and weight distribution with the help of step 4, and conduct comprehensive evaluation and correction. The experts evaluate through scoring and test data, and calculate the relative weights of indicators at each level.

[0018] Step 6: Comprehensive evaluation of large language models and human experts

[0019] The weights generated by the large language model in step 4 and the weights obtained by human experts through AHP in step 5 are weighted averaged to obtain the final comprehensive evaluation weights. Based on this weight, the final power system adaptive evaluation system is completed.

[0020] In the above technical solution, in step 1, the preprocessing includes cleaning, formatting and consistency verification.

[0021] In the above technical solution, in step 1, the corpus in the knowledge base includes but is not limited to the basic principles and steps of the AHP hierarchical analysis method and its application examples in power system evaluation, technical documents, standard operating manuals, historically constructed evaluation systems and expert reports.

[0022] In the above technical solution, in step 3, the large language model includes but is not limited to GLM-4.0.

[0023] In the above technical solution, in step 3, during the fine-tuning process of the large language model, an optimization algorithm and a learning rate adjustment strategy are used to ensure rapid convergence and optimization of the parameters of the large language model.

[0024] In the above technical solution, the optimization algorithm adopts Adam or SGD.

[0025] In the above technical solution, in step 3, during the fine-tuning of the large language model, a regularization technique is implemented to prevent the large language model from overfitting.

[0026] In the above technical solution, in step 5, human experts compare the indicators in pairs based on the analytic hierarchy process (AHP) to construct a judgment matrix.

[0027] In the above technical solution, in step 5, after calculating the relative weights of indicators at each level, a consistency check is performed to ensure the rationality and scientificity of the weight distribution.

[0028] Another aspect of the present invention also includes a power system adaptive evaluation system obtained by the construction method.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. The present invention provides a method for constructing an adaptive evaluation system for a power system based on LLM and analytic hierarchy process, which integrates multiple core technologies such as corpus collection, RAG framework knowledge enhancement, function call and automated execution engine, significantly reducing the need for manual intervention. Compared with traditional testing methods that rely on engineer experience and manual operations, the intelligent agent integrates a wide range of cross-domain knowledge (such as theoretical knowledge, experiential knowledge, historical test cases) through a language model, and can refer to more dimensional information during the test strategy optimization process. This fusion of multi-source information enables the intelligent agent to automatically generate the optimal test plan when dealing with complex test tasks, improve test efficiency, and reduce test time and cost.

[0031] 2. The evaluation system of the present invention is highly scalable and adaptable. The learning ability of the large language model enables it to quickly adapt to emerging technologies and changing industry standards, while the expert feedback mechanism ensures the pertinence of the system in different application scenarios. By combining the advantages of intelligence and human experience, the present invention can adapt to the diverse and complex needs of future power systems and provide a sustainable solution for long-term use.

[0032] 3. The present invention combines a large language model with natural language processing to enable the intelligent agent to achieve humanized test process management and strategy optimization. Users can interact with the intelligent agent through natural language, raise test requirements or questions, and the intelligent agent will automatically understand the user's intentions and generate corresponding test tasks and optimization suggestions. Compared with the complex configuration operations of traditional test platforms, the present invention provides a more intuitive and simple human-computer interaction method, which improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 The figure shows the overall architecture diagram of the power system adaptive evaluation system. DETAILED DESCRIPTION

[0034] The present invention is further described in detail below in conjunction with specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0035] A method for constructing a power system adaptive evaluation system based on LLM and analytic hierarchy process includes the following steps:

[0036] Step 1: Collect corpus and build a knowledge base through multi-source data:

[0037] In the early stages of adaptive testing, it is crucial to build a comprehensive knowledge base. This knowledge base should be able to integrate data from different sources, including but not limited to the basic principles and steps of the AHP (such as establishing a hierarchy, calculating weights, etc.), as well as its application examples in power system evaluation, technical documents, standard operating manuals, historically constructed evaluation systems, and expert reports. The data preprocessing process involves cleaning, formatting, and consistency verification to ensure the accuracy and availability of the data.

[0038] Structured data, such as the basic principles of AHP and technical documents, follow a certain format and organizational structure. Advanced text mining technology is used to remove redundancy and noise in the data and achieve refined management of knowledge. Structured data is represented by knowledge graph technology to form triples (Entity-Relationship-Entity) and stored in a relational database.

[0039] Unstructured data, such as standard operating manuals, historically constructed evaluation systems, and expert reports, do not have a fixed format, such as text files, images, and videos. Unstructured data is semantically enhanced using pre-trained large language models such as GLM-4.0 or NLP natural language processing, and is stored in a NoSQL database;

[0040] In this step, data fusion technology will also be used to unify the data from different sources and formats, standardize the structured data and unstructured data, and unify the format for subsequent processing. In addition, data quality control measures will be implemented, including data integrity, accuracy and timeliness checks to ensure the reliability of the data set. Through these measures, a solid, comprehensive and high-quality test corpus will be built, providing a solid data foundation for subsequent test strategy generation and optimization.

[0041] Step 2: Knowledge enhancement and processing based on RAG framework:

[0042] The present invention uses the Retrieval Augmented Generation (RAG) framework to deeply process the corpus collected in step 1 to support complex reasoning tasks. The data in the NoSQL database and the relational database are respectively converted into embedded knowledge vectors using the BERT word vector embedding model, stored in a unified test corpus, and interacted with the large language model to provide support for the decision-making of the intelligent agent.

[0043] In the application of the RAG framework, special attention will be paid to the dynamic updating and maintenance of knowledge to ensure that the knowledge base can reflect the latest technological developments and testing practices. In addition, an efficient retrieval mechanism will be implemented to enable the agent to quickly and accurately retrieve relevant information from the knowledge base to support the generation and adjustment of its testing strategy.

[0044] Step 3: Fine-tune and optimize the adaptability of large language models in specialized fields:

[0045] A pre-trained large language model (GLM-4.0 is recommended but not limited to) is selected for domain-specific fine-tuning. The fine-tuning process is carried out through continuous task-specific training sets and validation sets to evaluate the performance of the model on specific test tasks and adjust the model parameters accordingly. The optimized model will have stronger domain adaptability and can efficiently handle complex test scenarios. During the fine-tuning process, advanced optimization algorithms such as Adam or SGD and appropriate learning rate adjustment strategies will be used to ensure rapid convergence and optimization of model parameters. In addition, regularization techniques such as L1 or L2 regularization will be implemented to prevent overfitting of the model and improve its generalization ability. Through these measures, it will be ensured that the model can show excellent performance and adaptability when facing new test tasks.

[0046] When the system is running, the user's natural language needs are converted into query vectors through the BERT word vector embedding model. The similarity between the query vector and the knowledge vectors in the unified test corpus is calculated, and the knowledge vector with the highest similarity is selected. The corresponding knowledge entry is found with this vector and passed to the large language model.

[0047] Step 4: Generation of Adaptive Evaluation System

[0048] The generation of the adaptive evaluation system is achieved through the reasoning and generation capabilities of the Large Language Model (LLM) combined with prompts designed by human intervention. First, the large language model extracts key evaluation indicators based on the user's natural language needs, and uses the Retrieval Augmented Generation (RAG) method to select the most relevant knowledge fragments from the knowledge base as few-shot inputs (few-shot learning inputs) for reference. Based on these inputs, the model will dynamically generate a multi-level evaluation framework according to the specific goals of user needs, including defining the evaluation level and the initial weights of indicators at each level.

[0049] During this process, the large language model can adaptively adjust the evaluation system to ensure its applicability and robustness in a variety of scenarios. After the evaluation framework is generated, the model will accept multiple rounds of user feedback and iteratively optimize the evaluation system by adjusting indicator weights and optimizing the evaluation hierarchy. This method uses the powerful learning ability of the large language model and the flexibility of user feedback to make the generated evaluation system both comprehensive and adaptable to specific needs.

[0050] Step 5: Subjective evaluation by human experts

[0051] After the large language model constructs a preliminary evaluation system, human experts are introduced to conduct subjective evaluation to further optimize and refine the system. Several experts with experience in the field of power systems are invited to use the Analytic Hierarchy Process (AHP) to compare each evaluation indicator pairwise by constructing a pairwise comparison matrix.

[0052] Experts use specific scoring methods to evaluate the relative importance of different indicators based on their own professional knowledge and experience, and use the evaluation framework and weight distribution generated by the large language model to conduct comprehensive evaluation and correction. Subsequently, a consistency check is used to verify the expert's judgment matrix to ensure that the weights of each indicator are reasonable and scientific. Through this process, the subjective judgment of human experts will be organically combined with the system generated by the model to supplement the deficiencies that may exist in the model in specific application scenarios, and enhance the pertinence and practical application value of the evaluation system.

[0053] Step 6: Comprehensive evaluation of large language models and human experts

[0054] The evaluation results of the large language model and human experts are combined to form the final adaptive evaluation system for the power system. Specifically, the indicator weights generated by the large language model are weighted averaged with the weights determined by human experts based on the analytic hierarchy process (AHP) to ensure that the evaluation system achieves the best balance between comprehensiveness and specificity.

[0055] Through this comprehensive evaluation method, the present invention not only retains the advantages of large language models in data processing and knowledge integration, but also uses the domain experience of human experts to fine-tune and optimize the applicability of specific products or scenarios. The final evaluation system will be used for the comprehensive evaluation of power system control, providing a reliable basis for further optimization decisions and strategy formulation.

[0056] The above is only a preferred embodiment of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for constructing a power system adaptive evaluation system based on LLM and analytic hierarchy process, characterized in that: The following steps are involved: Step 1: Collect corpus and build a knowledge base through multi-source data: After preprocessing the corpus, the structured data in the corpus is represented by knowledge graph technology and stored in a relational database. The unstructured data in the corpus is processed and stored in a NoSQL database. The knowledge base is constructed using the relational database and the NoSQL database. Step 2: Knowledge enhancement and processing based on the RAG framework: The data in the relational database and NoSQL database use the BERT word vector embedding model to convert the text content into embedded knowledge vectors, store them in a unified test corpus, and interact with the large language model; Step 3: Fine-tune and optimize the adaptability of large language models in specialized fields: Fine-tune the selected large language model through continuous task-specific training sets and validation sets, and optimize the parameters of the large language model during the fine-tuning process. At runtime, the user's natural language needs are converted into query vectors through the BERT word vector embedding model. The query vector is similar to the knowledge vector in the vectorized knowledge base, and the knowledge vector with the highest similarity is selected. The vector is used to find the corresponding knowledge entry and passed to the large language model. Step 4: Generation of an adaptive power system evaluation system: Based on user needs and guidance of human intervention prompts, the large language model extracts key evaluation indicators of the power system and generates a multi-level evaluation framework, which includes an evaluation hierarchy structure and initial weights of evaluation indicators at each level. Through multiple rounds of feedback and iterative optimization with users, the evaluation hierarchy structure and weights of each level are further adjusted to ultimately form an adaptive power system evaluation system. Step 5, subjective evaluation by human experts: Invite multiple power system experts to use the evaluation framework and weight distribution obtained in step 4 to conduct comprehensive evaluation and correction. The experts evaluate through scoring and test data and calculate the relative weights of indicators at each level; Step 6: Comprehensive evaluation of large language models and human experts: The weights generated by the large language model in step 4 and the weights obtained by human experts through AHP in step 5 are weighted averaged to obtain the final comprehensive evaluation weights. Based on this weight, the final power system adaptive evaluation system is completed.

2. The method for constructing a power system adaptive evaluation system according to claim 1, characterized in that: In step 1, the preprocessing includes cleaning, formatting and consistency verification.

3. The method for constructing a power system adaptive evaluation system according to claim 1, characterized in that: In step 1, the corpus in the knowledge base includes but is not limited to the basic principles and steps of the AHP and its application examples in power system evaluation, technical documents, standard operation manuals, historically constructed evaluation systems and expert reports.

4. The method for constructing a power system adaptive evaluation system according to claim 1, characterized in that: In step 3, the large language model includes but is not limited to GLM-4.

0.

5. The method for constructing a power system adaptive evaluation system according to claim 1, characterized in that: In step 3, during the fine-tuning of the large language model, an optimization algorithm and a learning rate adjustment strategy are used to ensure rapid convergence and optimization of the parameters of the large language model.

6. The method for constructing a power system adaptive evaluation system according to claim 5, characterized in that: The optimization algorithm adopts Adam or SGD.

7. The method for constructing a power system adaptive evaluation system according to claim 1, characterized in that: In step 3, during the fine-tuning of the large language model, a regularization technique is implemented to prevent the large language model from overfitting.

8. The method for constructing a power system adaptive evaluation system according to claim 1, characterized in that: In step 5, human experts compare each indicator pairwise based on the hierarchical analysis method to construct a judgment matrix.

9. The method for constructing a power system adaptive evaluation system according to claim 1, characterized in that: In step 5, after calculating the relative weights of indicators at each level, a consistency check is performed to ensure the rationality and scientificity of the weight distribution.

10. A power system adaptive evaluation system obtained by the construction method according to any one of claims 1 to 9.

Images