Automatic generation method and device of operation and maintenance test questions, medium and equipment

Abstract

The invention discloses an automatic generation method and device of operation and maintenance test questions, a medium and equipment. The invention belongs to the technical field of computers. The method comprises the following steps: acquiring collected data of a target domain, and constructing a vector knowledge base and a knowledge graph of the target domain; obtaining a question setting demand input by a user, and matching the question setting demand in the vector knowledge base and the knowledge graph to obtain an association vector and an association entity relationship set; determining a first weight and a second weight; according to the association vector, the first weight, the association entity relationship set and the second weight, generating a text block after knowledge fusion; and calculating the confidence of each text block according to the difficulty of each text block, screening the text blocks based on the confidence, and inputting the screened text blocks into the large model to obtain operation and maintenance test questions. According to the scheme, the text blocks after knowledge fusion can be generated based on the vector knowledge base and the knowledge graph, the text blocks are screened and then input into the large model, and a more accurate operation and maintenance test question output result is obtained.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, medium and equipment for automatically generating operation and maintenance test questions. Background Art

[0002] With the rapid advancement of science and technology, operational and maintenance (O&M) requirements are increasing across various fields. For example, in the field of communications network O&M, as user needs become increasingly diverse and network environments become increasingly complex, the demand for network O&M is growing. While some companies have proposed introducing large language models (LLMs) to aid network O&M, LLMs still require extensive testing to assess the intelligence level of network O&M systems. However, traditional test question generation methods rely on manual question bank compilation, which is inefficient and lacks coverage of the latest technological developments, such as 5G / 6G, SDN / NFV. This results in O&M test questions lagging behind actual O&M requirements. Consequently, large language models are unable to effectively solve practical problems in network O&M systems, remaining outdated. Furthermore, the limited number of questions available also hinders comprehensive coverage, leading to a high number of challenges in real-world O&M scenarios. Therefore, the ability to automatically generate timely, high-value O&M test questions for large models in the network O&M field, ensuring that the difficulty, scope, number, and format of the questions meet actual O&M requirements, is a key factor in the practical deployment of large language models. Summary of the Invention

[0003] The present application proposes a method, device, medium and equipment for automatically generating operation and maintenance test questions, which are used to solve many problems such as the limited number of operation and maintenance test questions generated, insufficient knowledge coverage and knowledge lag. The technical solution provided by this application constructs a vector knowledge base and knowledge graph by acquiring data collected from the target field, matches related information and determines weights based on the user's question-setting requirements, generates text blocks, and then filters the text blocks according to the confidence level. After inputting the large model, the output operation and maintenance test questions are obtained. This method can automatically generate large-scale operation and maintenance test questions in the field of network operation and maintenance that are timely and high-value, effectively solving the drawbacks of traditional methods, ensuring that the difficulty, scope, quantity and form of the test questions meet the actual operation and maintenance needs, and providing strong support for the actual deployment and application of large language models in network operation and maintenance systems.

[0004] The present invention provides a method for automatically generating operation and maintenance test questions, the method comprising: Acquire collected data in the target domain, perform semantic segmentation on the collected data, and call a fine-tuned vector model in the target domain to perform vectorization processing to build a vector knowledge base; Perform entity recognition on the collected data based on a pre-built domain dictionary, perform dependency grammar analysis and relationship extraction on the collected data, obtain the relationships between entities, and construct a knowledge graph for the target domain; Obtaining the question-setting requirements input by the user, matching the question-setting requirements in the vector knowledge base and the knowledge graph, and obtaining an associated vector and an associated entity relationship set; Determining a first weight of the vector knowledge base and a second weight of the knowledge graph based on the question setting requirement; Generate a knowledge-fused text block based on the association vector and the first weight, and the associated entity relationship set and the second weight; A preset suppression algorithm is used to determine the confidence of each text block, and text blocks whose confidence meets the set conditions are screened as input data of the large model for the large model to output operation and maintenance test questions.

[0005] Furthermore, the confidence level includes high confidence, medium confidence and low confidence; Filter text blocks whose confidence meets the set conditions as input data for the large model, including: If the confidence of the text block is high confidence or medium confidence, it is determined that the text block meets the set conditions and is used as input data of the large model.

[0006] Furthermore, before inputting the large model, the method further includes: Use the confidence of the text block as the label of the text block; Entering a text block with a label into the macro model; When the large model outputs the operation and maintenance test questions, the operation and maintenance test questions generated by the text blocks with medium confidence labels are marked for users to view.

[0007] Furthermore, a preset suppression algorithm is used to determine the confidence of each text block, including: After obtaining a text block, performing context feature calculation on the text block to obtain a first confidence level; and calculating a second confidence level based on entity node confidence levels and relationship confidence levels in a set of entity relationships associated with the text block; wherein the confidence levels of the entity nodes are pre-set in the knowledge graph; The confidence of the text block is determined according to the first confidence and the second confidence.

[0008] Furthermore, performing context feature calculation on the text block to obtain a first confidence level includes: Extracting causal statements and constraints from the text block and calculating to obtain a first confidence level; The second confidence is calculated based on the entity node confidence and the relationship confidence in the associated entity relationship set of the text block, including: Reading the confidence of the entity node of the associated entity of the text block and the confidence of the relationship between the associated entities; Perform group-by-group calculation based on the confidence of the entity node and the confidence of the relationship to obtain the internal confidence of each group; The group-internal confidences of all groups associated with the current text block are comprehensively calculated to obtain the second confidence of the current text block.

[0009] Furthermore, determining a first weight of the vector knowledge base and a second weight of the knowledge graph based on the question setting requirement includes: Acquire three dimensional information in the question setting requirement through feature extraction; wherein the three dimensional information includes a difficulty level dimension, a question type complexity dimension, and a knowledge point density dimension; The first weight of the vector knowledge base and the second weight of the knowledge graph are determined based on the three dimensional information; wherein the first weight is inversely proportional to the values of the three dimensional information; and the second weight is directly proportional to the values of the three dimensional information.

[0010] Furthermore, the question setting requirement input by the user is obtained, and the question setting requirement is matched in the vector knowledge base and the knowledge graph to obtain an associated vector and an associated entity relationship set, including: Obtaining a question setting requirement input by a user, performing vectorization processing on the question setting requirement, and obtaining a question setting requirement vector; Calculating cosine similarity between the question-asking vector and the vectors in the vector knowledge base to obtain a correlation vector; as well as, Obtain the question requirements input by the user, execute Cypher queries through Neo4j to perform knowledge graph matching, and obtain a set of related entities and the relationships between related entities.

[0011] The present application also provides an automatic generation device for operation and maintenance test questions, the device comprising: A vector knowledge base construction module is used to obtain collected data in the target domain, perform semantic segmentation on the collected data, and call a fine-tuned vector model in the target domain to perform vectorization processing to construct a vector knowledge base; A knowledge graph construction module is used to perform entity recognition on the collected data based on a pre-built domain dictionary, and to perform dependency grammar analysis and relationship extraction on the collected data to obtain the relationships between entities and construct a knowledge graph for the target domain; A matching module is used to obtain the question setting requirements input by the user, match the question setting requirements in the vector knowledge base and the knowledge graph, and obtain an associated vector and an associated entity relationship set; A weight determination module, configured to determine a first weight of the vector knowledge base and a second weight of the knowledge graph based on the question setting requirement; A text block generation module, configured to generate a knowledge-fused text block based on the association vector and the first weight, and the associated entity relationship set and the second weight; The operation and maintenance test question output module is used to use a preset suppression algorithm to determine the confidence of each text block, and to screen text blocks whose confidence meets the set conditions as input data for the large model, so that the large model can output the operation and maintenance test questions.

[0012] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for automatically generating operation and maintenance test questions as described above.

[0013] An embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the automatic generation method of operation and maintenance test questions as described above when executing the computer program.

[0014] The embodiment of the present application adopts the following technical solutions: acquiring collected data of a target domain, performing semantic segmentation on the collected data, and calling a fine-tuned vector model of the target domain for vectorization processing to construct a vector knowledge base; performing entity recognition on the collected data based on a pre-built domain dictionary, and performing dependency grammar analysis and relationship extraction on the collected data to obtain the relationship between entities and construct a knowledge graph of the target domain; acquiring the question-setting requirements input by the user, matching the question-setting requirements in the vector knowledge base and the knowledge graph to obtain an associated vector and an associated entity relationship set; determining a first weight of the vector knowledge base and a second weight of the knowledge graph based on the question-setting requirements; generating a text block after knowledge fusion based on the associated vector and the first weight, as well as the associated entity relationship set and the second weight; using a preset suppression algorithm to determine the confidence of each text block, and screening text blocks whose confidence meets the set conditions as input data of the large model for the large model to output operation and maintenance test questions.

[0015] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: This solution constructs a vector knowledge base and knowledge graph in the target field. When the user's question-setting requirements are obtained, it can automatically match and generate fused text blocks. At the same time, after the text blocks are generated, they are screened to obtain text blocks with qualified confidence levels as input data for the large model to generate operation and maintenance test questions based on the large model. This setting of this solution can improve the accuracy of the operation and maintenance test questions output by the large model, avoid the situation where the generated operation and maintenance test questions are unusable due to the low confidence level of the input data of the large model, and reduce the workload of operation and maintenance personnel. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A flowchart of the method for automatically generating operation and maintenance test questions provided in Example 1 of the present application; Figure 2 This is a schematic diagram of the structure of the automatic generation device for operation and maintenance test questions provided in Example 3 of the present application; Figure 3 A schematic structural diagram of an electronic device provided in Example 4 of the present application. DETAILED DESCRIPTION

[0017] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0018] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0019] Example 1 Figure 1 This is a flow chart of the method for automatically generating operation and maintenance test questions provided in Example 1 of this application. Figure 1 As shown, the method includes: S11, acquiring collected data of the target domain, performing semantic segmentation on the collected data, and calling a fine-tuned vector model of the target domain to perform vectorization processing to build a vector knowledge base; In this solution, the target domain can be information and communications network operations and maintenance, but can also be expanded to other vertical sectors such as electricity and finance. Collected data refers to knowledge in multiple modalities, including text, slides, and videos, and encompasses heterogeneous data formats such as textbooks, manuals, and case studies. This solution can collect data using adaptive crawling algorithms or manually.

[0020] Semantic segmentation refers to a recursive segmentation algorithm based on semantic units. It divides the text into semantic units, dynamically adjusts the segmentation boundaries by calculating the semantic density index through a sliding window, and adopts a bidirectional overlap compensation mechanism to maintain semantic coherence.

[0021] After obtaining the semantic segmentation results, they can be vectorized using a fine-tuned vector model. This fine-tuned vector model can be acquired through enhanced learning of the semantic representation capabilities of specialized terms such as communication protocol terminology and device models. Converting text to a vector representation allows the fine-tuned vector model to generate paragraph- and document-level semantic vectors, facilitating computer understanding and processing of semantic information. After constructing a vector knowledge base, a database of vectorized communications operations domain knowledge can be stored for subsequent retrieval and matching.

[0022] This solution can extract text information from multi-modal data through an adaptive crawling algorithm combined with optical character recognition (OCR) and automatic speech recognition (ASR) technologies, and integrate domain dictionaries to achieve semantic-level deduplication and solve the compatibility problem of heterogeneous data formats.

[0023] S12, performing entity recognition on the collected data based on a pre-built domain dictionary, and performing dependency grammar analysis and relationship extraction on the collected data to obtain relationships between entities and construct a knowledge graph of the target domain; A domain dictionary can be a pre-built dictionary containing domain-specific vocabulary such as communication protocol terminology, device models, and fault codes, used to assist in entity recognition. Named Entity Recognition (NER) refers to identifying entities with specific meanings from collected data, such as device models, protocol terminology, and fault codes. Specifically, these can be 5G base stations, SDN controllers, etc. Dependency parsing is the grammatical structure analysis of sentences to determine the dependency relationships between words, such as subject-verb relationships, verb-object relationships, etc. Relationship extraction refers to extracting the association relationships between entities from text, such as dependency relationships, fault associations, and protocol hierarchies. Logical connections between entities, such as the connection dependency between routers and switches, the fault code 0x123 corresponding to device overheating, etc.

[0024] The knowledge graph is a structured knowledge network stored in the form of triples, namely entity-relationship-entity, which is used to accurately express the explicit logical relationships between entities in the communication operation and maintenance field.

[0025] This solution analyzes the sentence structure of collected data, extracts dependencies between words, and extracts a set of relationships between entities from the text based on the results of dependency grammar analysis. Alternatively, it uses entity recognition and relationship extraction to obtain a set of entities and their relationships. Furthermore, the entities and their relationships are stored as triples in a knowledge graph database, forming a knowledge graph for the target domain.

[0026] S13, obtaining a question setting requirement input by the user, matching the question setting requirement in the vector knowledge base and the knowledge graph, and obtaining an associated vector and an associated entity relationship set; The question requirement is the specific requirements for test question generation entered by the user, including difficulty level, question type, knowledge point range, and number. Once the question requirement is obtained, the text block vectors in the vector knowledge base that are semantically related to the question requirement can be obtained. The associated entity relationship set refers to the entities and relationships in the knowledge graph that are related to the user's question requirement.

[0027] In this solution, users can enter the following text as a requirement: Generate five multiple-choice questions about 5G troubleshooting, of medium difficulty. Then, in the vector knowledge base channel, the FAISS vector library and sentence-transformers model are used to calculate the cosine similarity between the user's query and the text blocks in the vector library, matching the most similar relevant knowledge block vectors. In the knowledge graph channel, a Cypher query is executed through Neo4j to match entities and relationships in the knowledge graph based on the keywords in the requirement. This matching process yields the associated vectors and associated entity relationship sets relevant to the requirement.

[0028] S14, determining a first weight of the vector knowledge base and a second weight of the knowledge graph based on the question setting requirement; Among them, the first weight is the importance of the vector knowledge base in test question generation, and the second weight is the importance of the knowledge graph in test question generation.

[0029] This technical solution uses a feature extractor to analyze three key dimensions of the question requirement: difficulty level, question type complexity, and knowledge point density. It then uses a sigmoid activation function to dynamically calculate the weights of the vector knowledge base and the knowledge graph. For example, if the requirement is a complex reasoning question, the logical relationships in the knowledge graph will be given a higher weight; if the requirement is a multiple-choice question involving fault cause analysis, the semantic generalization of the vector knowledge base will be given a higher weight.

[0030] S15, generating a knowledge-fused text block based on the association vector and the first weight, and the associated entity relationship set and the second weight; Among them, the text block is generated based on the associated information and weights. It should be noted that this has not yet been verified for confidence and cannot be used as input data for the large model.

[0031] This solution can fuse the association vector and the association entity relationship to obtain a text block. The fusion method here not only takes into account the data sources of the two knowledge bases, but also can fuse based on weights, thereby improving the matching degree between the fused text block and the subsequent question-setting requirements, and avoiding the phenomenon that the fused knowledge has a simple knowledge description, and the question-setting requirements require complex case questions, which makes it difficult for the text block input into the large model to support the question-setting requirements.

[0032] Fusion calculation involves integrating the semantic information of the association vector and the logical structure of the entity relationships, based on weighted proportions, into a unified knowledge input for LLM test question generation. The text block is the text description generated after the fusion calculation, containing the knowledge points, logical relationships, and semantic context required for the test question, such as a text framework summarizing the steps for troubleshooting a 5G base station connection failure between the core network. The text block content corresponding to the association vector is weighted according to the first weight. For example, a first weight of 0.7 retains 70% of the semantic information. The associated entity relationship set is converted into logical constraints according to the second weight. For example, a second weight of 0.8 mandates that the test question contain a device-fault-solution relationship chain. Further feature concatenation is performed, such as concatenating the weighted text content with the structured relationships into a fused knowledge input. Combining the above example, a text snippet describing a 5G base station connection failure and a base station-to-core network connection dependency can be used as input data for the large model.

[0033] S16, using a preset suppression algorithm to determine the confidence of each text block, and screening the text blocks whose confidence meets the set conditions as input data of the large model for the large model to output operation and maintenance test questions.

[0034] Among them, the text block is the text content formed by splicing the text fragments in the vector knowledge base and the entity relationships in the knowledge graph according to the weights.

[0035] The preset suppression algorithm is an algorithmic model used to quantify the accuracy of text blocks. It suppresses large model hallucinations through multi-dimensional constraints to ensure the accuracy of input knowledge. Specifically, it can detect knowledge conflicts in text blocks, such as entity relationships that violate domain common sense, assess semantic coherence, such as whether the text is logically consistent, and filter out low-quality or outdated information.

[0036] Confidence is an indicator that measures the reliability of a text block when it is used as input to a large model. Its value range is [0,1]. Specifically, it can be measured by semantic consistency, knowledge authority, domain adaptability, etc.

[0037] The set condition is a preset confidence threshold used to determine whether a text block meets the requirements of the large model input. For example, for difficult questions, a higher threshold, such as ≥0.8, can be set to ensure knowledge accuracy. For basic questions, a lower threshold, such as ≥0.6, can be set to balance knowledge coverage.

[0038] The large model's input data consists of a collection of text blocks that have been screened and confidence-qualified. This serves as direct knowledge input for the large language model to generate operations and maintenance test questions. This provides the LLM with integrated, confidence-qualified, high-quality knowledge, preventing test illusions caused by knowledge errors. The large model generates test questions in the information and communications network operations and maintenance field based on this input data, including multiple-choice, question-and-answer, and case analysis questions.

[0039] This technical solution, by building a dynamic heterogeneous knowledge fusion framework and combining RAG technology with dual-channel retrieval of a vector knowledge base and knowledge graphs, enables the automated generation of test questions for information and communication network operations and maintenance. Adaptive weight allocation and a causal suppression algorithm improve the accuracy of test questions. The system supports user customization based on difficulty and question type, covering the latest technological developments such as 5G / 6G, SDN / NFV, and ensuring efficient question generation and comprehensive knowledge coverage. The modular architecture design offers cross-domain adaptability and can be quickly adapted to other fields such as electricity and finance, providing key support for intelligent evaluation of large language models (LLMs) in specialized fields.

[0040] In one embodiment, optionally, obtaining a question setting requirement input by a user, matching the question setting requirement in the vector knowledge base and the knowledge graph, and obtaining an associated vector and an associated entity relationship set include: Obtaining a question setting requirement input by a user, performing vectorization processing on the question setting requirement, and obtaining a question setting requirement vector; Calculating cosine similarity between the question-asking vector and the vectors in the vector knowledge base to obtain a correlation vector; as well as, Obtain the question requirements input by the user, execute Cypher queries through Neo4j to perform knowledge graph matching, and obtain a set of related entities and the relationships between related entities.

[0041] Vectorization converts natural language requirements into computer-computable numerical vectors. A semantic embedding model is then used to extract semantic features from the text. The resulting high-dimensional numerical vector represents the semantic information of the user requirement and is used to calculate similarity with vectors in the vector knowledge base. Cosine similarity measures the directional similarity between two vectors, ranging from -1 to 1. Higher values indicate closer semantics. This quantifies the relevance between the requirement and the knowledge blocks in the vector knowledge base. After comparison, text block vectors in the vector knowledge base whose cosine similarity with the requirement vector exceeds a threshold are identified as associated vectors.

[0042] On the knowledge graph side, Neo4j is a graph database management system used to store and query knowledge graphs constructed in the form of triples, supporting efficient graph-structured retrieval. Cypher query is Neo4j's declarative query language, used to retrieve specific entities and relationships in the knowledge graph, such as querying the association between SDN controllers and fault codes. Entities in the knowledge graph that are relevant to the question requirements are associated entities, such as SDN controller, NFV architecture, and fault type 0x123. The associated entity relationship set is the set of relationships between entities obtained by the query, such as SDN controller → dependency → NFV infrastructure, fault code 0x123 → correspondence → controller overload, and so on.

[0043] This technical solution achieves dual-channel knowledge retrieval by matching user question requests simultaneously across a vector knowledge base and a knowledge graph. The vector knowledge base matching utilizes a semantic embedding model to capture the implicit semantics of the request, ensuring breadth of knowledge coverage; the knowledge graph matching extracts explicit logical relationships through structured queries, ensuring accuracy and logical rigor. Combining the matching results from these two channels quickly locates operational knowledge highly relevant to the request, providing comprehensive and accurate knowledge support for subsequent question generation, addressing the fuzzy and inefficient knowledge matching issues of traditional methods.

[0044] In one embodiment, optionally, determining the first weight of the vector knowledge base and the second weight of the knowledge graph based on the question setting requirement includes: Acquire three dimensional information in the question setting requirement through feature extraction; wherein the three dimensional information includes a difficulty level dimension, a question type complexity dimension, and a knowledge point density dimension; The first weight of the vector knowledge base and the second weight of the knowledge graph are determined based on the three dimensional information; wherein the first weight is inversely proportional to the values of the three dimensional information; and the second weight is directly proportional to the values of the three dimensional information.

[0045] Among them, feature extraction uses natural language processing (NLP) technology to extract key features from the user's question request text, which is used to quantify the constraints of test question generation. The difficulty level dimension refers to the difficulty of the test questions, which can be divided into elementary, such as basic concept recognition, intermediate, such as simple fault analysis, and advanced, such as comprehensive reasoning in complex scenarios. The question type complexity dimension is a measure of the question type's requirements for logical reasoning or knowledge integration ability. For example, the question type complexity of multiple-choice questions is less than that of fill-in-the-blank questions, and the question type complexity of fill-in-the-blank questions is less than that of case analysis questions, and so on. The knowledge point density dimension refers to the number of professional knowledge points involved in the test questions and the depth of association. If a question contains more than 5 knowledge points, it is determined that the knowledge point density is high.

[0046] The first weight represents the contribution of the vector knowledge base to question generation, ranging from 0 to 1, and is inversely proportional to the three dimensions. The second weight represents the contribution of the knowledge graph to question generation, ranging from 0 to 1, and is directly proportional to the three dimensions. Specifically, the higher the difficulty, question type complexity, and knowledge point density, the lower the first weight. Higher values for the three dimensions increase the second weight.

[0047] For example, if the question requirement is a multiple-choice question of elementary difficulty, involving only basic knowledge points of TCP / IP, and the values of the three dimensions are low, then the first weight is high, such as 0.8, and the question generation is dominated by the vector knowledge base; if the requirement is a case analysis question of advanced difficulty, involving 5G network slicing and SDN collaborative troubleshooting, and the values of the three dimensions are high, then the second weight is high, such as 0.8, and it is dominated by the knowledge graph.

[0048] Through dynamic weight setting, this solution can quantify the complexity of question-setting requirements through the weight distribution mechanism, realize the adaptive collaboration of the vector knowledge base and the knowledge graph, and dynamically adjust the knowledge source according to real-time needs. It is suitable for various user needs and ensures that the generated questions are in line with the technical attempts in the target field and meet the actual usage needs of users.

[0049] In one embodiment, optionally, the method further includes: The constructed vector knowledge base and the knowledge graph of the target domain are dynamically updated through retrieval enhancement generation technology.

[0050] Retrieval-augmented Generation (RAG) is a technical framework that combines external knowledge base retrieval with the generation capabilities of large language models. In this solution, RAG enhances the accuracy of LLM-generated test questions by retrieving the latest knowledge from vector knowledge bases and knowledge graphs in real time.

[0051] This technical solution can monitor industry data sources through crawling algorithms and set technology dynamic trigger thresholds to ensure that knowledge can be collected and updated after the release of new technologies. Compared with traditional manual sorting, the efficiency is significantly improved, so that the test question generation system can keep up with the evolution of communication technology. The time difference between operation and maintenance test questions and actual operation and maintenance needs is within a controllable range, thereby improving the accuracy and completeness of knowledge.

[0052] In one embodiment, optionally, the confidence level includes high confidence level, medium confidence level, and low confidence level; Filter text blocks whose confidence meets the set conditions as input data for the large model, including: If the confidence of the text block is high confidence or medium confidence, it is determined that the text block meets the set conditions and is used as input data of the large model.

[0053] Among them, high confidence refers to the confidence interval where the text block's knowledge accuracy and semantic coherence are extremely high, for example, it is set to [0.8, 1.0]. High-confidence text blocks have basically no knowledge conflicts and can have high semantic consistency scores. In addition, the knowledge source of high-confidence text blocks is industry standard documents or authoritative operation and maintenance manuals. Medium confidence refers to the confidence interval where the text block's knowledge is basically reliable but has slight uncertainty, which can be set to [0.6, 0.8]. Medium confidence may have 1-2 non-core knowledge conflicts, and the semantic consistency score can be close to full marks. In addition, its knowledge source can refer to the manufacturer's technical white papers or recent operation and maintenance cases. Low confidence refers to the confidence interval where the text block's knowledge reliability is insufficient. Low-confidence text blocks may have core knowledge conflicts and poor semantic consistency. Its knowledge source may include forum posts, non-authoritative self-media, or technical documents that were published a long time ago.

[0054] This technical solution can prioritize the confidence of each text block. For example, the confidence of text block A is 0.92, that is, high confidence; the confidence of text block B is 0.71, that is, medium confidence; and the confidence of text block C is 0.55, that is, low confidence.

[0055] When selecting eligible text blocks, low-confidence blocks can be filtered out: blocks with a confidence level less than 0.6 can be directly excluded to prevent the large model from outputting incorrect operation and maintenance questions. High- and medium-confidence blocks can be retained as input data. Furthermore, these blocks can be differentiated based on the output question types. For example, high-confidence blocks can be prioritized for complex question types, such as case analysis questions, while medium-confidence blocks can be marked as requiring verification and used to generate basic question types, such as multiple-choice questions. Low-confidence blocks can be omitted from the large model's input data.

[0056] This technical solution, through confidence graded screening, can use high and medium confidence graded screening to ensure the accuracy of complex test questions, improve the efficiency of generating basic test questions by retaining medium confidence text blocks, and can effectively eliminate the illusion phenomenon of generating operation and maintenance test questions for large models by strictly excluding low confidence knowledge.

[0057] In one embodiment, optionally, before inputting the large model, the method further includes: Use the confidence of the text block as the label of the text block; Entering a text block with a label into the macro model; When the large model outputs the operation and maintenance test questions, the operation and maintenance test questions generated by the text blocks with medium confidence labels are marked for users to view.

[0058] Among them, labels can be used to mark and distinguish input data. For example, text blocks with medium confidence can be marked with labels to provide users with suggestions for checking before using the questions after the corresponding test questions are generated. Text blocks with high confidence can be left unmarked or marked accordingly to distinguish them from questions generated from text blocks with medium confidence.

[0059] When a labeled text block is fed into the large model, it includes not only the knowledge content but also the labels. The large model can then label the generated test questions accordingly based on the labels of the input knowledge. For example, questions generated by high-confidence knowledge are directly output without special labels, while questions generated by medium-confidence knowledge are automatically labeled to provide user guidance.

[0060] This technical solution can automatically distinguish the confidence of knowledge and label relatively high-risk input data, that is, text blocks with medium confidence, to reduce the workload of manual review.

[0061] In one embodiment, optionally, a preset suppression algorithm is used to determine the confidence level of each text block, including: After obtaining a text block, performing context feature calculation on the text block to obtain a first confidence level; and calculating a second confidence level based on entity node confidence levels and relationship confidence levels in a set of entity relationships associated with the text block; wherein the confidence levels of the entity nodes are pre-set in the knowledge graph; The confidence of the text block is determined according to the first confidence and the second confidence.

[0062] The first confidence level is a reliability indicator calculated based on the contextual features of the question. It measures the consistency between the semantics of the test text and the knowledge in the operation and maintenance field. The value range is [0,1]. The higher the value, the more reasonable the text logic.

[0063] The entity node confidence is the pre-set reliability weight of each entity node in the knowledge graph. The second confidence is a comprehensive indicator calculated based on the confidence of the entity nodes in the set of associated entity relationships. It measures the accuracy of the structured knowledge involved in the test questions and has a value range of [0,1].

[0064] The first confidence level can be specifically extracted based on causality and constraints. Natural language processing technology is used to extract causal statements and constraints from the question text. Semantic matching is performed between the extracted context features and the standard operation and maintenance knowledge text in the vector knowledge base to verify the rationality of the causal logic. The initial confidence level calculation is based on the semantic similarity of the matching rules and context features, and the Sigmoid function is used to calculate the first confidence level.

[0065] The second confidence is the aggregated confidence of the entity nodes. Specifically, the entity node confidence may be obtained by extracting a preset confidence of each entity node.

[0066] The comprehensive confidence level is obtained by combining the first confidence level and the second confidence level in a preset ratio, such as 50% each, to obtain the final confidence level of the text block. The formula example is: Comprehensive confidence = α × first confidence + (1-α) × second confidence Among them, α is a balance parameter, which is adjusted according to the complexity of the question type.

[0067] The technical logic behind confidence calculation can be divided into two levels: semantics and logic. The semantic level ensures that the natural language expression of the question text conforms to the semantic standards of the operation and maintenance domain, preventing the LLM from generating syntactically correct but semantically incorrect content. The logical level verifies the accuracy of the structured knowledge involved in the question through the confidence of the entity relationships in the knowledge graph.

[0068] In this solution, the weights of the first and second confidence levels can be automatically adjusted for different question types. For example, multiple-choice or true-or-false questions rely more on the semantic coherence of the text, so the first confidence level can be set higher. Essay questions or case analysis questions rely more on the logical accuracy of the knowledge graph, so the second confidence level can be set higher.

[0069] This technical solution can switch different weights in the process of question generation based on actual question-setting requirements, so that the test points of the questions match the current question types. It can provide key control rules for question generation and ensure the accuracy of the generated questions.

[0070] In one embodiment, optionally, performing context feature calculation on the text block to obtain a first confidence level includes: Extracting causal statements and constraints from the text block and calculating to obtain a first confidence level; The second confidence is calculated based on the entity node confidence and the relationship confidence in the associated entity relationship set of the text block, including: Reading the confidence of the entity node of the associated entity of the text block and the confidence of the relationship between the associated entities; Perform group-by-group calculation based on the confidence of the entity node and the confidence of the relationship to obtain the internal confidence of each group; The group-internal confidences of all groups associated with the current text block are comprehensively calculated to obtain the second confidence of the current text block.

[0071] A causal statement is a natural language fragment within a text block that expresses causal logic. For example, "An SDN controller misconfiguration leads to abnormal network traffic" consists of two parts: the cause: configuration error and the result: abnormal traffic. Constraints are descriptions of the scope of application of the knowledge in the question, such as "In 5G standalone networking mode, when the base station load exceeds 80%." These constraints are used to limit the applicable scenarios of the knowledge point.

[0072] Causal logic verification involves semantically matching the extracted causal pairs with the standard causal knowledge in the vector knowledge base and calculating cosine similarity. Constraint coverage evaluates whether the constraints have corresponding entity relationships in the knowledge graph.

[0073] Entity node confidence is the pre-annotated reliability value of each entity node in the knowledge graph. Relationship confidence is the pre-annotated reliability value of the relationships between entities in the knowledge graph. Groups are groupings of sets of related entity relationships based on their semantic types. In-group confidence is a comprehensive confidence indicator for each group of entity relationships, reflecting the reliability of the knowledge in that group.

[0074] Example: The Failure Cause group has 2 relations: Relation 1: Confidence = 0.9 × 0.85 = 0.765; Relation 2: Confidence = 0.8 × 0.8 = 0.64; Intraclass reliability = (0.765 + 0.64) / 2 = 0.7025; The second confidence level is obtained by summing the intra-group confidence levels of all groups weighted by group importance.

[0075] This technical solution verifies the first confidence level using the dual characteristics of causality and constraints to ensure that the semantic logic of the questions conforms to operation and maintenance specifications. It also uses grouping calculations to differentiate the importance of different types of knowledge for the second confidence level. For example, the fault cause group is given a higher weight than the solution group, allowing for more rigorous verification of core knowledge points. This effectively improves the quality of questions generated by LLM, especially for complex questions.

[0076] Example 2 This embodiment is a preferred embodiment of the present application. In order to enable those skilled in the art to more clearly understand the technical solution of the present application, this preferred embodiment provides some specific implementation methods. It can be understood that the scope of protection of the present application includes but is not limited to the implementation methods provided in this embodiment.

[0077] This solution proposes an automatic generation method for large-scale operation and maintenance test questions in the field of information and communication network operation and maintenance based on LLM. It can automatically generate timely and high-value large-scale operation and maintenance test questions in the field of network operation and maintenance according to user testing requirements, and realize user customization of the difficulty, scope, quantity and form of the test questions. Ultimately, it can realize a complete technical solution in the field of communication operation and maintenance, from unstructured data collection to semantic preservation processing, and from domain optimization vectorization to dynamic and controllable generation.

[0078] 1. The large-scale test model proactively collects knowledge from multiple modalities, including text, slides, and videos. To address the unstructured nature of data in the communications operations and maintenance field, we developed an adaptive crawling algorithm based on a rules engine. This algorithm uses optical character recognition (OCR) and automatic speech recognition (ASR) technologies for text recognition and extraction. It integrates domain dictionaries (such as communication protocol terminology and device model codes) to achieve semantic-level deduplication and address the compatibility issues of heterogeneous data formats, including textbooks, manuals, and case studies.

[0079] 2. The obtained text is segmented based on the semantic unit recursive segmentation algorithm (SU-RSA). The semantic density index (SDI) within the text block is calculated through a sliding window. The segmentation boundary is dynamically adjusted, and a two-way overlap compensation mechanism is used to maintain the semantic coherence of the text, ensuring that adjacent text blocks have some overlapping content to reduce the possibility of information loss.

[0080] 3. The first-level vectorization uses a lightweight model to quickly generate paragraph-level vectors for coarse-grained recall. The second-level vectorization uses a deep model to generate document-level semantic vectors, improving accuracy. The vector model is fine-tuned based on O&M domain corpus to enhance the semantic representation of specialized terminology and build a vector knowledge base for the communications O&M domain.

[0081] 4. Extract entities (such as device models, protocol terms, and fault codes) and relationships (such as dependencies, fault associations, and protocol hierarchies). Combine the rule engine with deep learning models (such as BERT-CRF) for entity recognition (NER). Use dependency parsing and relation extraction models (such as REBEL) to construct entity relationships and build a knowledge graph for the communications operations and maintenance domain.

[0082] 5. Load the constructed vector database and knowledge graph, and build a RAG chain through the Retrieval-Augmented Generation (RAG) technology to achieve knowledge enhancement of LLM.

[0083] 6. The user inputs the question requirement independently, and the system vectorizes the input text and converts the query text into the corresponding query embedding q so that it can be compared with the vectors in the existing vector database for similarity.

[0084] 7. Perform dual-channel parallel retrieval based on user needs. The vector knowledge base channel uses the FAISS vector library and, based on the sentence-transformers / paraphrase-multilingual-MiniLM-L12-v2 model, calculates the cosine similarity between the user query content and text blocks in the professional vector library for communication network operation and maintenance, and returns the five relevant knowledge blocks with the highest similarity.

[0085] 8. At the same time, the knowledge graph channel executes Cypher queries through Neo4j to obtain entities and corresponding relationships in the graph.

[0086] 9. Dynamically integrate the knowledge obtained from the retrieval vector library and the knowledge graph, analyze the three key dimensions of the query through the feature extractor: difficulty level, question type complexity, and knowledge point density, and use the Sigmoid activation function to dynamically calculate weights to achieve adaptive weight distribution.

[0087] 10. Design a suppression algorithm based on causal reasoning to suppress the generation of large model hallucinations. Set the confidence threshold according to the difficulty of the question and establish a verification benchmark. Then, extract causal statements and constraints from the text and structured causal relationships from the knowledge graph. Calculate the initial confidence of the causal statement based on matching rules and context features, and perform weighted calculation based on the confidence of the knowledge graph nodes. Perform threshold decision verification on the calculated comprehensive confidence, explicitly suppress statements below the threshold, and enhance the display of high-confidence causal chains. Finally, integrate the verification conclusions and confidence information into the retrieval results to provide purified enhanced context for LLM.

[0088] 11. Design prompt templates for automatic test question generation to prevent the model from generating irrelevant or inaccurate content when lacking contextual information. This alleviates the model's illusions to a certain extent. Through the parameterized design of prompt words, the difficulty, scope, number, and form of the test questions can be customized by users, ensuring that the final responses generated by the LLM meet user needs and improve the quality of the questions.

[0089] 12. RAG technology, knowledge graph and prompt words enhance LLM. LLM combines the professional knowledge in the knowledge base with the language generation ability inherent in the model to automatically generate questions.

[0090] This technical solution has the following technical effects: Dynamic heterogeneous knowledge fusion framework: Apply the RAG framework to test question generation, integrate structured question banks, unstructured operation and maintenance documents, and voice and video materials through hybrid retrieval strategies, and solve the data island problem of traditional methods. Fusion of knowledge bases and knowledge graphs: Knowledge bases excel at storing unstructured text (such as manuals and case studies), preserving semantic ambiguity through vectorization. Knowledge graphs precisely express explicit logical relationships between entities in the form of triples. The combination of the two achieves the dual expression of "semantic generalization and logical precision."

[0091] Dual-channel parallel retrieval mechanism: performs parallel retrieval of the vector knowledge base and the knowledge graph, and after obtaining the retrieval results, performs adaptive weight allocation based on the complexity of the question type, difficulty level, knowledge point density, etc.

[0092] Causal suppression algorithm: Suppressing LLM illusion by quantifying knowledge reliability.

[0093] Cross-domain adaptability and expansion: The modular architecture design supports rapid adaptation to other vertical fields (such as electricity, finance, etc.), and cross-industry applications can be achieved by simply replacing the knowledge base and domain dictionary.

[0094] Example 3 Figure 2 This is a schematic diagram of the structure of the automatic generation device for operation and maintenance test questions provided in Example 3 of this application. Figure 2 As shown, the device includes: A vector knowledge base construction module 21 is used to obtain collected data in a target domain, perform semantic segmentation on the collected data, and call a fine-tuned vector model in the target domain to perform vectorization processing to construct a vector knowledge base; A knowledge graph construction module 22 is configured to perform entity recognition on the collected data based on a pre-built domain dictionary, and to perform dependency parsing and relationship extraction on the collected data to obtain relationships between entities and construct a knowledge graph for the target domain; A matching module 23 is used to obtain a question setting requirement input by a user, match the question setting requirement in the vector knowledge base and the knowledge graph, and obtain an associated vector and an associated entity relationship set; A weight determination module 24 is configured to determine a first weight of the vector knowledge base and a second weight of the knowledge graph based on the question setting requirement; A text block generation module 25 is configured to generate a knowledge-fused text block based on the association vector and the first weight, and the associated entity relationship set and the second weight; The operation and maintenance question output module 26 is used to determine the confidence of each text block using a preset suppression algorithm, and screen text blocks whose confidence meets the set conditions as input data of the large model for the large model to output the operation and maintenance question.

[0095] This device can execute the method for automatically generating operation and maintenance test questions provided in the first embodiment above, and has corresponding functional units and beneficial effects, which will not be described in detail here.

[0096] Example 4 Those skilled in the art will appreciate that embodiments of the present invention may provide methods, systems, or computer program products. Therefore, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0097] Therefore, the present application also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any embodiment of the present application.

[0098] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0099] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0100] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0101] Further, Figure 3 This is a schematic diagram of the structure of an electronic device provided in Example 4 of this application. Figure 3 As shown, the present application also proposes an electronic device (or computing device), including a processor 11, a memory 12, and a computer program stored in the memory 12 and executable on the processor 11, wherein the processor 11 implements the method described in any embodiment of the present application when executing the computer program.

[0102] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. Memory may include non-permanent storage in computer-readable media, random access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium. Computer-readable media, including both permanent and non-permanent, removable and non-removable media, may be implemented using any method or technology for information storage. Information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change RAM (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition in this article, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0103] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0104] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A method for automatically generating operation and maintenance test questions, characterized in that: The method comprises: Acquire collected data in the target domain, perform semantic segmentation on the collected data, and call a fine-tuned vector model in the target domain to perform vectorization processing to build a vector knowledge base; Perform entity recognition on the collected data based on a pre-built domain dictionary, perform dependency grammar analysis and relationship extraction on the collected data, obtain the relationships between entities, and construct a knowledge graph for the target domain; Obtaining the question-setting requirements input by the user, matching the question-setting requirements in the vector knowledge base and the knowledge graph, and obtaining an associated vector and an associated entity relationship set; Determining a first weight of the vector knowledge base and a second weight of the knowledge graph based on the question setting requirement; Generate a knowledge-fused text block based on the association vector and the first weight, and the associated entity relationship set and the second weight; A preset suppression algorithm is used to determine the confidence of each text block, and text blocks whose confidence meets the set conditions are screened as input data of the large model for the large model to output operation and maintenance test questions.

2. The method according to claim 1, characterized in that The confidence level includes high confidence, medium confidence and low confidence; Filter text blocks whose confidence meets the set conditions as input data for the large model, including: If the confidence of the text block is high confidence or medium confidence, it is determined that the text block meets the set conditions and is used as input data of the large model.

3. The method according to claim 1, characterized in that Before inputting the large model, the method further includes: Use the confidence of the text block as the label of the text block; Entering a text block with a label into the macro model; When the large model outputs the operation and maintenance test questions, the operation and maintenance test questions generated by the text blocks with medium confidence labels are marked for users to view.

4. The method according to claim 1, wherein The confidence level of each text block is determined using a preset suppression algorithm, including: After obtaining a text block, performing context feature calculation on the text block to obtain a first confidence level; and calculating a second confidence level based on entity node confidence levels and relationship confidence levels in a set of entity relationships associated with the text block; wherein the confidence levels of the entity nodes are pre-set in the knowledge graph; The confidence of the text block is determined according to the first confidence and the second confidence.

5. The method according to claim 4, characterized in that Calculating a context feature of the text block to obtain a first confidence level includes: Extracting causal statements and constraints from the text block and calculating to obtain a first confidence level; The second confidence is calculated based on the entity node confidence and the relationship confidence in the associated entity relationship set of the text block, including: Reading the confidence of the entity node of the associated entity of the text block and the confidence of the relationship between the associated entities; Perform group-by-group calculation based on the confidence of the entity node and the confidence of the relationship to obtain the internal confidence of each group; The group-internal confidences of all groups associated with the current text block are comprehensively calculated to obtain the second confidence of the current text block.

6. The method according to claim 1, characterized in that Determining a first weight of the vector knowledge base and a second weight of the knowledge graph based on the question setting requirement includes: Acquire three dimensional information in the question setting requirement through feature extraction; wherein the three dimensional information includes a difficulty level dimension, a question type complexity dimension, and a knowledge point density dimension; The first weight of the vector knowledge base and the second weight of the knowledge graph are determined based on the three dimensional information; wherein the first weight is inversely proportional to the values of the three dimensional information; and the second weight is directly proportional to the values of the three dimensional information.

7. The method according to claim 1, characterized in that Obtain the question-setting requirements input by the user, match the question-setting requirements in the vector knowledge base and the knowledge graph, and obtain the associated vector and the associated entity relationship set, including: Obtaining a question setting requirement input by a user, performing vectorization processing on the question setting requirement, and obtaining a question setting requirement vector; Calculating cosine similarity between the question-asking vector and the vectors in the vector knowledge base to obtain a correlation vector; as well as, Obtain the question requirements input by the user, execute Cypher queries through Neo4j to perform knowledge graph matching, and obtain a set of related entities and the relationships between related entities.

8. An automatic generation device for operation and maintenance test questions, characterized in that: The device comprises: A vector knowledge base construction module is used to obtain collected data in the target domain, perform semantic segmentation on the collected data, and call a fine-tuned vector model in the target domain to perform vectorization processing to construct a vector knowledge base; A knowledge graph construction module is used to perform entity recognition on the collected data based on a pre-built domain dictionary, and to perform dependency grammar analysis and relationship extraction on the collected data to obtain the relationships between entities and construct a knowledge graph for the target domain; A matching module is used to obtain the question setting requirements input by the user, match the question setting requirements in the vector knowledge base and the knowledge graph, and obtain an associated vector and an associated entity relationship set; A weight determination module, configured to determine a first weight of the vector knowledge base and a second weight of the knowledge graph based on the question setting requirement; A text block generation module, configured to generate a knowledge-fused text block based on the association vector and the first weight, and the associated entity relationship set and the second weight; The operation and maintenance test question output module is used to use a preset suppression algorithm to determine the confidence of each text block, and to screen text blocks whose confidence meets the set conditions as input data for the large model, so that the large model can output the operation and maintenance test questions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

Images