Large language model implicit knowledge extraction method and device based on knowledge engineering technology mapping, equipment and medium

Abstract

The embodiment of the application discloses a large language model implicit knowledge extraction method and device based on knowledge engineering technology mapping, equipment and medium, and relates to the technical field of natural language processing. The method comprises the following steps: receiving an input question; performing knowledge type identification on the input question to obtain a knowledge type corresponding to the input question; determining a target follow-up question strategy corresponding to the knowledge type from a plurality of preset knowledge engineering follow-up question strategies; and performing a plurality of rounds of follow-up questions on the input question according to the target follow-up question strategy to obtain an implicit knowledge extraction result, wherein the input of each round of follow-up questions comprises a dialogue context formed by a question and an answer of a previous round. The present application combines knowledge type identification with knowledge engineering follow-up question strategy routing, solves the problem of insufficient activation of implicit knowledge caused by the use of a unified follow-up question sequence for all questions in the existing follow-up question method, and realizes differentiated and deep extraction of implicit knowledge in a large language model.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method, apparatus, device, and medium for extracting implicit knowledge from a large language model based on knowledge engineering mapping. Background Technology

[0002] Large language models are increasingly being applied in high-risk fields such as medical diagnosis, legal reasoning, and educational tutoring. The training corpora of large language models contain a large amount of tacit knowledge derived from human experts, such as procedural knowledge, evaluative knowledge, classification knowledge, boundary condition knowledge, and causal knowledge. However, by default, large language models tend to output surface-level factual content, and the aforementioned tacit knowledge is difficult to activate spontaneously through a single question-and-answer session. To address this, researchers have proposed various follow-up questioning strategies to guide the model to output deeper knowledge content through multiple rounds of interaction.

[0003] Existing technologies include the following two solutions: Option 1: A hierarchical probing method based on Bloom's cognitive taxonomy. Probing prompts are designed sequentially according to the six levels of Bloom's cognitive taxonomy (memory, comprehension, application, analysis, evaluation, and creation), and six rounds of sequential probing are performed on the large language model. The implementation process is as follows: First, an initial question is posed to the model to obtain an initial answer. Then, probing prompts are generated layer by layer in a fixed order: memory, comprehension, application, analysis, evaluation, and creation, and submitted to the model. Each round of probing uses the previous round's answer as context, and finally, the six rounds of answers are summarized as the knowledge extraction result. This method guides the model from factual retelling to comprehensive creation through the progression of cognitive levels.

[0004] Option 2: A structured probing method based on deep hierarchical levels. The probing process is divided into three deep levels: D1 (factual level), D2 (procedural level), and D3 (strategic level). Probing questions are posed to the model sequentially from D1 to D3. The implementation process is as follows: after triggering a response at the D1 level with the original question, the model is asked to explain the underlying mechanism using the D2 probing prompt, and then the model is asked to reveal implicit rules using the D3 probing prompt, for a total of three rounds of probing. This method guides the model to output deeper reasoning content through a progressive structure of deep hierarchical levels.

[0005] However, both existing solutions have the following drawbacks: both solutions use a uniform, fixed questioning sequence for all input questions, failing to differentiate the activation mechanisms of different knowledge types. This results in certain types of tacit knowledge not being effectively activated, leading to insufficient knowledge extraction depth. Furthermore, Solution 1 enforces all rounds according to the Bloom six-layer framework, where the three shallow layers—memory layer, understanding layer, and application layer—contribute very little to the activation of deep tacit knowledge, causing a systematic waste of computational resources. Summary of the Invention

[0006] This invention provides a method, apparatus, device, and medium for extracting implicit knowledge from a large language model based on knowledge engineering technology mapping, so as to achieve differentiated and in-depth extraction of implicit knowledge from the large language model.

[0007] According to one aspect of the present invention, a method for extracting implicit knowledge from a large language model based on knowledge engineering mapping is provided, comprising: Input reception problem; The input question is subjected to knowledge type identification to obtain the knowledge type corresponding to the input question. The knowledge type includes one of the following: procedural knowledge, evaluative knowledge, classification knowledge, boundary condition knowledge, and causal knowledge. Based on the knowledge type, a target questioning strategy corresponding to the knowledge type is determined from a set of preset knowledge engineering questioning strategies. The knowledge engineering questioning strategies include thinking aloud, critical incident technique, comparative case method, exception detection method, and upward ladder method. According to the target probing strategy, multiple rounds of probing are performed on the input question to obtain implicit knowledge extraction results. The input of each round of probing includes the dialogue context formed by the questions and answers of the previous rounds.

[0008] According to another aspect of the present invention, a device for extracting implicit knowledge from a large language model based on knowledge engineering mapping is provided, comprising: The question receiving module is used to receive input questions; The knowledge type identification module is used to identify the knowledge type of the input question and obtain the knowledge type corresponding to the input question. The knowledge type includes one of procedural knowledge, evaluative knowledge, classification knowledge, boundary condition knowledge, and causal knowledge. The strategy matching module is used to determine the target inquiry strategy corresponding to the knowledge type from a number of preset knowledge engineering inquiry strategies based on the knowledge type. The knowledge engineering inquiry strategies include thinking aloud, critical event technique, comparative case method, exception detection method, and upward ladder method. The implicit knowledge extraction module is used to perform multiple rounds of follow-up questions on the input question according to the target follow-up questioning strategy to obtain implicit knowledge extraction results. The input of each round of follow-up questions includes the dialogue context formed by the questions and answers of the previous rounds.

[0009] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the method for extracting implicit knowledge from a large language model based on knowledge engineering mapping as described in any embodiment of the present invention.

[0010] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the method for extracting implicit knowledge from a large language model based on knowledge engineering mapping as described in any embodiment of the present invention.

[0011] The technical solution of this invention classifies input questions according to knowledge type and routes them to corresponding knowledge engineering questioning strategies based on the classification results. This allows the questioning strategies to apply differentiated guidance to the activation mechanisms of different implicit knowledge, overcoming the deficiency of using a uniform questioning sequence for all questions in existing technologies. Furthermore, by appending each round of questioning prompts to the end of the dialogue message queue while preserving the complete dialogue context, subsequent rounds of questioning can further deepen the understanding based on the explicit knowledge from the previous round, effectively improving the depth of implicit knowledge extraction.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 A flowchart illustrating a method for extracting implicit knowledge from a large language model based on knowledge engineering mapping, provided in this embodiment of the invention; Figure 2 The diagram shown is a mapping relationship between knowledge types and knowledge engineering inquiry strategies provided in an embodiment of the present invention. Figure 3 A flowchart illustrating another method for extracting implicit knowledge from a large language model based on knowledge engineering mapping, provided as an embodiment of the present invention; Figure 4 A schematic diagram of the structure of a device for extracting implicit knowledge from a large language model based on knowledge engineering technology mapping, provided in an embodiment of the present invention; Figure 5A schematic diagram of the structure of an electronic device for implementing the implicit knowledge extraction method of a large language model based on knowledge engineering technology mapping according to an embodiment of the present invention. Detailed Implementation

[0015] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0016] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0017] To further clarify the technical solutions of the embodiments of the present invention, the prior art is further described as follows: Both Scheme 1 and Scheme 2 employ a uniform, fixed sequence of follow-up questions for all input questions, failing to identify and differentiate the types of knowledge involved. However, the cognitive structures and activation mechanisms of procedural knowledge, evaluative knowledge, categorical knowledge, boundary condition knowledge, and causal knowledge differ: procedural knowledge requires activation through simulating a step-by-step execution process; evaluative knowledge requires triggering judgment criteria through extreme cases; categorical knowledge requires revealing distinguishing dimensions through comparative cases; boundary condition knowledge requires activation through specifically probing exceptional cases; and causal knowledge requires repeatedly probing the causal chain. Because the two schemes do not apply corresponding follow-up strategies to different knowledge types, specific types of tacit knowledge cannot be effectively activated, resulting in insufficient depth of knowledge extraction. This invention identifies the knowledge type of the input question and determines the corresponding target follow-up strategy from multiple preset knowledge engineering follow-up strategies based on the identified knowledge type. This matches the follow-up method with the activation mechanism of tacit knowledge, achieving targeted extraction of different knowledge types.

[0018] Scheme 1 enforces all six rounds of follow-up questions according to Bloom's six-layer cognitive framework. However, the shallow-level follow-up questions (L1 (memory), L2 (understanding), and L3 (application) only guide the model to repeat or rewrite known facts, contributing very little to the activation of deeper knowledge such as procedural judgments, boundary conditions, and implicit heuristics. Because Scheme 1 does not differentiate the contribution of each level to implicit knowledge extraction, the computational cost of approximately half of the follow-up questioning rounds cannot be converted into effective knowledge extraction benefits, resulting in a systematic waste of computational resources. This invention skips the L1 (memory), L2 (understanding), and L3 (application) levels in Bloom's taxonomy, directly performing follow-up questions at the L4 (analysis), L5 (evaluation), and L6 (creation) levels, thus reducing ineffective computational overhead while ensuring extraction depth.

[0019] Figure 1 This is a flowchart illustrating a method for extracting implicit knowledge from a large language model based on knowledge engineering mapping, as provided in an embodiment of the present invention. This embodiment is applicable to scenarios involving the extraction of implicit knowledge from large language models. The method can be executed by a device for extracting implicit knowledge from a large language model based on knowledge engineering mapping. This device can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method specifically includes the following steps: S110, Input reception problem.

[0020] The input questions are domain-specific questions entered by the user. These questions can be natural language questions from scenarios such as medical diagnosis, legal reasoning, engineering maintenance, educational Q&A, or enterprise knowledge base construction. Input questions can be in text format, supporting single or batch input. In the case of batch input, the input data can be organized in JSONL format, with each line containing a sample question to be processed.

[0021] Specifically, the system receives raw question text submitted by users through an input interface. For single inputs, it directly receives the question submitted by the user through the interactive interface. For batch inputs, it reads the question data line by line from a JSONL format input file, with each line containing a sample number and the corresponding question text. The received raw questions serve as the starting point for subsequent knowledge type identification and multiple rounds of follow-up questioning.

[0022] S120. Perform knowledge type identification on the input question to obtain the knowledge type corresponding to the input question.

[0023] Among them, knowledge types include one of the following: procedural knowledge, evaluative knowledge, categorical knowledge, boundary condition knowledge, and causal knowledge.

[0024] Procedural knowledge refers to knowledge related to operational steps, execution processes, and workflows. Evaluation knowledge refers to knowledge related to judgment criteria, trade-offs, and solution selection. Classification knowledge refers to knowledge related to object differentiation, category identification, and classification dimensions. Boundary condition knowledge refers to knowledge related to rule exceptions, taboo conditions, and scope of application. Causal knowledge refers to knowledge related to causal mechanisms, underlying principles, and causal chains.

[0025] In some possible implementations, a few-shot suggestion approach can be used to identify the knowledge type of the input question. Classification suggestions containing definitions and labeled examples of various knowledge types can be input together with the input question into a large language model; the structured classification results output by the large language model can then be obtained. The structured classification results include knowledge type labels and confidence scores.

[0026] Specifically, the classification prompts consist of system role descriptions, definitions of five knowledge types, and at least two labeled examples for each knowledge type. The question to be processed is the final user message input. The classification prompts pre-set definitions and typical language features for each knowledge type. For example, typical language features for procedural knowledge include operation-oriented words such as "how to do it," "steps," and "go through the process"; typical language features for evaluative knowledge include judgment-oriented words such as "which is better," "is it suitable," and "should be chosen"; typical language features for classification knowledge include distinction-oriented words such as "how to distinguish," "identify," and "which category does it belong to"; typical language features for boundary condition knowledge include exception-oriented words such as "when should not," "contraindications," and "exceptions"; and typical language features for causal knowledge include causal-oriented words such as "why," "mechanism," and "what causes it." These keywords are used to assist in prompting rather than mechanical matching; the final category is determined by a large language model combined with the semantics of the question.

[0027] The structured classification results output by the large language model are in JSON format and contain at least three fields: knowledge_type (knowledge type label), confidence (confidence score), and reasoning (classification reasoning basis). The knowledge type label takes one of the five knowledge types mentioned above. The confidence score indicates the reliability of the classification result.

[0028] In some possible implementations, the knowledge type identification of the input question further includes: if the confidence score is lower than a preset threshold or the structured classification result fails to be parsed, determining the knowledge type as a preset fallback type.

[0029] Specifically, the JSON structure returned by the large language model can be directly parsed. If parsing fails, the last JSON fragment containing the `knowledge_type` field is extracted from the response and parsed again. If parsing still fails, or the confidence score is below a preset threshold (e.g., below 0.5), the knowledge type is determined to be a preset fallback type. In the specific implementation, this fallback type can be procedural knowledge to trigger the corresponding verbal questioning strategy, ensuring that the subsequent questioning process can be executed stably.

[0030] S130. Based on the knowledge type, determine the target inquiry strategy corresponding to the knowledge type from a plurality of preset knowledge engineering inquiry strategies.

[0031] Among them, knowledge engineering inquiry strategies include thinking aloud, critical incident technique, comparative case study, exception detection, and upward ladder method.

[0032] Specifically, there is a one-to-one mapping relationship between knowledge types and knowledge engineering inquiry strategies. Based on the knowledge type identified in step S120, a strategy corresponding to that knowledge type is selected from a plurality of preset knowledge engineering inquiry strategies as the target inquiry strategy.

[0033] In some possible implementations, if the knowledge type is procedural knowledge, then the thought-aloud method is identified as the target inquiry strategy. The thought-aloud method guides the model to explicitly output execution steps, path branches, and checkpoints. If the knowledge type is evaluative knowledge, then the critical incident technique is identified as the target inquiry strategy. The critical incident technique guides the model to reveal judgment criteria and trade-off logic through extreme or exceptional cases. If the knowledge type is categorical knowledge, then the comparative case method is identified as the target inquiry strategy. The comparative case method guides the model to reveal implicit categorical dimensions through differences between similar cases. If the knowledge type is boundary condition knowledge, then the exception detection method is identified as the target inquiry strategy. The exception detection method guides the model to output situations where rules do not apply, taboo conditions, and red flag signals. If the knowledge type is causal knowledge, then the upward ladder method is identified as the target inquiry strategy. The upward ladder method guides the model to inquire layer by layer along the causal chain and extract dominant principles.

[0034] like Figure 2 The diagram shown illustrates the mapping relationship between knowledge types and knowledge engineering inquiry strategies provided in this embodiment of the invention. The mechanism of the verbal thinking method lies in requiring the model to re-solve the problem and explicitly state the steps, micro-decision-making, path bifurcation, and uncertainty handling that are typically omitted. The output targets are the operation steps, checkpoints, stopping conditions, and common mistakes made by beginners.

[0035] The mechanism of critical incident technology is to use specific events such as failure of standard solutions, danger or abnormality as triggers to inquire what differences change the decision. Its output targets are judgment thresholds, trade-off logic, high-risk signals and strategy switching conditions.

[0036] The mechanism of the comparative case method is to compare cases that are superficially similar but have different processing methods. It requires the model to name distinguishable dimensions and observable clues, and the output targets are classification axis, identification features, misclassification risk and diversion rules.

[0037] The mechanism of exception detection is to specifically inquire when the rule is not applicable, the earliest abnormal signal, the harmful consequences and red flag combinations, and output the contraindications, exception conditions, alternative paths and escalation rules.

[0038] The mechanism of the upward ladder method is to ask questions layer by layer along the "why it is important", distinguish the causal chain from the surface correlation to the basic principle, and output the mechanism of action, constraints, dominant factors and transferable practical rules.

[0039] S140. Perform multiple rounds of follow-up questions on the input question according to the target follow-up questioning strategy to obtain the implicit knowledge extraction results.

[0040] The input for each round of follow-up questions includes the dialogue context formed by the questions and answers from the previous rounds.

[0041] Specifically, the input question can be submitted to a large language model to obtain an initial answer. Then, based on the target follow-up questioning strategy, follow-up question prompts are generated sequentially at the analysis layer, evaluation layer, and creation layer. Each follow-up question prompt is appended to the end of the dialogue message queue, and the large language model is called after each appending to obtain the answer for that round.

[0042] For example, a dialogue message queue can be maintained. In round 0, the message queue contains system messages and user messages (i.e., the original input question). After calling the large language model, an initial answer is obtained and appended to the end of the message queue. In round 1, analysis-layer follow-up prompts are generated based on the target follow-up strategy. These prompts are appended to the end of the message queue as user messages. At this point, the message queue contains system messages, the original question, the initial answer, and analysis-layer follow-up prompts. After calling the large language model, an analysis-layer answer is obtained. In round 2, evaluation-layer follow-up prompts are appended to the end of the message queue. At this point, the message queue contains all the aforementioned content plus the evaluation-layer follow-up prompts. After calling the large language model, an evaluation-layer answer is obtained. In round 3, creation-layer follow-up prompts are appended to the end of the message queue. At this point, the message queue contains all the aforementioned content plus the creation-layer follow-up prompts. After calling the large language model, a creation-layer answer is obtained.

[0043] Therefore, when the follow-up prompts are input into the large language model in the next round, the dialogue message queue includes the input question, the initial answer, and the follow-up prompts and corresponding answers from previous rounds. This cumulative message queue management method allows the follow-up questions to be further deepened based on the knowledge already explicitly presented in the previous round, forming a continuous follow-up questioning process from structural analysis and exception evaluation to rule generation, rather than each round submitting follow-up questions to the model independently.

[0044] In some possible implementations, after completing the initial question and answer and multiple rounds of follow-up questions, structured output data can be generated based on the input question, knowledge type, target follow-up question strategy, and prompts and answers for each round of follow-up questions.

[0045] The structured output data should include at least the knowledge type, the target questioning strategy, the prompts for each round of questioning, and the answers to each round of questioning. The structured output data can be organized in JSON format and used to build domain knowledge bases, generate expert rules of experience, or serve as the basis for reasoning by downstream agents.

[0046] The technical solution of this invention classifies input questions according to knowledge type and routes them to corresponding knowledge engineering questioning strategies based on the classification results. This allows the questioning strategies to apply differentiated guidance to the activation mechanisms of different implicit knowledge, overcoming the deficiency of using a uniform questioning sequence for all questions in existing technologies. Furthermore, by appending each round of questioning prompts to the end of the dialogue message queue while preserving the complete dialogue context, subsequent rounds of questioning can further deepen the understanding based on the explicit knowledge from the previous round, effectively improving the depth of implicit knowledge extraction.

[0047] Figure 3 This is a flowchart illustrating another method for extracting implicit knowledge from a large language model based on knowledge engineering mapping, provided as an embodiment of the present invention. This embodiment is a further optimization of the implicit knowledge extraction results obtained by performing multiple rounds of follow-up questioning on the input question according to the target follow-up questioning strategy in the above embodiments. Technical terms that are the same as or corresponding to those in the above embodiments will not be repeated here. Figure 3 As shown, the method specifically includes the following steps: S310, Input Reception Problem.

[0048] S320. Perform knowledge type identification on the input question to obtain the knowledge type corresponding to the input question.

[0049] S330. Based on the knowledge type, determine the target inquiry strategy corresponding to the knowledge type from a plurality of preset knowledge engineering inquiry strategies.

[0050] S340. Input the input question into the large language model to obtain an initial answer.

[0051] Specifically, the original input question is submitted to the large language model as a user message. Before submission, a system role prompt message is set in the dialogue message queue. This system role prompt guides the large language model to answer the question as a domain expert, providing accurate, clear, and actionable instructions. After invoking the large language model, an initial answer is obtained and appended to the end of the dialogue message queue as an assistant message. At this point, the dialogue message queue contains system messages, user messages (original question), and assistant messages (initial answer).

[0052] S350. Based on the target questioning strategy, generate questioning prompts for each round in sequence at the analysis layer, evaluation layer, and creation layer.

[0053] In this embodiment, the analysis layer, evaluation layer, and creation layer are respectively the L4 analysis layer, L5 evaluation layer, and L6 creation layer in Bloom's Taxonomy of Cognition. This multi-round questioning skips the L1 memory layer, L2 comprehension layer, and L3 application layer in Bloom's Taxonomy of Cognition, and directly begins questioning from the L4 analysis layer.

[0054] Specifically, based on the target questioning strategy determined in step S230, first-round questioning prompts, second-round questioning prompts, and third-round questioning prompts are generated respectively. The first-round questioning prompts correspond to the L4 analysis level, the second-round questioning prompts correspond to the L5 evaluation level, and the third-round questioning prompts correspond to the L6 creation level. Under different target questioning strategies, the specific content of each round of questioning prompts is different.

[0055] For example, when the goal-oriented questioning strategy is the thought-aloud method, the L4 analysis layer question prompts require the model to decompose the key steps of the process and identify the decision points where the path branches. The L5 evaluation layer question prompts require the model to evaluate the steps where novices are most prone to making mistakes and the earliest error signals that can be detected. The L6 creation layer question prompts require the model to generate an actionable checklist, which includes decision checkpoints, verification criteria, and stopping rules.

[0056] When the target inquiry strategy is critical event technology, the L4 analysis layer inquiry prompts require the model to analyze the key interaction factors affecting the decision outcome, the L5 evaluation layer inquiry prompts require the model to describe specific cases where the standard solution has failed or is at risk, and the L6 creation layer inquiry prompts require the model to synthesize and generate a decision framework, listing the non-negotiable checkpoints and coverage conditions.

[0057] When the target inquiry strategy is the comparative case method, the L4 analysis layer inquiry prompts require the model to analyze two superficially similar cases that require different treatments and point out the distinguishing features. The L5 evaluation layer inquiry prompts require the model to point out the distinguishing dimensions used when classifying cases. The L6 creation layer inquiry prompts require the model to generate a triage rule table.

[0058] When the target probing strategy is the exception detection method, the probing prompt at the L4 analysis layer requires the model to analyze the specific conditions under which the method is not applicable, the probing prompt at the L5 evaluation layer requires the model to point out the earliest warning signal for each exception condition and the consequence caused by ignoring the signal, and the probing prompt at the L6 creation layer requires the model to write clear override rules and red flag combinations.

[0059] When the target probing strategy is the upward ladder method, the probing prompt at the L4 analysis layer requires the model to analyze the mechanism and explain the causal chain, the probing prompt at the L5 evaluation layer requires the model to analyze why the mechanism is important under real-world constraints, and the probing prompt at the L6 creation layer requires the model to extract the underlying governing principle behind the causal chain and convert it into transferable practical rules.

[0060] In some possible implementations, the above-mentioned rounds of probing prompts have pre-designed fixed templates. The probing prompt templates for the L4 analysis layer, L5 evaluation layer and L6 creation layer corresponding to different knowledge types are as follows.

[0061] Procedural knowledge (think-aloud method) L4 analysis layer: Please decompose the key steps of this process and identify the decision points where paths diverge. At each step, what are the micro-decisions that you execute automatically but do not explicitly state? L5 evaluation layer: Please evaluate the links in this process where novices are most prone to make mistakes. What is the earliest detectable signal after taking the wrong path? Why is this error easy to overlook? L6 creation layer: Please generate an actionable checklist for this process, including clear decision checkpoints, verification criteria and stop rules, with a level of precision that allows trainees to perform safely.

[0062] Evaluative knowledge (critical incident technique) L4 analysis layer: Please analyze the key interacting factors that jointly influence the outcome of this decision. What is the structure of this decision? Which variables are the most important and why? L5 evaluation layer: Please describe a specific case where the standard solution would fail or be dangerous. What makes this case different? At which node might an inexperienced person fail to perceive that it is an exception? L6 creation layer: Based on the above analysis, please comprehensively generate a decision framework that lists non-negotiable checkpoints and override conditions that will change the overall processing strategy.

[0063] Taxonomic knowledge (contrast case method) L4 analysis layer: Please analyze two cases that are superficially similar but require different treatment schemes. What is the single most important feature that distinguishes them? Why does this feature change the treatment path? L5 Assessment Level: Please indicate the distinguishing dimensions you used when classifying these types of cases. Which observable clues—outcome, timing, context—reliably categorize cases before taking action? L6 Creation Layer: Please generate a concise triage rule table: if [clue A], execute X; if [clue B], execute Y. Include the most dangerous misclassification scenarios and how to avoid them.

[0064] Boundary condition knowledge (exception detection method) L4 Analysis Layer: Please analyze situations where the described method is not applicable. Specify the specific conditions under which the decision-making process should be changed—patient factors, timing, context, etc.

[0065] L5 Assessment Layer: For each exceptional condition: What is the earliest warning sign that professionals should detect? What are the consequences of ignoring this sign? Approximately how often do these exceptions occur? L6 Creation Layer: Please write clear coverage rules: conditions that are forbidden by the standard scheme, what the alternatives are, and the "red flag combination" that should be used to immediately initiate the upgrade.

[0066] Causal knowledge (the upward ladder method) L4 Analysis Layer: Please analyze its mechanism: Why is this intervention or discovery effective at the principle level? Please specify the causal chain, rather than just describing the correlation.

[0067] L5 Evaluation Layer: Going a step further: Why is this mechanism important under realistic constraints and trade-offs? What happens if we ignore this mechanism? Under what circumstances does it become the dominant factor? L6 Creation Layer: Extract the underlying dominant principle behind this causal chain and then transform it into a transferable rule of practice so that professionals do not have to re-derive the mechanism every time.

[0068] Explanation of terminology substitutions for different application areas: The terms "professionals," "practitioners," and "trainees" in the five types of prompt templates mentioned above can be adaptively replaced in different application scenarios. For example, in a medical diagnosis scenario, they can correspond to "clinical doctors," "medical staff," and "medical students"; in an engineering operation and maintenance scenario, they can correspond to "engineers," "operators," and "technicians"; in a legal reasoning scenario, they can correspond to "lawyers," "legal personnel," and "legal assistants"; and in an educational tutoring scenario, they can correspond to "teachers," "mentors," and "students." The referential expressions such as "this process," "this method," and "this intervention" in each prompt are automatically associated with the specific content of the current input question and previous answers during the execution of the dialogue context. The above term replacements do not change the core semantics and mechanism of the prompts.

[0069] S360. Add each round of follow-up prompts to the end of the dialogue message queue, and call the large language model after each round of addition to obtain the answer for that round, so as to obtain the implicit knowledge extraction result.

[0070] When the follow-up prompts are input into the large language model in the next round, the dialogue message queue includes the input question, the initial answer, and the follow-up prompts and corresponding answers from the previous rounds.

[0071] Specifically, the first-round follow-up prompt generated in step S350 is appended as a user message to the end of the dialogue message queue. At this point, the dialogue message queue contains system messages, the original question, the initial answer, and the first-round follow-up prompt. Further, the first-round answer is obtained after calling the large language model, and this first-round answer is appended as an assistant message to the end of the dialogue message queue.

[0072] Furthermore, the second-round follow-up prompt is appended as a user message to the end of the dialogue message queue. At this point, the dialogue message queue contains system messages, the original question, the initial answer, the first-round follow-up prompt, the first-round answer, and the second-round follow-up prompt. After calling the large language model, the second-round answer is obtained and appended as an assistant message to the end of the dialogue message queue.

[0073] Finally, the third-round follow-up prompt is appended as a user message to the end of the dialogue message queue. At this point, the dialogue message queue contains system messages, the original question, the initial answer, the first-round follow-up prompt, the first-round answer, the second-round follow-up prompt, the second-round answer, and the third-round follow-up prompt. After calling the large language model, the third-round answer is obtained and appended as an assistant message to the end of the dialogue message queue.

[0074] Therefore, when the follow-up prompts are input into the large language model in the next round, the dialogue message queue includes the input question, the initial answer, and the follow-up prompts and corresponding answers from previous rounds. This cumulative message queue management method allows the follow-up questions to further deepen the knowledge already explicitly presented in the previous round. The three rounds of follow-up questions are executed at the L4 analysis level, L5 evaluation level, and L6 creation level of Bloom's Taxonomy of Cognition, respectively, skipping the L1 memory level, L2 comprehension level, and L3 application level, thus avoiding the waste of computational resources caused by low-value rounds.

[0075] After completing the three rounds of follow-up questions, the initial answer and the answers to the three rounds of follow-up questions are used together as the results of tacit knowledge extraction.

[0076] The technical solution of this invention identifies the knowledge type of the original question and routes it to the corresponding knowledge engineering questioning strategy based on the identification result. Then, it directly performs three rounds of questioning at the L4 analysis layer, L5 evaluation layer, and L6 creation layer of Bloom's Taxonomy of Cognition, skipping the three shallow layers from L1 to L3 and avoiding waste of computational resources. Simultaneously, by appending each round of questioning prompts to the end of the dialogue message queue, each round of questioning is based on the complete dialogue context of all previous rounds. Subsequent rounds of questioning can further deepen the understanding of the knowledge already expressed in the previous round, achieving gradual mining and cumulative extraction of tacit knowledge, and achieving a deeper knowledge extraction depth with the same or fewer rounds of questioning.

[0077] This invention employs a combined mechanism of "knowledge type identification—knowledge engineering technology mapping—high-level specialized probing," enabling large language models to select differentiated probing paths based on the type of tacit knowledge involved in the input question, rather than applying a uniform, fixed probing template to all questions. Therefore, under similar or identical probing rounds, this invention can more fully activate different types of tacit knowledge, allowing the model's answers to not only include surface facts and general explanations, but also to further output operational steps, micro-decision-making processes, judgment criteria, classification basis, exceptions, risk signals, causal chains, and dominant principles from expert practice. This improves the depth, breadth, and quality of tacit knowledge extraction under the same number of probing rounds.

[0078] This invention utilizes a high-order specialized probing mechanism based on knowledge engineering techniques. It directly designs probing directions around tacit knowledge types such as procedural, evaluative, categorical, boundary condition, and causal knowledge, ensuring that each round of probing focuses more intently on key content within expert knowledge. Therefore, when achieving similar knowledge extraction depth or response quality, this invention reduces low-value probing rounds and ineffective model calls, lowers token consumption, computational overhead, and response time, and improves the execution efficiency of the tacit knowledge extraction process.

[0079] Figure 4 This is a schematic diagram of a device for extracting implicit knowledge from a large language model based on knowledge engineering mapping, provided as an embodiment of the present invention. Figure 4 As shown, the device includes: Question receiving module 410 is used to receive input questions; The knowledge type identification module 420 is used to identify the knowledge type of the input question and obtain the knowledge type corresponding to the input question. The knowledge type includes one of procedural knowledge, evaluative knowledge, classification knowledge, boundary condition knowledge, and causal knowledge. The strategy matching module 430 is used to determine the target inquiry strategy corresponding to the knowledge type from a plurality of preset knowledge engineering inquiry strategies based on the knowledge type. The knowledge engineering inquiry strategies include thinking aloud, critical event technique, comparative case method, exception detection method and upward ladder method. The implicit knowledge extraction module 440 is used to perform multiple rounds of follow-up questions on the input question according to the target follow-up questioning strategy to obtain implicit knowledge extraction results. The input of each round of follow-up questions includes the dialogue context formed by the questions and answers of the previous rounds.

[0080] In some possible implementations, the knowledge type identification module 420 includes: The few-shot suggestion submodule is used to input classification suggestion words containing definitions and labeled examples of various knowledge types into the large language model along with the input question using a few-shot suggestion method. The result acquisition submodule is used to acquire the structured classification results output by the large language model. The structured classification results include knowledge type labels and confidence scores.

[0081] In some possible implementations, the knowledge type identification module 420 is further used for: If the confidence score is lower than a preset threshold or the structured classification result fails to be parsed, the knowledge type will be determined as a preset fallback type.

[0082] In some possible implementations, the policy matching module 430 includes: The first matching submodule is used to determine the verbal thinking method as the target questioning strategy when the knowledge type is procedural knowledge. The second matching submodule is used to identify key event techniques as the target inquiry strategy when the knowledge type is evaluative knowledge. The third matching submodule is used to determine the comparative case method as the target inquiry strategy when the knowledge type is categorical knowledge. The fourth matching submodule is used to determine the exception detection method as the target inquiry strategy when the knowledge type is boundary condition knowledge; The fifth matching submodule is used to determine the upward ladder method as the target inquiry strategy when the knowledge type is causal knowledge.

[0083] In some possible implementations, the implicit knowledge extraction module 440 includes: The initial answer generation submodule is used to input the input question into the large language model and obtain an initial answer. The hierarchical prompt word generation submodule is used to generate prompt words for each round of follow-up questions in the analysis layer, evaluation layer and creation layer in sequence according to the target follow-up questioning strategy. The iterative follow-up questioning submodule is used to append each round of follow-up question prompts to the end of the dialogue message queue, and call the large language model to obtain the answer for the current round after each round of appending; wherein, when the follow-up question prompts of the next round are input into the large language model, the dialogue message queue includes the input question, the initial answer, and the follow-up question prompts and corresponding answers of the previous rounds.

[0084] In some possible implementations, the analysis layer, the evaluation layer, and the creation layer are respectively the L4 analysis level, L5 evaluation level, and L6 creation level in Bloom's Taxonomy of Cognition. When the implicit knowledge extraction module 540 performs multiple rounds of follow-up questions, it skips the L1 memory level, L2 comprehension level, and L3 application level in Bloom's Taxonomy of Cognition.

[0085] In some possible implementations, the device further includes: The structured output module is used to generate structured output data based on the input question, the knowledge type, the target follow-up strategy, and the prompts and answers for each round of follow-up questions; the structured output data includes at least the knowledge type, the target follow-up strategy, the prompts for each round of follow-up questions, and the answers for each round of follow-up questions.

[0086] The implicit knowledge extraction device for large language models based on knowledge engineering mapping provided in this embodiment of the invention can execute the implicit knowledge extraction method for large language models based on knowledge engineering mapping provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0087] Figure 5 This is a schematic diagram of an electronic device for implementing the implicit knowledge extraction method for large language models based on knowledge engineering mapping, as described in this embodiment of the invention. The electronic device is intended to represent various forms of digital computers, such as laptops, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0088] like Figure 5As shown, the electronic device 50 includes at least one processor 51 and a memory, such as a read-only memory (ROM) 52 and a random access memory (RAM) 53, communicatively connected to the at least one processor 51. The memory stores computer programs executable by the at least one processor. The processor 51 can perform various appropriate actions and processes based on the computer program stored in the ROM 52 or loaded from storage unit 58 into the RAM 53. The RAM 53 can also store various programs and data required for the operation of the electronic device 50. The processor 51, ROM 52, and RAM 53 are interconnected via a bus 54. An input / output (I / O) interface 55 is also connected to the bus 54.

[0089] Multiple components in electronic device 50 are connected to I / O interface 55, including: input unit 56, such as keyboard, mouse, etc.; output unit 57, such as various types of monitors, speakers, etc.; storage unit 58, such as disk, optical disk, etc.; and communication unit 59, such as network card, modem, wireless transceiver, etc. Communication unit 59 allows electronic device 50 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0090] Processor 51 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 51 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 51 performs the various methods and processes described above, such as the implicit knowledge extraction method for large language models based on knowledge engineering techniques.

[0091] In some embodiments, the method for extracting implicit knowledge from a large language model based on knowledge engineering mapping can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 58. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 50 via ROM 52 and / or communication unit 59. When the computer program is loaded into RAM 53 and executed by processor 51, one or more steps of the method for extracting implicit knowledge from a large language model based on knowledge engineering mapping described above can be performed. Alternatively, in other embodiments, processor 51 can be configured to perform the method for extracting implicit knowledge from a large language model based on knowledge engineering mapping by any other suitable means (e.g., by means of firmware).

[0092] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific integrated circuits (ASICs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0093] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0094] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0095] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0096] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0097] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0098] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0099] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for extracting implicit knowledge from a large language model based on knowledge engineering mapping, characterized in that, include: Input reception problem; The input question is subjected to knowledge type identification to obtain the knowledge type corresponding to the input question. The knowledge type includes one of the following: procedural knowledge, evaluative knowledge, classification knowledge, boundary condition knowledge, and causal knowledge. Based on the knowledge type, a target questioning strategy corresponding to the knowledge type is determined from a set of preset knowledge engineering questioning strategies. The knowledge engineering questioning strategies include thinking aloud, critical incident technique, comparative case method, exception detection method, and upward ladder method. According to the target probing strategy, multiple rounds of probing are performed on the input question to obtain implicit knowledge extraction results. The input of each round of probing includes the dialogue context formed by the questions and answers of the previous rounds.

2. The method according to claim 1, characterized in that, The knowledge type identification of the input question includes: A few-shot suggestion method is adopted, in which classification suggestion words containing definitions and labeled examples of various knowledge types are input together with the input question into the large language model; Obtain the structured classification results output by the large language model, which include knowledge type labels and confidence scores.

3. The method according to claim 2, characterized in that, The knowledge type identification of the input question also includes: If the confidence score is lower than a preset threshold or the structured classification result fails to be parsed, the knowledge type will be determined as a preset fallback type.

4. The method according to claim 1, characterized in that, The step of determining the target inquiry strategy corresponding to the knowledge type from a set of preset knowledge engineering inquiry strategies, based on the knowledge type, includes: When the knowledge type is procedural knowledge, the verbal thinking method is identified as the target inquiry strategy. When the knowledge type is evaluative knowledge, the key incident technique is identified as the target inquiry strategy; When the knowledge type is classified knowledge, the comparative case method is determined as the target inquiry strategy; When the knowledge type is boundary condition knowledge, the exception detection method is determined as the target inquiry strategy; When the knowledge type is causal knowledge, the upward ladder method is determined as the target inquiry strategy.

5. The method according to claim 1, characterized in that, The step of performing multiple rounds of follow-up questioning on the input question according to the target follow-up questioning strategy includes: The input question is fed into the large language model to obtain an initial answer; Based on the target questioning strategy, questioning prompts are generated sequentially at the analysis layer, evaluation layer, and creation layer. Each round of follow-up prompts is appended to the end of the dialogue message queue, and the large language model is called after each round of appending to obtain the answer for that round. When the follow-up prompts are input into the large language model in the next round, the dialogue message queue includes the input question, the initial answer, and the follow-up prompts and corresponding answers from the previous rounds.

6. The method according to claim 5, characterized in that, The analysis layer, the evaluation layer, and the creation layer are respectively the L4 analysis level, L5 evaluation level, and L6 creation level in Bloom's Taxonomy of Cognition, and the multi-round follow-up questioning skips the L1 memory level, L2 comprehension level, and L3 application level in Bloom's Taxonomy of Cognition.

7. The method according to claim 1, characterized in that, Also includes: Based on the input question, the knowledge type, the target questioning strategy, and the prompts and answers for each round of questioning, structured output data is generated; The structured output data includes at least the knowledge type, the target questioning strategy, the prompts for each round of questioning, and the answers to each round of questioning.

8. A device for extracting implicit knowledge from a large language model based on knowledge engineering mapping, characterized in that, include: The question receiving module is used to receive input questions; The knowledge type identification module is used to identify the knowledge type of the input question and obtain the knowledge type corresponding to the input question. The knowledge type includes one of procedural knowledge, evaluative knowledge, classification knowledge, boundary condition knowledge, and causal knowledge. The strategy matching module is used to determine the target inquiry strategy corresponding to the knowledge type from a number of preset knowledge engineering inquiry strategies based on the knowledge type. The knowledge engineering inquiry strategies include thinking aloud, critical event technique, comparative case method, exception detection method, and upward ladder method. The implicit knowledge extraction module is used to perform multiple rounds of follow-up questions on the input question according to the target follow-up questioning strategy to obtain implicit knowledge extraction results. The input of each round of follow-up questions includes the dialogue context formed by the questions and answers of the previous rounds.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the method for extracting implicit knowledge from a large language model based on knowledge engineering mapping as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the method for extracting implicit knowledge from a large language model based on knowledge engineering mapping as described in any one of claims 1-7.

Images