Programmed knowledge construction method and system based on large model

Abstract

The invention discloses a method and a system for constructing programmed knowledge based on a large model, which are implemented in a manner of structured storage of programmed knowledge by extracting and summarizing key processes and elements of sample data through an automatic or semi-automatic method. The core of the method is to realize the construction of programmed knowledge through the technical architecture of key process extraction, key process summarization, node element extraction in the process and node element summarization of sample data; and then storing in a way of process nodes, node elements and tree structures of corresponding samples, so as to facilitate subsequent recall use. According to the method, key processes and node elements are automatically extracted and verified through a large model, manual intervention is reduced, the processing speed and accuracy can be remarkably improved, and the error rate is reduced; the SOP can be constructed efficiently and accurately, and the problem of seamless integrated use of the SOP and the large model is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing and artificial intelligence, and more particularly to a programmatic knowledge construction method and system based on a large model. BACKGROUND

[0002] At present, the rise of large model technology has promoted the development of intelligent agents based on prompt words and the training of vertical model such as thought chain distillation. However, in enterprise-level application scenarios, the existing technical solutions have significant limitations.

[0003] On the one hand, although traditional vector library and knowledge graph technology can effectively handle factual knowledge questions and answers, it is difficult to support business needs that require complex logical reasoning, and its ability boundary limits the depth of application in process-oriented and decision-making tasks. On the other hand, although the reasoning technology based on thought chain (CoT) has reasoning ability, its general process paradigm often deviates from real and detailed business scenarios. This deviation leads to "hallucinations" in the reasoning process of large models, and the output deviates from the actual business rules or operational details, resulting in insufficient reliability and practicality.

[0004] At the same time, the "standard operating procedure" (SOP) that enterprise operations highly depend on, i.e. programmatic knowledge, is mainly constructed by industry experts manually in the traditional way. This approach has many pain points: the construction process is tedious and time-consuming, requiring a large amount of expert effort to sort out process details; program content is prone to missing key details or incomplete scenario coverage, affecting guidance; more importantly, the completed SOP document usually exists in the form of natural language or table, which is difficult for large models to directly and accurately understand and utilize. This makes the valuable knowledge contained in SOP unable to effectively empower large model-driven intelligent applications, resulting in resource waste and capability bottlenecks.

[0005] Therefore, there is an urgent need for a technical solution that can efficiently and accurately construct SOP automatically or semi-automatically, and solve the problem of seamless integration of SOP and large models. SUMMARY

[0006] In view of this, in order to solve the problems in the prior art, the present application provides a programmatic knowledge construction method and system based on a large model, which aims to utilize the cognitive and generative capabilities of large models, combine sample data of specific business scenarios, and automatically or semi-automatically construct a structured SOP knowledge system that is understandable by large models, and design an efficient calling mechanism to enable large models to strictly follow SOP for reliable reasoning and operation guidance, thereby bridging the gap between existing technology and complex enterprise business needs.

[0007] To achieve the above-mentioned purpose, the technical solution adopted by the present application is as follows:

[0008] In a first aspect, the present application provides a large model-based procedural knowledge construction method, which comprises the following steps:

[0009] S1, using a large model to extract key processes of problem reasoning from sample data, and using the large model to check the relevance and rationality of the key processes;

[0010] S2, classifying the extracted key processes into multiple groups, and performing process node fusion and conflict resolution on the processes within the same group, and merging each group of key processes into a process main body through a large model;

[0011] S3, correcting the key processes and verifying the rationality, accuracy and necessity of the process nodes;

[0012] S4, according to the corrected and verified key processes, using a large model + Prompt to extract element information of the process nodes corresponding to the sample data from the sample data, and checking the relevance and rationality of the element information;

[0013] S5, merging the node elements according to the nodes through a large model, and checking the relevance and rationality of the element merging result;

[0014] S6, correcting the elements in the process nodes and verifying the rationality, accuracy and necessity of the elements in the process nodes;

[0015] S7, based on the sample data set, evaluating the procedural knowledge validity of the key processes and process node elements;

[0016] S8, storing the key processes, process nodes, node elements and sample data according to the tree structure.

[0017] In one specific embodiment, in S1, the large model extracts the key processes of problem reasoning from the sample data, outputs in mermaid format, and the model output contains complete reasoning chain of key decision points, conditional branches and derivation steps; a double-model consensus mechanism is also set during the relevance and rationality check, and iteration is terminated only when both large models agree on the process rationality, and finally a standardized Mermaid file with version mark and a verification report are output.

[0018] In one specific embodiment, in S2, the similarity and difference between processes are calculated by cosine similarity, the extracted key processes are classified into multiple groups, and all key processes classified into the same group are deeply fused to form a unified and optimal process representation within the group.

[0019] In one specific embodiment, in S3, the key processes are corrected and verified through industry expert experience or a large model consensus mechanism.

[0020] In a specific embodiment, in the S4, the large model extraction node element is extracted by writing a Prompt prompt word, wherein the prompt word includes: task description, main process mermaid expression, process node name and description, sample query and answer data, and output element format.

[0021] In a specific embodiment, in the S5, the correlation verification includes:

[0022] Context correlation verification: using a large model to analyze and extract the integrated element set, verifying whether each element is highly related to the core business activities of the node it belongs to;

[0023] Process coherence verification: checking whether the node element supports its node position in the process;

[0024] The rationality verification includes:

[0025] Business logic rationality: using the domain knowledge of a large model to evaluate whether the element conforms to industry common sense and business logic;

[0026] Clarity and operability: evaluating whether the element description is clear and unambiguous.

[0027] In a specific embodiment, in the S6, the correction and verification of the node element are completed by an industry expert deep verification or a large model consensus mechanism; wherein the large model consensus mechanism is that multiple independent large models independently evaluate and vote on each element under each process node in the knowledge base, including rationality voting, accuracy voting and necessity voting.

[0028] In a specific embodiment, in the S7, the effectiveness evaluation and verification of procedural knowledge are performed using a control experiment method, that is, a large model uses two methods to reason about sample data, one method is to not add procedural knowledge in the reasoning prompt word, and the other method is to add procedural knowledge in the reasoning prompt word, and the improvement effect of the reasoning performance of the latter method is evaluated compared with the former method.

[0029] In a specific embodiment, in the S8, the knowledge is stored in a tree structure with a key process as the root, a process node as the first branch, a node element as the second branch, and sample data as the leaf.

[0030] Secondly, the application also provides a large model-based procedural knowledge construction system, which includes a key process acquisition module, a node element acquisition module, a procedural knowledge evaluation module and a procedural knowledge storage module, and is applied to the large model-based procedural knowledge construction method to realize large model-based procedural knowledge construction.

[0031] Compared with existing technologies, the method and system for constructing procedural knowledge based on a large model provided by this invention have at least the following beneficial technical effects:

[0032] 1. High efficiency and automation: By automatically extracting and verifying key process and node elements through large models, manual intervention is reduced, significantly improving processing speed and accuracy, and reducing error rate.

[0033] 2. Intelligent conflict resolution: By integrating processes using semantic alignment and structural alignment technologies, branch path conflicts and timing contradictions are effectively resolved, ensuring the logical coherence of the process and the rationality of business operations.

[0034] 3. Dual Quality Assurance: Combining manual correction by industry experts with a large-scale model voting mechanism, the accuracy and necessity of processes and elements are optimized, balancing the efficiency of automation with the advantages of human experience.

[0035] 4. Structured Knowledge Management: Adopting a tree-like storage architecture (key processes as roots, node elements as branches, and samples as leaves) to achieve hierarchical organization of procedural knowledge, facilitating rapid retrieval and reuse.

[0036] 5. Quantifiable performance improvement: The effect of procedural knowledge injection is verified through controlled experiments, which significantly enhances the seriousness of reasoning, the reliability of tracing the source, and the fidelity of the answer, providing clear proof of validity.

[0037] The objectives and other advantages of the present invention can be realized and obtained by means of the structures specifically pointed out in the written description and drawings.

[0038] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0041] Figure 1 This is a schematic diagram illustrating the principle of the procedural knowledge construction method based on a large model provided in an embodiment of the present invention.

[0042] Figure 2This is a schematic diagram of the electronic device structure provided in an embodiment of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0044] In the description of this invention, it should be noted that some processes described in this application specification and drawings include multiple operations that appear in a specific order. However, it should be clearly understood that these operations may be performed in any order or in parallel. Furthermore, various numbers are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0045] Therefore, the following detailed description of the embodiments of the present invention is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0046] See Figure 1 As shown, this invention provides a method for constructing procedural knowledge based on a large model. It extracts and summarizes key processes and elements of sample data (e.g., user dialogue data, traffic accident liability determination documents, patient medical records, etc.) using automated or semi-automated methods, and stores this knowledge in a structured manner. The core of this method lies in constructing procedural knowledge through a technical architecture that includes key process extraction from sample data, key process summarization, extraction of node elements within the process, and summarization of node elements. Then, it stores the knowledge in a tree structure of process nodes, node elements, and corresponding samples for convenient subsequent retrieval.

[0047] In this embodiment of the invention, the steps for constructing and storing procedural knowledge are as follows:

[0048] The first step is to extract the key processes of problem reasoning from each sample data using large language model technology and output them in mermaid format. Then, the large model is used to verify the relevance and rationality of the key processes in mermaid format.

[0049] The second step involves classifying and identifying the key processes extracted from all sample data using a large model, dividing them into multiple groups according to the classification. Then, all processes within the same group are merged into the main process backbone using large model technology through process node fusion (semantic alignment, structural alignment) and conflict resolution (branch path conflict, timing contradiction resolution) techniques. (The main process backbone can be defined by industry experts during initialization, or it can be left undefined.)

[0050] The third step involves industry experts manually revising the key processes generated in the second step. If industry experts are unavailable, multiple large models can be used to vote on the overall rationality and accuracy of the process, as well as the necessity of each node. Because there is only one key process, manual verification is inexpensive, making manual revision by industry experts the preferred option.

[0051] The fourth step involves using a large model and Prompt technique to extract the key information that needs to be considered at each process node from each sample data point. Then, the large model is used to verify the relevance and rationality of the key information extracted from each sample data point.

[0052] The fifth step is to merge the node elements extracted from the sample data according to the nodes using a large model, and then verify the relevance and rationality of the element merging results using the large model.

[0053] The sixth step involves industry experts manually revising the elements within the overall process nodes. If industry experts are unavailable, multiple large models can be used to vote on the rationality, accuracy, and necessity of each element in the process nodes. Because the number of elements in a node is relatively small compared to the sample data, manual verification is inexpensive, making manual revision by industry experts the preferred method.

[0054] Step 7: Using a sample dataset, evaluate the procedural knowledge of the extracted key processes and process node elements. Two methods are used to infer the sample data using the same large model: one method does not add procedural knowledge to the inference prompts, and the other method adds procedural knowledge to the inference prompts. The performance improvement of the latter method compared to the former method is evaluated.

[0055] The eighth step is to store the key processes, process nodes, node elements, and corresponding cases in a tree structure so that they can be easily recalled when needed in the future.

[0056] The specific embodiments and working principle of the present invention are described below:

[0057] Step 1: Key Process Extraction and Verification:

[0058] 1. Extraction of key reasoning processes and Mermaid formatting:

[0059] Given the sample dataset:

[0060] a) Sample data preprocessing:

[0061] Collected sample data is cleaned and standardized in a uniform format.

[0062] Identify and label core elements such as problem type, known conditions, and objective solution terms to construct structured input.

[0063] b) Large model-driven process extraction:

[0064] Mermaid i =LLM extract (Prompt flow ,d i )

[0065] Design targeted prompts to guide the large language model to gradually break down the problem-solving logic.

[0066] The model output contains a complete reasoning chain including key decision points, conditional branches, and derivation steps.

[0067] It automatically converts natural language descriptions of processes into standardized Mermaid flowchart syntax.

[0068] c) Mermaid syntax standardization:

[0069] Verify syntax compliance (node ​​naming conventions, arrow pointing validity, etc.).

[0070] Add necessary comments and explanations (such as loop termination conditions and parallel operation identifiers).

[0071] Outputs .md files, supporting visual debugging.

[0072] 2. Logic verification and optimization of the flowchart:

[0073] a) Correlation verification:

[0074] Score rel =f(LLM) verify (d i ,answer i Mermaid i ))

[0075] Validation using a large model: Input the original question, labeled answer, and Mermaid flowchart; the model is required to determine the consistency between the two. i It is d i The standard answer.

[0076] Quantitative evaluation indicators: step coverage (whether key operations are omitted) and redundancy (whether irrelevant nodes exist).

[0077] Location deviation (such as incorrect branch conditions or missing feedback loops).

[0078] b) Reasonableness verification:

[0079]

[0080] In the formula, This is an indicator function used to detect logical conflicts.

[0081] Logical conflict detection: Verify contradictions in the flowchart (such as mutually exclusive conditions being true simultaneously).

[0082] Causal chain integrity analysis: Examine the causal coherence of the derivation steps (e.g., whether any necessary prerequisite steps are missing).

[0083] Complexity assessment: The controllability of nested loop depth and number of parallel paths is graded.

[0084] c) Iterative optimization mechanism:

[0085] Based on the verification results, a revision instruction is generated and fed back to the process extraction module for regeneration.

[0086] Set up a dual-model consensus mechanism: the iteration will only terminate when both major models agree that the process is reasonable;

[0087] The final output is a standardized Mermaid file with version markings and a verification report.

[0088] Step Two: Categorize, integrate, and merge key processes into the main framework:

[0089] This step involves effectively integrating the key processes extracted from each sample data to form a unified, standardized, and reasonable process backbone.

[0090] Let the process set be:

[0091] The specific steps are as follows:

[0092] 1. Process classification and grouping:

[0093] a) Large-scale model classification and recognition:

[0094] All extracted key processes are vectorized using a semantic vector model. Then, the similarity and differences between processes are calculated using cosine similarity. That is:

[0095]

[0096] Where d is the feature dimension; f i For the i-th process in the process set; emb i Indicates the relationship with f i The embedding vector obtained after the text is modeled using an embedding model.

[0097] Similarity:

[0098] Difference: div(f i ,f j ) = 1 - sim(f i ,f j ).

[0099] Among them, f j For the j-th process in the process set; emb j Indicates the relationship with f j The embedding vector obtained after the text is modeled using an embedding model.

[0100] b) Category Classification:

[0101] Based on the identification results of the large model, key processes that are semantically similar, have similar goals, or belong to the same business scenario are grouped into the same logical group. This ultimately results in multiple clear process groups. That is:

[0102] Based on the similarity threshold τ and the maximum intra-group distance ∈, the process is divided into k mutually exclusive groups.

[0103]

[0104] Among them, processes that are semantically similar, have similar goals, or are in the same scenario satisfy sim(f i ,f j )≥τ.

[0105] 2. Intra-team process integration and conflict resolution:

[0106] For all key processes grouped together, deep fusion is performed to create a unified, optimal process representation within the group. This involves two key alignment processes and a conflict resolution process:

[0107] a) Integration of process nodes:

[0108] Semantic Alignment (Node Name and Description Similarity): A large model is used to calculate the semantic similarity of nodes across different processes. Activity nodes with different names but identical or highly similar descriptions are identified and merged as the same node. The model needs to understand the core intent of the node descriptions. For any pair of nodes, semantic similarity is calculated:

[0109] s sem (u,v)=LLM sim (desc(u),desc(v))∈[0,1]

[0110] Here, desc(u) and desc(v) are the text descriptions of u and v in the process node.

[0111] Merging criteria:

[0112]

[0113] Where, θ sem The similarity threshold is usually θ. sem >0.8.

[0114] Structural Alignment (In / Out Degree Comparison): Analyze the structural context of nodes. Check if nodes with the same or similar semantics have the same source node for their input edges (in-degree) and the same downstream node pointed to by their output edges (out-degree). Ensure that the position and connection logic of the merged nodes in the process are logically correct. For nodes satisfying (u,v)≥θ sem The nodes are used to verify structural consistency.

[0115]

[0116] Among them, deg in (u) is the in-degree of node u, deg out (u) is the out-degree of node u, deg in (v) is the in-degree of node v, deg out (v) is the out-degree of node v.

[0117] Final merge condition: align in (u,v)∧align out (u,v).

[0118] b) Conflict resolution:

[0119] Branch path conflict handling: When different process samples within a group exhibit different branch paths at the same decision point:

[0120] Retain high-frequency paths: Count the frequency of each branch path in the sample, and retain the path with the highest frequency as the main path of the decision point in the fused process.

[0121] Handling of temporal conflicts: When there are conflicts in the execution order of certain activities in the process samples within the group, a decision node is inserted, that is, a judgment logic is inserted before the conflicting node in the process to express it clearly.

[0122] 3. Merging of main processes:

[0123] a) Backbone Initialization (Optional): The merge operation can begin with an initialized backbone process. This backbone process can:

[0124] Predefined by industry experts, representing the best practices or standard process framework in the field.

[0125] Alternatively, without predefining, you can start building directly from the first merged group process.

[0126] b) Large Model-Driven Merging: Merge the currently merged group process (or the first group process) into the main process. This process reuses the node fusion (semantic alignment, structural alignment) and conflict resolution (branch path conflict, timing contradiction handling) techniques mentioned above.

[0127] The large model is responsible for:

[0128] Identify the corresponding nodes and paths of the main process and the processes to be merged;

[0129] Perform semantic matching and structural alignment;

[0130] Detect and resolve new or previously unresolved conflicts during the merge process;

[0131] Ensure that the merged process is logically consistent.

[0132] Step 3: Manual correction and verification of key processes:

[0133] This step aims to ensure that the integrated key process backbone generated in the second step possesses a high degree of accuracy, rationality, and business applicability. The process will then undergo final verification and optimization through industry expert experience or a large-scale model consensus mechanism.

[0134] 1. Industry expert-led revision (preferred path):

[0135] a) Reasonableness Review: Industry experts, based on their deep domain knowledge and practical experience, assess the overall logical reasonableness and business feasibility of the process. They check whether the process complies with industry standards, best practices, and the actual constraints of the specific business scenario.

[0136] b) Accuracy verification: Experts carefully check whether the description of each activity node in the process accurately reflects the actual operation, whether the conditions set for the decision points are correct, and whether the inputs / outputs meet expectations, to ensure that the process model truly reflects the essence of the business.

[0137] c) Necessity Verification: Experts examine the necessity of each node. Redundant, repetitive, or non-core nodes are eliminated, process paths are optimized, and overall efficiency is improved. It is confirmed that no key nodes have been omitted.

[0138] d) Logical consistency check: Focus on checking whether there are potential infinite loops, unreachable paths or logical conflicts in the process (especially the timing or branch contradictions introduced after merging), to ensure that the process model structure is rigorous and the execution path is clear.

[0139] e) Semantic refinement and annotation: Optimize the naming and description of nodes and gateways to make them clearer and more unambiguous.

[0140] f) Value and compliance assessment: Assess whether the process effectively supports business objectives and confirm that it complies with relevant laws, regulations, or security requirements.

[0141] 2. Large-scale model consensus verification (an alternative when no experts are involved):

[0142] a) Multi-model voting mechanism: When industry expert resources are unavailable, deploy multiple independent Large Language Model (LLM) instances (which may use models with different architectures or different parameter levels).

[0143] b) Overall rationality vote: Each LLM independently evaluates the rationality of the entire process backbone in terms of contextual logic, business common sense and goal consistency, and gives a vote of "rational" or "needs correction" and the reasons.

[0144] c) Node Necessity Voting: Each LLM independently evaluates and votes on the necessity of each node in the process, explaining the basis for its judgment (e.g., functional duplication, inefficiency, etc.). That is:

[0145]

[0146] Where, ω m It is the weight of the m-th model, Vote nec (v) is the result of node v being voted on by multiple large models.

[0147] d) Conflict detection and voting: LLMs identify potential structural conflicts (infinite loops, unreachable nodes) and semantic conflicts (contradictory conditions, inconsistent inputs and outputs) in the process and perform voting and marking.

[0148] e) Results Summary and Decision-Making: Summarize the voting results and reasons for all LLMs:

[0149] To assess the overall rationality of the process and the necessity of each node, the voting results from different large language models are weighted and summed, and a fixed threshold is set. For unreasonable processes or unnecessary nodes that are below the threshold, corrections are triggered.

[0150] For conflict detection, any significant conflict identified by any model needs to be recorded and processed.

[0151] Based on feedback from all models, guide necessary automation or semi-automation modifications to the main process flow.

[0152] Its decision function is:

[0153]

[0154] Where v is the current node and η is the threshold for whether it is needed.

[0155] 3. Revision and Final Confirmation:

[0156] Based on the revisions suggested by industry experts or the consensus results of large-scale model voting, specific adjustments and optimizations were made to the core of key processes, including:

[0157] Modify or delete unreasonable / unnecessary nodes;

[0158] Adjust branch logic or gateway settings to resolve conflicts;

[0159] Refine node descriptions and path annotations;

[0160] Optimize the overall structure.

[0161] After correction, a small-scale second expert review or a rapid large-scale model verification may be conducted as needed to ensure that the correction does not introduce new problems.

[0162] The final output is an authoritative key process backbone that has been validated and optimized, representing best practices or core logic.

[0163] Step 4: Extraction and Verification of Process Node Elements:

[0164] This step aims to refine the core business element information corresponding to each process node from the original sample data, based on the authoritative key process backbone confirmed in step three, and to ensure the accuracy and applicability of these elements. The specific operations are as follows:

[0165] 1. Extraction of process node elements:

[0166] Elements ij =LLM element (Prompt element (decision i ,v j )

[0167] Among them, Elements ij It is a sample decision i v on the critical process P j Elements within a node.

[0168] a) Feature extraction Prompt design:

[0169] Write prompts and extract elements from each key activity node in the main process. The prompts include: task description, main process mermaid expression, node name and description, sample query and answer data, output element format, etc.

[0170] b) Large-scale model-driven feature extraction:

[0171] Using a Large Language Model (LLM), and based on a pre-designed prompt, various element information from key nodes in the sample data is extracted step-by-step according to the workflow. The output is a set of element information extracted from various related samples for each node.

[0172] 2. Element information relevance verification:

[0173] The extracted element information, the corresponding sample text fragments, and the definitions of their respective process nodes are then input back into the large model. That is:

[0174] Valid element =Check[LLM verify (Elements,v)>γ]

[0175] Here, γ is the threshold for testing the validity of the elements under this node.

[0176] The design of the Prompt instruction requires judgment on a large model:

[0177] Does this element information actually originate from the provided sample fragment?

[0178] Is this element information highly relevant to the business semantics of the target process node?

[0179] Is there any misinterpretation or over-interpretation?

[0180] The model outputs a relevance score (e.g., highly correlated / correlated / weakly correlated / uncorrelated) and the reasoning behind the judgment.

[0181] 3. Verification of the rationality of element information:

[0182] The extracted element information, its corresponding process node, and the context of the entire process backbone are input into the large model.

[0183] The Prompt instruction design requires the large model to judge the reasonableness of the element's information appearing at this node based on business common sense and domain knowledge. The model outputs a reasonableness score (e.g., reasonable / mostly reasonable / questionable / unreasonable) and specific points of contention / suggestions.

[0184] 4. Validation result processing and feature library construction:

[0185] a) For elements marked as unreasonable:

[0186] The original samples can be manually reviewed and confirmed based on the reasons provided by the verification model.

[0187] Alternatively, large models can be used to optimize or re-extract feature descriptions.

[0188] b) Construct a structured feature library:

[0189] The verified and corrected element information is organized according to process nodes and associated with its source samples to form a structured, high-quality process node element knowledge base.

[0190] Step 5: Node element merging and fusion verification:

[0191] This step aims to integrate the node element information extracted and verified from the sample data in step four, forming a standardized and comprehensive element knowledge system for each process node through intelligent fusion technology, and ensuring the reliability and business value of the fusion results. The specific operations are as follows:

[0192] 1. Node element integration:

[0193] a) Synonymous element merging: Using large language model technology, identify elements with different expressions but the same semantics extracted from different samples under the same node, and merge them into a unified expression;

[0194] b) Conflict Labeling: Clearly mark significant contradictions or differences between different samples regarding two elements under the same node. Retain conflict information and the identifier of the source sample for subsequent verification or manual judgment;

[0195] c) Redundancy Removal: Remove obviously duplicate or irrelevant element entries.

[0196] 2. Node element correlation verification (large model driven):

[0197] a) Contextual relevance verification: Utilize large-scale model analysis to extract and integrate the set of elements, verifying whether each element is highly relevant to the core business activities of its respective node. Eliminate "noise" elements that are clearly irrelevant to the node's goals or operations.

[0198] b) Process coherence verification: Check whether the node element supports its node position in the process:

[0199] Can the input elements be satisfied by the output elements of the upstream node?

[0200] Do the output elements meet the input element requirements of the downstream nodes?

[0201] The large model evaluates whether the set of factors ensures the smooth connection of the node in the process.

[0202] 3. Node element rationality verification (large model driven):

[0203] a) Business logic rationality: Utilize domain knowledge from the large model to assess whether the elements conform to industry common sense and business logic.

[0204] b) Clarity and operability: Assess whether the descriptions of the elements are clear, unambiguous, and sufficient to guide actual operations.

[0205] 4. Output and Iteration:

[0206] Output a structured element information database for each process node, after integration, cleaning, and verification. This database includes: unified and clear element descriptions, marked conflicts and suggested solutions, and element source sample identifiers (optional, for traceability).

[0207] Report any issues found during the verification process (such as missing elements, serious conflicts, or unreasonable items), and if necessary, trigger adjustments to the element extraction prompt or a small-scale review of sample data.

[0208] Step Six: Manual Correction of Process Node Elements:

[0209] This step aims to provide a final quality check on the standardized process node elements constructed in step five, ensuring their accuracy, completeness, and business applicability. Optimization and correction of the node elements are achieved through in-depth validation by domain experts or a large-scale model consensus mechanism.

[0210] 1. Industry expert-led revision (recommended approach):

[0211] a) Element accuracy verification: Industry experts, based on their professional knowledge, review the element descriptions under each node in the knowledge base item by item.

[0212] Verify that the terminology is accurate, unambiguous, and conforms to industry standards;

[0213] Verify whether the rules / constraints reflect the actual business requirements and are clearly stated;

[0214] Confirm that the input / output definitions are complete and accurate, with no key items missing;

[0215] Review whether the role / resource assignments are reasonable and supported by the necessary permissions;

[0216] Are the assessment indicators / risk settings realistic and measurable?

[0217] b) Factor rationality assessment: Experts assess the rationality of factors within the specific context of each node, including:

[0218] Is the set of elements logically consistent? Are there any contradictions (such as rule conflicts or input-output mismatches)?

[0219] Do the requirements of the elements align closely with the core functions and business objectives of the node? Are there any over-designed or redundant requirements?

[0220] Are the elements consistent with the element requirements of upstream and downstream nodes (such as input dependencies and output connections)?

[0221] c) Necessity assessment of elements: Experts rigorously examine the value of each element, including:

[0222] Is this element indispensable for the proper execution of node activities or the achievement of business objectives?

[0223] Are there any duplicate or overlapping elements? They can be merged or deleted.

[0224] Have any key elements that significantly impact the efficiency, quality, and compliance of node execution been overlooked?

[0225] d) Conflict and Variation Handling: Conduct a final review of the scene dependency elements or condition conflicts identified in Step 5, including:

[0226] Confirm that the scene division is reasonable and the labeling is clear;

[0227] Determine whether the conflict needs further resolution (such as clarifying priorities or unifying rules) or whether the marked state should be maintained;

[0228] Decide whether to include low-frequency or specific scenario elements in the node element knowledge.

[0229] e) Semantic refinement and structural optimization: Optimize the simplicity, consistency and readability of element descriptions to ensure that the knowledge base is easy to understand and use.

[0230] 2. Large-scale model consensus verification (an alternative when no experts are involved):

[0231] a) Independent review of multiple models: Deploy multiple independent Large Language Model (LLM) instances.

[0232] b) Element-level triple voting: Each LLM independently conducts three evaluation votes for each element under each process node in the knowledge base:

[0233] Reasonableness vote: Is this element logically reasonable within the node context? (Reasonable / Questionable / Unreasonable);

[0234] Accuracy vote: Are the element descriptions clear, accurate, and unambiguous? (Accurate / Needs correction / Inaccurate)

[0235] Necessity voting: Is this element essential to the node? (Necessary / Optional / Redundant).

[0236] The model needs to output the voting reasons and specific modification suggestions.

[0237] c) Conflict and consistency checks:

[0238] The evaluation was conducted using a voting process across multiple large language models, including:

[0239] Are there logical conflicts between elements within the same node?

[0240] Are there any upstream or downstream conflicts between elements (especially inputs and outputs) between nodes?

[0241] Re-evaluate the reasonableness of the conflicts / variants marked in step 5.

[0242] d) Results summary and decision-making, including:

[0243] The final state of each element is determined by majority voting (e.g., if two or more models consider an element to be "unreasonable" or "redundant", it is marked as needing correction / deletion).

[0244] Summarize all inaccuracies and suggested corrections pointed out by the models;

[0245] Summarize all new conflict points and consistency issues identified by the models;

[0246] The combined results generate a detailed correction list of node element knowledge.

[0247] 3. Correct node element knowledge:

[0248] A revised list generated based on specific opinions from industry experts or consensus reached through large-scale model voting;

[0249] Correct inaccurate, unreasonable, or redundant element descriptions or directly delete redundant items;

[0250] Supplement the identified key missing elements;

[0251] Optimize element semantics and structure;

[0252] Resolve conflicts (based on expert rulings or by adopting the majority model recommendations / rules).

[0253] The advantages of manual correction are significant: Since the fifth step has already fused massive amounts of sample elements, the total number of elements in the final node is greatly reduced (far less than the original sample data). This allows industry experts to efficiently and thoroughly complete the full review. Their judgments, based on domain experience, are typically far superior to automated methods in terms of accuracy, rationality, necessity assessment, and conflict resolution, and they can also uncover subtle business logic that models struggle to capture. Therefore, this approach is preferred.

[0254] Large-scale model voting provides a viable alternative: When expert resources are lacking, multi-model independent voting and cross-validation mechanisms provide a systematic and automated verification scheme. By leveraging the "collective intelligence" among models, the robustness and reliability of the judgment are improved, making it an effective alternative.

[0255] Step Seven: Evaluation and Quantitative Verification of the Validity of Procedural Knowledge

[0256] This step aims to quantify and evaluate, through rigorous experimental design, the improvement effect of the procedural knowledge of key process and node elements generated in step six on the reasoning performance of real-world business problems. A controlled experimental approach is used to verify the value of knowledge fusion, namely:

[0257]

[0258] Where, Δ acc >0 indicates that procedural knowledge injection is effective; ans i That is, answer i , is the i-th sample d i The standard answer; ans base (d i The answer is obtained through reasoning by a large model without adding procedural knowledge prompts. enh (d i () is the answer obtained through reasoning by adding procedural knowledge prompts.

[0259] The specific process is as follows:

[0260] 1. Evaluation Framework Design:

[0261] a) Evaluation objective: To verify whether injecting structured procedural knowledge (key processes + node elements) can significantly improve the reasoning accuracy, logical rigor, and decision reliability of large models in relevant business scenarios.

[0262] b) Core Method: Controlled Experiment Method. Using the same large model, perform two inference iterations on the same sample dataset:

[0263] Control group (Baseline): No procedural knowledge is added to the prompts.

[0264] Experimental Group (Enhanced): Procedural knowledge (key processes + node elements) is explicitly injected into the prompt words.

[0265] c) Evaluation indicators:

[0266] Reasoning seriousness: refers to the degree to which the reasoning process and logic on which the model's answer is based can be understood and explained by humans;

[0267] Source reliability: This mainly examines whether the information source on which the answer is based is reliable and accurate, and whether the original information source can be clearly traced.

[0268] Answer fidelity: This measures whether the answer is faithful to the user's question and whether the answer is accurate. In other words, it measures whether the content of the answer is consistent with the objective facts of the user's question and whether there are any errors, omissions, or distortions of the facts.

[0269] 2. Sample dataset:

[0270] Data source: Reuse the original sample data parsed in step one and the elements extracted in step four (to ensure knowledge consistency).

[0271] Standard answer: Each sample is pre-labeled by industry experts with the corresponding reasoning, source tracing, and answer content.

[0272] 3. Interpretation of assessment results and procedural knowledge:

[0273] a) Validation of effectiveness: If the experimental group is significantly better than the control group in the core indicators, then the value of procedural knowledge is confirmed.

[0274] b) Identifying weaknesses: Analyze the errors still existing in the experimental group and trace them back to specific process nodes or missing / contradictory elements. Identify knowledge gaps (such as uncovered abnormal scenarios).

[0275] c) Closed-loop optimization: Feedback the evaluation results to the preceding steps: revise the main process (steps two / three); supplement node elements (steps four / five); optimize the knowledge fusion strategy (step six).

[0276] Step 8: Tree-structured storage of procedural knowledge:

[0277] This step constructs a tree-structured knowledge storage system with key processes as the root, node elements as branches, and sample data as leaves, achieving hierarchical organization and efficient retrieval of procedural knowledge. The specific tree structure design is shown in Table 1 below:

[0278] Table 1 Tree Structure

[0279] Hierarchy Composition content Data source Root node Key process backbone Step two fusion result Primary branch Process node sequence Step three verification node Secondary branch Node element classification Step five element library Leaf node Associated sample data Original business sample

[0280] This architecture organically integrates procedural knowledge with business samples, providing full-dimensional support of "nodes-elements-samples" for scenario implementation.

[0281] The following is a specific implementation example to illustrate the present invention:

[0282] I. Construction of Procedural Knowledge:

[0283] The following uses the determination of liability in a traffic accident as an example. Based on 1185 sample data points, the procedural knowledge extracted according to the above steps is as follows:

[0284] 1. Key Processes:

[0285] Since the standard operating procedure for this case is relatively simple, the key process extracted is as follows:

[0286] Identification of illegal acts --> Determination of causation --> Application of law --> Determination of responsibility.

[0287] 2. Node elements:

[0288] Each node extracts many elements; here, only the first five are extracted for illustration:

[0289] ① Elements of the "Illegal Behavior Identification" node:

[0290] Identification of illegal elements: factors such as driver qualifications, vehicle condition, driving behavior, and environmental adaptability;

[0291] Determining illegality: Compare with specific legal provisions (down to the article / clause / item);

[0292] Dynamic element verification: Vehicle motion state and road environment adaptability review;

[0293] Qualification and compliance review: Verification of the match between the licensed vehicle type and the vehicle's technical parameters;

[0294] Determination of subjective fault: Assessment of the subjective culpability of the actor's breach of duty of care.

[0295] ② Elements of the "Causal Relationship Determination" node:

[0296] Direct cause analysis: The inevitable connection between the illegal act and the occurrence of the accident;

[0297] Indirect cause analysis: The role of illegal acts in amplifying the consequences of the accident;

[0298] Multi-causal competitive and cooperative handling: ranking of causal forces and interaction effects;

[0299] Dynamic risk prediction: Professional assessment of the predictability of accidents due to illegal activities;

[0300] Possibility of Avoiding Outcomes: Analysis of the Correlation Between Illegal Acts and the Avoidance of Harmful Outcomes.

[0301] ③ Elements of the "Applicable Law" node:

[0302] Handling conflicting legal provisions: the priority of application between the Basic Law and its implementing regulations;

[0303] Situations involving the transformation of liability: Determination of the connection between administrative violations and criminal offenses;

[0304] Disclaimer applies in special circumstances such as intentional injury by the victim;

[0305] Determining the compliance of elements of a crime: Examining the correspondence between illegal elements and the elements of a crime under the law;

[0306] Related to the purpose of regulation and protection: Whether the illegal act falls within the scope of protection of the legal provisions.

[0307] ④ Elements of the "Responsibility Division" node:

[0308] Criteria for determining primary and secondary responsibility: Criteria for determining responsibility include the proportion of causal force, degree of fault, and risk control capability;

[0309] Verification of exemption circumstances: exemption circumstances such as emergency avoidance / force majeure / third-party fault, etc.

[0310] Special terms apply: Special terms such as those for special vehicles / special road sections / special time periods;

[0311] Degree of infringement of right-of-way: Assessment of substantial damage to right-of-way priority;

[0312] Attention to duty of care hierarchy: a classification of the levels of duty of care for different traffic participants.

[0313] II. Evaluation of the effectiveness of procedural knowledge:

[0314] The validity verification results of the inference data evaluation based on the QwQ-32B large model, using 1185 sample data points and injecting procedural knowledge into the prompts, are shown in Table 2.

[0315] Table 2 Validation Results

[0316] Dimension Injected procedural knowledge cue word Uninjected procedural knowledge cue word Reasoning seriousness 0.8386 0.7599 Traceability reliability 0.7508 0.6592 Answer fidelity 0.7681 0.4988

[0317] The test results show that when using the same model for reasoning with prompts infused with procedural knowledge, the seriousness of reasoning, the reliability of tracing back to the source, and the fidelity of the answer are all significantly improved, proving that procedural knowledge is effective.

[0318] As will be apparent to those skilled in the art from the above description of the embodiments, the present invention provides a method for constructing procedural knowledge based on a large model, which has the following advantages:

[0319] 1. High efficiency and automation: By automatically extracting and verifying key process and node elements through large models, manual intervention is reduced, significantly improving processing speed and accuracy, and reducing error rate.

[0320] 2. Intelligent conflict resolution: By integrating processes using semantic alignment and structural alignment technologies, branch path conflicts and timing contradictions are effectively resolved, ensuring the logical coherence of the process and the rationality of business operations.

[0321] 3. Dual Quality Assurance: Combining manual correction by industry experts with a large-scale model voting mechanism, the accuracy and necessity of processes and elements are optimized, balancing the efficiency of automation with the advantages of human experience.

[0322] 4. Structured Knowledge Management: Adopting a tree-like storage architecture (key processes as roots, node elements as branches, and samples as leaves) to achieve hierarchical organization of procedural knowledge, facilitating rapid retrieval and reuse.

[0323] 5. Quantifiable performance improvement: The effect of procedural knowledge injection is verified through controlled experiments, which significantly enhances the seriousness of reasoning, the reliability of tracing the source, and the fidelity of the answer, providing clear proof of validity.

[0324] In addition, the present invention also provides a procedural knowledge construction system based on a large model. The system includes a key process acquisition module, a node element acquisition module, a procedural knowledge evaluation module, and a procedural knowledge storage module. It is applied to the procedural knowledge construction method based on a large model described in the above embodiments to realize the procedural knowledge construction based on a large model.

[0325] The procedural knowledge construction system based on a large model provided in this embodiment of the invention has the same implementation principle and technical effects as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in this embodiment can be referred to the corresponding content in the aforementioned method embodiment, and will not be repeated here.

[0326] Furthermore, referring to Figure 2 As shown, this embodiment of the invention also provides an electronic device, which may include a processor 10, a memory 11, a communication bus 12 and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, wherein the processor executes the computer program to perform the above-described method.

[0327] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units, microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control core of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 and calls data stored in the memory 11 to perform various functions and process data within the electronic device.

[0328] Furthermore, embodiments of the present invention also provide a storage medium storing one or more programs readable by a computing device, the one or more programs including instructions that, when executed by the computing device, cause the computing device to perform the above-described method.

[0329] Examples of computer-readable storage media herein include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, DVD-ROM, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or high-speed digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage system, optical data storage system, hard disk, solid-state drive, and any other system described above configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer such that the processor or computer can execute the computer program.

[0330] The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent systems, and servers. Furthermore, in one example, the computer program, along with any associated data, data files, and data structures, is distributed across a networked computer system, such that the computer program, along with any associated data, data files, and data structures, is stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0331] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, electronic devices, or computer program products, etc. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0332] It should be noted that the word "comprising" does not exclude the presence of components or steps not listed in the claims. The words "a" or "an" preceding a component do not exclude the presence of a plurality of such components. This invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer.

[0333] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0334] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for constructing procedural knowledge based on a large model, characterized in that, The method includes the following steps: S1. Use a large model to extract the key processes of problem reasoning from the sample data, and use the large model to verify the relevance and rationality of the key processes. S2. Classify the extracted key processes into multiple groups, and merge the process nodes and resolve conflicts within the same group. Then, merge each group of key processes into the main process through a large model. S3. Revise key processes and verify the overall rationality, accuracy, and necessity of process nodes. S4. Based on the revised and verified key processes, use the large model + Prompt to extract the element information of the corresponding process nodes from the sample data, and verify the relevance and rationality of the element information. S5. Merge node elements according to nodes using the large model, and verify the relevance and rationality of the element merging results. S6. Correct the elements in the process nodes and verify the rationality, accuracy and necessity of the elements in the process nodes. S7. Based on the sample dataset, evaluate the effectiveness of procedural knowledge for key processes and process node elements; S8. Store key processes, process nodes, node elements, and sample data in a tree structure.

2. The method for constructing procedural knowledge based on a large model according to claim 1, characterized in that, In S1, the large model extracts the key reasoning process from the sample data and outputs it in Mermaid format. The model output includes a complete reasoning chain containing key decision points, conditional branches, and derivation steps. A dual-model consensus mechanism is also set up during the verification of relevance and rationality. The iteration is terminated only when both large models agree that the process is rational. Finally, a standardized Mermaid file with version marking and a verification report are output.

3. The method for constructing procedural knowledge based on a large model according to claim 1, characterized in that, In step S2, the similarity and difference between processes are calculated by cosine similarity. The extracted key processes are classified into multiple groups, and all key processes in the same group are deeply integrated to form a unified and optimal process representation within the group.

4. The method for constructing procedural knowledge based on a large model according to claim 1, characterized in that, In S3, key processes are corrected and verified through industry expert experience or large model consensus mechanisms.

5. The method for constructing procedural knowledge based on a large model according to claim 1, characterized in that, In step S4, by writing prompt words, the large model can extract node features. The prompt words include: task description, main process mermaid expression, node name and description of the process, query and answer data of the sample, and output feature format.

6. The method for constructing procedural knowledge based on a large model according to claim 1, characterized in that, In S5, the correlation verification includes: Contextual relevance verification: Utilize large model analysis to extract and integrate the set of elements, and verify whether each element is highly relevant to the core business activities of its respective node; Process coherence check: Check whether the node element supports its node position in the process; Reasonableness checks include: Business logic rationality: Utilize domain knowledge from the large model to assess whether the elements conform to industry common sense and business logic; Clarity and operability: Assess whether the descriptions of the elements are clear and unambiguous.

7. The method for constructing procedural knowledge based on a large model according to claim 1, characterized in that, In S6, the correction and verification of node elements are completed through in-depth verification by industry experts or a large model consensus mechanism. The large model consensus mechanism is as follows: multiple independent large models independently evaluate and vote on each element under each process node in the knowledge base, including reasonableness vote, accuracy vote and necessity vote.

8. The method for constructing procedural knowledge based on a large model according to claim 1, characterized in that, In S7, the effectiveness of procedural knowledge is evaluated and verified using a controlled experiment. Two methods are used to reason on the sample data through a large model: one method is to not add procedural knowledge to the reasoning prompts, and the other method is to add procedural knowledge to the reasoning prompts. The improvement in reasoning performance of the latter method compared to the former method is evaluated.

9. The method for constructing procedural knowledge based on a large model according to claim 1, characterized in that, In S8, knowledge is stored in a tree structure with key processes as the root, process nodes as first-level branches, node elements as second-level branches, and sample data as leaves.

10. A procedural knowledge construction system based on a large model, characterized in that, The system includes: a key process acquisition module, a node element acquisition module, a procedural knowledge evaluation module, and a procedural knowledge storage module. When applied, it executes a procedural knowledge construction method based on a large model as described in any one of claims 1-9 to realize the construction of procedural knowledge based on a large model.

Images