Method and device for evaluating and analyzing reasoning ability of large model based on thinking data

Abstract

The invention discloses a thinking data-based large model reasoning ability evaluation and analysis method and device. According to the method, a data flow diagram is constructed through a variable use-definition chain during running of a specific language LMCL in the dynamic monitoring field, logic variable LVAR nodes are extracted, redundant edges are eliminated, and a thinking map with a direct dependency relationship is generated. Based on the thinking map, a five-dimensional evaluation system including reasoning efficiency, key node recognition capability, reasoning generality, multi-path reliability and accumulative hierarchical reasoning is designed, and an internal mechanism of a model reasoning process is quantitatively analyzed. A semantic rule is extracted through a frequent mode of mining successful and failed thinking data, reasoning path probability distribution is integrated in combination with a path aggregation strategy, an interpretable cue word optimization strategy is generated, and a large model reasoning process is dynamically injected. Through the rule guidance and path equalization strategy, the model reasoning accuracy is improved, and the problems of thinking data isomerism, single evaluation and black box enhancement in the traditional technology are solved.

Description

Technical Field

[0001] This invention relates to the field of large model analysis technology, specifically to a method and apparatus for evaluating and analyzing the reasoning ability of large models based on thought data. Background Technology

[0002] In recent years, large-scale models have been increasingly applied to complex reasoning tasks. Researchers have significantly improved the logical reasoning ability of models by generating intermediate reasoning steps (such as CoT and Self-Consistency). Building on this, structured methods such as Mind Tree (ToT) and Mind Graph (GoT) have further expanded the diversity of reasoning paths, exploring multi-path solutions and achieving backtracking optimization through tree-like or graph-like frameworks. Meanwhile, a series of evaluation benchmarks (such as BLEU, ROUGE, and TruthfulQA) have been proposed to measure the accuracy, coherence, and reliability of model reasoning results. Some studies have also attempted to enhance the reasoning ability of models through reinforcement learning or few-shot learning strategies (such as STaR and XoT).

[0003] Currently, these technologies still face multiple challenges in practical applications: the heterogeneity of thought data makes cross-task transfer difficult, and the lack of a unified data model makes it difficult to effectively integrate the reasoning processes generated by different models; most existing enhancement methods rely on black-box operations, resulting in insufficient interpretability of reasoning paths and difficulty in tracing the root cause of errors; the evaluation system focuses too much on the correctness of the results and neglects the quantitative analysis of the logic of the reasoning process, the ability to identify key nodes, and the efficiency of multi-path decision-making, resulting in a lack of targeted guidance for model optimization.

[0004] Therefore, how to invent a large-scale model reasoning ability evaluation and analysis method that can improve the efficiency and accuracy of transition metal catalyst research has become an urgent problem to be solved. Summary of the Invention

[0005] To address this, the present invention provides a method and apparatus for evaluating and analyzing large-scale reasoning ability based on thought data. An automatic mind map construction algorithm ensures the theoretical correctness of the thought data. Simultaneously, five indicators are used to quantify and analyze the reasoning process, making the evaluation more comprehensive and interpretable.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for evaluating and analyzing the reasoning ability of large-scale models based on thinking data, comprising:

[0007] A data flow graph is constructed by dynamically monitoring variables in the domain-specific language LMCL runtime using a -definition chain; logical variables LVAR are extracted from the data flow graph to generate an LVAR node set; indirect edges in the LVAR node set are removed, and direct edges are retained to generate a preprocessed LVAR node set; redundant edges in the preprocessed LVAR node set are eliminated through breadth-first search and strongly connected component analysis strategies to construct a mind graph.

[0008] Based on the mind map, a five-dimensional evaluation system is designed, including reasoning efficiency, key node identification ability, reasoning commonality, multi-path reasoning reliability, and cumulative hierarchical reasoning. The reasoning ability of the large model is evaluated through the five-dimensional evaluation system, and evaluation analysis results are generated.

[0009] Based on the evaluation and analysis results, frequent patterns are mined from the thinking data of success and failure, and explanatory rules are extracted from the frequent patterns; the probability distribution of reasoning paths is integrated through a path aggregation strategy; a prompt word optimization strategy is generated based on the explanatory rules and the probability distribution; the prompt words are optimized through the prompt word optimization strategy to generate optimized prompt words.

[0010] The optimized prompts are injected into the large model's inference process, and the large model performs accurate inference through rule guidance and path balancing strategies.

[0011] As a preferred solution for evaluating and analyzing the reasoning ability of large models based on thinking data, the monitoring statements include: assignment statements, if / while statements, and query statements, in the process of constructing the data flow graph by dynamically monitoring the variables of the domain-specific language LMCL runtime using the -definition chain.

[0012] The expression for the data flow graph is:

[0013] G = (V, E)

[0014] In the formula, G is the data flow graph; V is the edge set; and E is the vertex set.

[0015] As a preferred method for evaluating and analyzing reasoning ability based on large-scale models of thought data, the expression of the thought map is:

[0016] G S =(S,E′)

[0017] In the formula, G S Let S be a mind map; S be the set of LVAR nodes; and E′ be the set of direct edges of S.

[0018] As a preferred method for evaluating and analyzing the reasoning ability of large models based on cognitive data, the process of evaluating the reasoning ability of large models using the aforementioned five-dimensional evaluation system includes:

[0019] Evaluation of inference efficiency for large models using frequent subgraph mining:

[0020] If max v∈P1 |C(v)|>max v∈P2 If |C(v)| or L(P1)>L(P2), then the reasoning efficiency is: E(M1)>E(M2);

[0021] In the formula, M1 and M2 are the large models; P1 and P2 are the frequent patterns mined by M1 and M2, respectively; C(v) is the set of child nodes of node v; L(P) is the inference level at which the frequent pattern P first appears.

[0022] The ability to identify key nodes in a large model is evaluated using node influence scores.

[0023] The formula for calculating the node influence score is as follows:

[0024]

[0025] In the formula, NIS is the node influence score; w1, w2, w3 are user-defined weights, satisfying w1 + w2 + w3 = 1; ρ I,S R is the Pearson correlation coefficient. local For local stability; The average score of each node;

[0026] The similarity of inference distributions among large models is evaluated using KL divergence:

[0027] The expression for the KL divergence is:

[0028]

[0029] In the formula, x is the inference path of the large model, and P(x) and Q(x) are the probability distributions of the inference paths of the two large models, respectively.

[0030] Reliability evaluation of multi-path inference for large models using path aggregation method:

[0031] The formula for calculating the path aggregation probability value is:

[0032]

[0033] In the formula, P(v n |v0,h) represents the distance from node v1 to node v n Along the path The probability of; For mind mapping G S All subpaths that follow path h; For node v i-1 The set of child nodes;

[0034] The cumulative hierarchical inference capability of large models is evaluated using cumulative hierarchical inference scores:

[0035] The expression for the cumulative hierarchical inference score is:

[0036]

[0037] In the formula, CTRS is the cumulative hierarchical reasoning score; The weights assigned to each layer i; L is the total number of layers in each inference path; N is the total number of inference paths; δ(·) is a correct node matching function; v j,i and v′ j,i These are nodes on the actual reasoning path and nodes on the reference reasoning path, respectively.

[0038] As a preferred solution for the evaluation and analysis method of large-scale model reasoning ability based on thinking data, in the process of integrating the probability distribution of reasoning paths through the path aggregation strategy, several reasonings are performed on the same question to obtain several thinking maps; several successful reasoning paths are extracted from the several thinking maps; the several successful reasoning paths are sorted and classified according to their probabilities, and one path and its probability are selected from each category as an example of a prompt word.

[0039] This invention also provides a device for evaluating and analyzing the reasoning ability of large models based on thinking data, which, based on the above method for evaluating and analyzing the reasoning ability of large models based on thinking data, includes:

[0040] The mind graph construction module is used to dynamically monitor variables in the domain-specific language LMCL runtime using a -definition chain to construct a data flow graph; extract the logical variable LVAR from the data flow graph to generate an LVAR node set; remove indirect edges from the LVAR node set and retain direct edges to generate a preprocessed LVAR node set; and eliminate redundant edges in the preprocessed LVAR node set using a breadth-first search and strongly connected component analysis strategy to construct the mind graph.

[0041] The large model reasoning ability assessment module is used to design a five-dimensional assessment system based on the mind map, including reasoning efficiency, key node identification ability, reasoning commonality, multi-path reasoning reliability, and cumulative hierarchical reasoning; the reasoning ability of the large model is assessed through the five-dimensional assessment system, and assessment analysis results are generated.

[0042] The prompt word optimization module is used to mine frequent patterns from successful and failed thought data based on the evaluation and analysis results, and extract explanatory rules from the frequent patterns; integrate the probability distribution of reasoning paths through a path aggregation strategy; generate a prompt word optimization strategy based on the explanatory rules and the probability distribution; and optimize the prompt words through the prompt word optimization strategy to generate optimized prompt words.

[0043] The large model inference optimization module is used to inject the optimized prompts into the large model inference process, and enable the large model to perform accurate inference through rule guidance and path balancing strategies.

[0044] As a preferred solution for a large-scale model reasoning ability evaluation and analysis device based on thinking data, the mind map construction module includes the following monitoring statements during the construction of the data flow graph: assignment statements, if / while statements, and query statements, in the process of dynamically monitoring the variables of the domain-specific language LMCL runtime using a -definition chain.

[0045] The expression for the data flow graph is:

[0046] G = (V, E)

[0047] In the formula, G is the data flow graph; V is the edge set; and E is the vertex set.

[0048] As a preferred embodiment of a large-scale model reasoning ability evaluation and analysis device based on thinking data, the expression of the thinking map in the thinking map construction module is:

[0049] G S =(S,E′)

[0050] In the formula, G S Let S be a mind map; S be the set of LVAR nodes; and E′ be the set of direct edges of S.

[0051] As a preferred embodiment of a large-scale model reasoning ability evaluation and analysis device based on thinking data, the large-scale model reasoning ability evaluation module evaluates the reasoning ability of the large model using the five-dimensional evaluation system as follows:

[0052] Evaluation of inference efficiency for large models using frequent subgraph mining:

[0053] If max v∈P1 |C(v)|>max v∈P2 If |C(v)| or L(P1)>L(P2), then the reasoning efficiency is: E(M1)>E(M2);

[0054] In the formula, M1 and M2 are the large models; P1 and P2 are the frequent patterns mined by M1 and M2, respectively; C(v) is the set of child nodes of node v; L(P) is the inference level at which the frequent pattern P first appears.

[0055] The ability to identify key nodes in a large model is evaluated using node influence scores.

[0056] The formula for calculating the node influence score is as follows:

[0057]

[0058] In the formula, NIS is the node influence score; w1, w2, w3 are user-defined weights, satisfying w1 + w2 + w3 = 1; ρ I,S R is the Pearson correlation coefficient. local For local stability; The average score of each node;

[0059] The similarity of inference distributions among large models is evaluated using KL divergence:

[0060] The expression for the KL divergence is:

[0061]

[0062] In the formula, x is the inference path of the large model, and P(x) and Q(x) are the probability distributions of the inference paths of the two large models, respectively.

[0063] Reliability evaluation of multi-path inference for large models using path aggregation method:

[0064] The formula for calculating the path aggregation probability value is:

[0065]

[0066] In the formula, P(v n |v0,h) represents the distance from node v1 to node v n Along the path The probability of; For mind mapping G S All subpaths that follow path h; For node v i-1 The set of child nodes;

[0067] The cumulative hierarchical inference capability of large models is evaluated using cumulative hierarchical inference scores:

[0068] The expression for the cumulative hierarchical inference score is:

[0069]

[0070] In the formula, CTRS is the cumulative hierarchical reasoning score; The weights assigned to each layer i; L is the total number of layers in each inference path; N is the total number of inference paths; δ(·) is a correct node matching function; v j,i and v′ j,i These are nodes on the actual reasoning path and nodes on the reference reasoning path, respectively.

[0071] As a preferred embodiment of a large-scale model reasoning ability evaluation and analysis device based on thinking data, the prompt word optimization module performs several reasoning operations on the same question to obtain several thinking maps during the process of integrating the probability distribution of reasoning paths through the path aggregation strategy; extracts several successful reasoning paths from the several thinking maps; sorts and classifies the several successful reasoning paths according to their probabilities, and selects one path and its probability from each category as an example of a prompt word.

[0072] This invention has the following advantages: It constructs a data flow graph by dynamically monitoring variables in the domain-specific language LMCL runtime using a definition chain; extracts the logical variable LVAR from the data flow graph to generate an LVAR node set; removes indirect edges from the LVAR node set, retaining direct edges, to generate a preprocessed LVAR node set; eliminates redundant edges in the preprocessed LVAR node set using breadth-first search and strongly connected component analysis strategies to construct a mind graph; based on the mind graph, it designs a system that includes inference efficiency, key node identification capability, inference commonality, multi-path inference reliability, and cumulative hierarchical inference... This invention employs a five-dimensional evaluation system to assess the reasoning ability of a large model, generating evaluation analysis results. Based on these results, frequent patterns are mined from successful and failed thought data, and explanatory rules are extracted from these patterns. A path aggregation strategy is used to integrate the probability distribution of reasoning paths. Based on the explanatory rules and the probability distribution, a prompt word optimization strategy is generated. The prompt words are then optimized using this strategy to generate optimized prompt words. These optimized prompt words are injected into the large model's reasoning process, and rule-guided and path balancing strategies are used to ensure accurate reasoning. This invention standardizes the modeling of thought data by constructing a domain-specific language (LMCL), solving the cross-task migration barrier caused by data heterogeneity in traditional methods. It dynamically monitors variable dependencies during program execution and automatically generates thought maps, ensuring the integrity and traceability of reasoning paths. Based on the five-dimensional evaluation system, this invention, for the first time, incorporates core capabilities such as reasoning efficiency, key node identification, and multi-path reliability into a quantitative analysis framework, breaking through the limitations of traditional evaluations that only focus on the correctness of results, and providing multi-dimensional improvement directions for model optimization. This invention mines frequent patterns and extracts semantic rules from successful and failed reasoning cases, combining this with path aggregation strategies to generate interpretable prompts. This transforms the black-box operation of the reasoning process into transparent decision-making, improving the accuracy of model reasoning. The technical system of this invention forms a closed-loop chain of "data modeling → quantitative evaluation → strategy optimization → task enhancement," which not only accurately identifies weaknesses in model reasoning but also achieves continuous iterative improvement of reasoning capabilities through dynamic injection of optimization rules, providing a systematic solution for large-scale model optimization in complex task scenarios. Attached Figure Description

[0073] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0074] The structures, proportions, sizes, etc. illustrated in this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed herein, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0075] Figure 1 This is a flowchart illustrating the large-scale model reasoning ability evaluation and analysis method based on thinking data provided in Embodiment 1 of the present invention.

[0076] Figure 2 This is a schematic diagram of the BNF language definition in the large model reasoning ability evaluation and analysis method based on thinking data provided in Embodiment 1 of the present invention;

[0077] Figure 3 This is a schematic diagram of the LMCL code in the large model reasoning ability evaluation and analysis method based on thinking data provided in Embodiment 1 of the present invention;

[0078] Figure 4 This is a pseudocode diagram of the mind map automatic construction algorithm in the large model reasoning ability evaluation and analysis method based on thinking data provided in Embodiment 1 of the present invention;

[0079] Figure 5 This is a schematic diagram of a mind map under the probabilistic graphical model in the large-scale model reasoning ability evaluation and analysis method based on thinking data provided in Embodiment 1 of the present invention;

[0080] Figure 6 This is a schematic diagram of mind map construction in the large-scale model reasoning ability evaluation and analysis method based on thinking data provided in Embodiment 1 of the present invention;

[0081] Figure 7 This is a schematic diagram illustrating the evaluation of reasoning efficiency between large models A and B using frequent patterns in the large model reasoning ability evaluation and analysis method based on thinking data provided in Embodiment 1 of the present invention.

[0082] Figure 8 This is a schematic diagram of path aggregation in the mind map of the large model reasoning ability evaluation and analysis method based on thinking data provided in Embodiment 1 of the present invention;

[0083] Figure 9 This is a schematic diagram of the rules extracted by the large-scale model reasoning ability evaluation and analysis method based on thinking data provided in Embodiment 1 of the present invention.

[0084] Figure 10 This is a schematic diagram of the experimental task overview in a mind map provided in one possible embodiment of the present invention, as shown in Embodiment 1.

[0085] Figure 11 This is a schematic diagram of the reasoning efficiency evaluation results in a mind map in one possible embodiment of Embodiment 1 of the present invention;

[0086] Figure 12 This is a schematic diagram of the multi-dimensional indicator evaluation results in a mind map in one possible embodiment of Embodiment 1 of the present invention;

[0087] Figure 13 This is a schematic diagram of KL divergence between different models in a mind graph provided in one possible embodiment of the present invention, as shown in Embodiment 1.

[0088] Figure 14 This is a schematic diagram illustrating the performance optimization of cue words on three tasks in a mind map in one possible embodiment of Embodiment 1 of the present invention;

[0089] Figure 15 This is a schematic diagram of the architecture of the large-scale model reasoning ability evaluation and analysis device based on thinking data provided in Embodiment 2 of the present invention. Detailed Implementation

[0090] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0091] Example 1

[0092] See Figure 1 Embodiment 1 of the present invention provides a method for evaluating and analyzing the reasoning ability of large models based on thinking data, including the following steps:

[0093] S1. By dynamically monitoring the variables of the domain-specific language LMCL runtime using the -definition chain, a data flow graph is constructed; logical variables LVAR are extracted from the data flow graph to generate an LVAR node set; indirect edges in the LVAR node set are removed, and direct edges are retained to generate a preprocessed LVAR node set; redundant edges in the preprocessed LVAR node set are eliminated through breadth-first search and strong connected component analysis strategies to construct a mind graph;

[0094] S2. Based on the mind map, design a five-dimensional evaluation system that includes reasoning efficiency, key node identification ability, reasoning commonality, multi-path reasoning reliability, and cumulative hierarchical reasoning; evaluate the reasoning ability of the large model through the five-dimensional evaluation system and generate evaluation analysis results.

[0095] S3. Based on the evaluation and analysis results, mine frequent patterns from the thinking data of success and failure, and extract explanatory rules from the frequent patterns; integrate the probability distribution of reasoning paths through a path aggregation strategy; generate a prompt word optimization strategy based on the explanatory rules and the probability distribution; optimize the prompt words through the prompt word optimization strategy to generate optimized prompt words.

[0096] S4. Inject the optimized prompts into the large model's inference process, and enable the large model to perform accurate inference through rule guidance and path balancing strategies.

[0097] In this embodiment, in step S1, a data flow graph is constructed by dynamically monitoring the variables of the domain-specific language LMCL runtime using a -definition chain; logical variables LVAR are extracted from the data flow graph to generate an LVAR node set; indirect edges in the LVAR node set are removed, and direct edges are retained to generate a preprocessed LVAR node set; redundant edges in the preprocessed LVAR node set are eliminated through a breadth-first search and strongly connected component analysis strategy to construct a mind graph.

[0098] LMCL is a domain-specific language for human-scale model interaction, used to model thought-data graphs. For example... Figure 2 As shown, the type system and syntax of LMCL are defined using the Backus-Naur Form. This language supports primitive types (Int, Float, Str, and the generic variable type LVar), as well as composite types, including arrays, tuples, and function types. Composite types support nested composition to express complex data structures and function signatures. LMCL's syntax is based on a declarative programming paradigm, where programs consist of declarations and tasks. Tasks, as the core logical carriers of the program, encapsulate various operations and interactions with the larger model.

[0099] In this context, a task is the basic building block of LMCL, defined by the keyword "task", and optionally followed by a list of tasks. <llm-config>Parameters are used to configure the large model's interactive environment. Each task consists of a parameter list and several statements that define its execution logic. This structure allows developers to flexibly specify input and output types for tasks and implement complex task decomposition and processing through nested logic.

[0100] LMCL supports both external function calls and internal function definitions. To ensure the reliability of the analytical algorithms relied upon during mind graph construction, LMCL restricts external function calls to only statically typed pure functions with the type signature <Type-static> → <Type-static>, and requires referential transparency. In contrast, internally defined functions are not subject to this constraint because inlining is applied during program compilation.

[0101] Statements are used to control the flow of a program in a specific task. In addition to classic program control logic, LMCL provides three statement types specifically for interaction with large models: (1) query is used to send queries to the large model; (2) reflect provides dynamic adjustment capabilities. By combining feedback from the large model, the program can adaptively adjust task parameters or modify operation logic based on runtime information, thereby achieving self-adjusting interaction; (3) prompt allows the generation of static or dynamic prompt words by combining templated strings and context embeddings, and is usually used as a tool to enhance context for interactive tasks. Based on these three dedicated statements, LMCL supports dynamic content generation and variable interpolation. In the query and reflect statements, each top-level string is treated as a query sent directly to the large model. These query strings allow two special escape subfields, similar to Python's f-strings: {LVAR} dynamically inserts variable values ​​in the current scope; [LVAR] represents a phrase generated by the large model, allowing the model to populate content in the prompt word.

[0102] like Figure 3 The image shows a schematic diagram of LMCL code, demonstrating reasoning based on thought data in a 24-point game task. The task defines the main program structure, including the configuration of the large model and decomposition and analysis functions for generating a set of candidate answers and evaluating each answer individually. The templated string prompt is also included. <tot-decompose-example>"and" <value-example>"Used to initialize the large model context, nums will dynamically replace placeholders. The keywords query and reflect enable the large model to solve and evaluate sub-problems, and store the output in "[CHILD]" and "[EVAL]" respectively.

[0103] In this embodiment, a data flow graph is constructed by dynamically monitoring the variables of the domain-specific language LMCL runtime using a -definition chain;

[0104] Specifically, during the monitoring phase, a data flow graph is constructed by dynamically monitoring the runtime behavior of the LMCL. When a variable is used, its user is recorded to establish relationships between variables. For array-type variables, they are treated as a group of variables, where each array element is considered an independent variable. The monitoring process gradually expands the data flow graph G = (V, E) as the LMCL program executes, where V is initialized with the set of program input variables; E is initially an empty set. Three types of statements in the program need to be monitored:

[0105] a1. Assignment Statement: Records the dependency relationship between the left-hand variable (var) and the right-hand expression (expr). If an assignment involves multiple variables (e.g., assignment between tuples), it is recursively split into multiple independent assignments. For array assignments, a dependency relationship is established for each index, i.e., var[i] ← expr[i]. The left-hand variable var is inserted as a new vertex v into the vertex set V of the data flow graph. For the right-hand expression expr, all variables in it and their corresponding vertices u1,...,u in the current vertex set V are recursively extracted. k Subsequently, by adding edges (u1,v),...,(u k The edge set E is expanded using V to represent data dependencies. It's important to note that whenever a variable with the same name is reassigned, a new vertex is created in V, ensuring an accurate representation of variable value evolution during program execution.

[0106] a2. if / while statements: Additional edges need to be added between the conditional expression `expr` and the variables `var` on the left-hand side of all assignment statements in the basic block controlled by that condition `expr`. If some variables are recognized as loop invariants in the `while` statement, or are executed uniformly in all conditional branches, then these edges are not needed. Let Vs be the set of vertices corresponding to the variables in `expr`, and V... d Let V be the set of vertices corresponding to the variables in var. Then, let V be the set of vertices corresponding to (u,v) (where u∈V). s And v∈V d Add it to edge set E.

[0107] a3. Query Statement: For each large model call, add an edge between {LVAR} and [LVAR]. Each suggestion word triggers an independent large model call. In this case, each {LVAR} in the query has a direct dependency on [LVAR]. However, in the reflect operation, {LVAR} and [LVAR] are of type Str, and therefore are not included in the analysis.

[0108] In this embodiment, the logical variable LVAR is extracted from the data flow graph to generate an LVAR node set; indirect edges in the LVAR node set are removed, and direct edges are retained to generate a preprocessed LVAR node set; redundant edges in the preprocessed LVAR node set are eliminated through breadth-first search and strong connected component analysis strategies to construct a mind graph.

[0109] Specifically, in the graph construction phase, the focus is on the data flow relationships between all LVAR variables to build a mind map. First, the relationships within the set of nodes S (e.g., all LVAR nodes) in the data flow graph G are formally defined.

[0110] For a data flow graph G = (V, E), let... Let S be a subset of G. Given nodes u, v ∈ S, if (u, v) ∈ E, or if there exists a simple path from u to v that does not contain any other nodes in S, then there is a direct edge between u and v, denoted as E. The goal of mind mapping is to capture the information flow relationships between all LVAR variables. In the data flow graph G, let S be the set of all LVAR nodes. Furthermore, S is the set of nodes in the mind graph, and its edge set contains all direct edges from S in G.

[0111] For a data flow graph G = (V, E), with LVAR node set S, its mind map is defined as G. S = (S, E′), where (u, v) ∈ E′ if and only if there exists a direct edge in G.

[0112] Specifically, mind map construction algorithms such as Figure 4 As shown:

[0113] b1. Given a data flow graph G = (V, E) and a set of LVAR nodes. Use BFS to preprocess connectivity:

[0114] Specifically, extract all edges in G whose endpoints are both in S, forming the edge set E. S .

[0115] b2. Subsequently, construct the quotient graph G′=G\(S,E) S The graph G has removed all nodes belonging to S and their corresponding edges.

[0116] b3. Perform Breadth-First Search (BFS) on G′ to determine the strong connectivity between nodes in G′. During the BFS traversal, for each unvisited node u, the algorithm adds it to a queue, marks it as visited, and records its predecessor set φ(u). As the BFS progresses, for each unvisited successor node k of node v, they are added to a queue and their predecessor set φ(k) is updated to be the same as φ(u).

[0117] b4. Based on the connectivity relationships obtained in b1, construct mind map G. S The edge set of G such that G S The edges in G are either from existing edges in G or from paths in G′ that do not contain LVAR nodes, ensuring that all edges are direct edges in G (lines 12-19).

[0118] Specifically, the algorithm computes the predecessor set ψ(u) for each variable u∈S, which contains all predecessor nodes of u in G that are not in S. For each u∈S, the algorithm iterates through its successor node k in G that is not in S, and for each such k, it iterates through all v∈S. If the intersection of φ(k) (i.e., the predecessor set of k in G′) and ψ(v) (i.e., the non-S predecessor set of v in G) is not empty, then the edge (u,v) is added to E. res .

[0119] This step ensures that the edges in the mind graph reflect the direct dependencies between nodes in S, while avoiding the introduction of redundancy or indirect dependencies.

[0120] b5. Return to Mind Map G S =(S,E res ), where E res These are all the direct edges identified through the steps described above.

[0121] In this embodiment, the time complexity of the automatic mind graph construction algorithm mainly includes two parts: the preprocessing stage and the direct edge determination stage. The preprocessing stage involves constructing the quotient graph G′=G\(S,E). S The process involves performing a BFS (Browse-First Search) operation on the nodes to determine connectivity. This process requires traversing all nodes and edges, resulting in a time complexity of O(|V|+|E|). The operation to determine direct edges primarily involves judging pairwise combinations of all nodes in S, thus its time complexity is O(|S|). 2 ·d), where d represents the maximum out-degree of a node in set S; therefore, the total time complexity of the entire algorithm is O(|V|+|E|+|S|). 2 ·d).

[0122] In this embodiment, the probability distribution of each inference path can be quantified under the probabilistic graphical model.

[0123] In the mind map G=(V,E), each node v(o′) represents the output generated by the large model during the inference process. If the output o′ is based on the previous inputs o1, o2, ..., o n And based on the clue word p, the node is defined as follows: Where v(o′)∈V. Each edge e(o′,o) represents the dependency relationship between nodes v(o′) and v(o). When node v(o′) is added, based on P(o′|o1,o2,...,o n This will form a set of edges {e1(o1,o′),...,e n (o n The mind map, denoted as ,o′)}∈E, updates dynamically with the interaction between humans and the large model, and is recursively represented as G. i =(V i E i ), where V i =V i-1 ∪V add E i =E i-1 ∪E add V add (or E) add () represents a newly generated node (or edge) in each human-large model interaction. The reasoning path in the mind graph G can be represented as:

[0124]

[0125] In the formula, Pre(oi) represents o i The set of preceding inputs. This representation ensures that the output of each node can be represented as the product of conditional probabilities along its inference path, thus providing a theoretically sound probabilistic framework for the correct construction of the graph.

[0126] like Figure 5 As shown, a mind map is presented for the array sorting task.

[0127] Node v(o4) is generated from v(o2) and v(o3), that is... The construction of edges e(o2,o4) and e(o3,o4) is based on P(o4|o2,o3). In the next reasoning step, node v(o5) is introduced, and edge e(o4,o5) is added. The complete thought data graph G is recursively constructed as follows:

[0128] P(o5|o0,o1,o2,o3,o4)=P(o5|o4)P(o4|o2,o3)P(o3|o1)P(o2|o1)P(o1|o0)

[0129] In one possible embodiment, Figure 6 This demonstrates the automatic construction process of the mind map. During runtime, `root` is first assigned the value `nums`, creating the node `{root,nums}` and an edge `{(root,nums)}`. Subsequently, `nums` is provided as input to the larger model via a query. Inside the `while` loop, `cnt` establishes edges with all variables within the loop body. In each iteration, `CHILD` receives the output of the larger model and adds an edge `{(nums,CHILD)}`. Furthermore, a new node is created for `cnt` in each iteration. When n=4, the complete data flow G is as follows: Figure 6 As shown in (a), only LVAR nodes and their direct edges and simple paths are retained to construct the mind graph. Figure 6 (b) A mind map G is shown when the input is "1,3,6,7" in the 24-point game task. S .

[0130] In this embodiment, in step S2, based on the mind map, a five-dimensional evaluation system is designed, including reasoning efficiency, key node identification ability, reasoning commonality, multi-path reasoning reliability, and cumulative hierarchical reasoning; the reasoning ability of the large model is evaluated through the five-dimensional evaluation system, and evaluation analysis results are generated.

[0131] Specifically, in order to comprehensively measure the reasoning characteristics of large models and evaluate their reasoning capabilities, five evaluation dimensions were designed:

[0132] Reasoning efficiency measures a model's ability to arrive at the correct solution using fewer intermediate steps and shorter reasoning paths. It reflects both efficiency and accuracy by measuring a model's ability to avoid redundant or erroneous reasoning in complex tasks.

[0133] Crucial Node Judgment: Evaluates the model's ability to identify key reasoning steps that significantly impact the validity of the answer, thereby improving the accuracy and efficiency of reasoning.

[0134] Reasoning Commonality: This analyzes whether different large models employ similar reasoning paths, strategies, and thought processes when solving problems. Higher similarity in the model's reasoning process indicates stronger global stability and universality of the reasoning path.

[0135] Multipath Reasoning Reliability: This refers to the ability of a large model to evaluate the reasonableness of multiple reasoning paths from a question to an answer and their credibility to the final result. It reflects how effectively the model integrates information from multiple paths while filtering out invalid reasoning paths.

[0136] Cumulative Tiered Reasoning: This evaluates the ability of large models to maintain logical correctness and consistency as reasoning depth increases. It emphasizes the ability of large models to prevent error accumulation in multi-step logical reasoning, ensuring stable and reliable performance in complex problem-solving scenarios.

[0137] In this embodiment, Frequent Subgraph Mining (FPM) is used to evaluate the inference efficiency of large models.

[0138] Frequent patterns capture recurring subgraph structures within the inference path, providing insights into whether the model converges efficiently to a solution or experiences frequent backtracking. Patterns in the mind graph represent model behavior during inference, and their topology reflects decision efficiency. Excessive edges in a pattern indicate excessive backtracking and low inference efficiency. Furthermore, since the output of the large model is generated, the position of frequent patterns in the inference graph reveals temporal inference behavior—lower positions indicate early backtracking and inefficient inference.

[0139] Specifically, given two frequent patterns P1 and P2 mined from large models M1 and M2, if max v∈P1 |C(v)|>max v∈P2 |C(v)|, where C(v) represents the set of child nodes of node v; or if L(P1) > L(P2), where L(P) represents the inference level at which the frequent pattern P first appears, then the inference efficiency E(M1) > E(M2).

[0140] In this embodiment, the Node Influence Score (NIS) is used to evaluate whether the large model assigns higher scores to key nodes that are more likely to arrive at the correct answer.

[0141] Specifically, define node influence I i The importance of a node is measured by its frequency of occurrence in successful reasoning paths. Specifically, for all categories of nodes V' and successful paths P in all mind graphs to be evaluated... S For each node v i ∈V′, its node influence I i for:

[0142]

[0143] In the formula, It is an indicator function, when node v i The value is 1 if it appears in the successful path p, and 0 otherwise, which is the node's own successful path marker.

[0144] The evaluation scores of various mind maps are measured. Based on the influence of each node (I). i With the corresponding average evaluation score S i This invention uses the Pearson correlation coefficient ρ I,S To calculate the global correlation between the two.

[0145] This paper calculates the local correlation ρ for each subset of nodes of category j in V'. j .

[0146]

[0147] In the formula, m represents the number of local correlations; This is the mean of these local correlations;

[0148]

[0149] In the formula, Var(ρ) represents the variance.

[0150] Smaller variance indicates consistent local correlation, while larger variance indicates high volatility. This invention normalizes the variance to represent the node influence I. i And node score S i Local stability between:

[0151]

[0152] In the formula, R local For local stability;

[0153] To facilitate subsequent comprehensive evaluation, this invention normalizes the average node score as follows:

[0154]

[0155] In the formula, Score the nodes; The average score of each node;

[0156] The formula for calculating node influence score is:

[0157]

[0158] In the formula, NIS is the node influence score; w1, w2, w3 are custom weights that satisfy w1+w2+w3=1.

[0159] NIS evaluates the reasoning ability of large models through three key factors. A higher ρ (probability of reasoning) I,S This indicates a strong consistency between high-influence nodes and high-scoring nodes, reflecting the model's effectiveness in identifying key nodes. Higher local stability R0 local This demonstrates the consistency of the large model in identifying high-influence nodes across different node subsets and categories. Lower... This indicates that the model is too conservative and fails to highlight key nodes. By integrating these factors, a higher NIS value indicates that the large model can more accurately and consistently identify high-influence nodes at both the macro and micro levels.

[0160] In this embodiment, KL divergence is used to quantitatively measure the similarity of inference distributions between different large models, and its formula is expressed as:

[0161]

[0162] In the formula, x is the inference path of the large model, and P(x) and Q(x) are the probability distributions of the inference paths of the two large models, respectively.

[0163] Specifically, the probability distribution of the inference path is first constructed using a random walk method. The specific steps are as follows:

[0164] c1. Starting from a fixed node v0∈V, generally the root node;

[0165] c2. In each step t, from the current node v t The set of child nodes C(v) t Randomly select the next node v in ) t+1 until the target node is reached or the maximum path length L is reached. max ;

[0166] c3. Repeat the above steps until the termination condition is met, thereby obtaining the path set.

[0167] c4. Encode the path sequence into a fixed format, count the frequency of each path, and calculate the probability distribution by normalizing the total number of paths. To avoid the zero probability problem in logarithmic calculations, a small smoothing term is added to the probability distribution for smoothing.

[0168] Then, based on the model and Path distribution and Calculate the KL divergence between them Lower KL divergence typically indicates that large models have more efficient path selection and stronger inference capabilities. Simultaneously, lower KL divergence also suggests greater global stability and generalization ability. Therefore, we use a weighted average KL divergence to evaluate the inference ability of large models, denoted as:

[0169]

[0170] In the formula, w j D represents the normalized weights of the target model j, indicating the proportion of its accuracy to the total accuracy; KL (i‖j) represents the KL divergence between model i and model j.

[0171] In addition to the random path sampling method mentioned above, this invention also employs a weighted path sampling method, which assigns higher sampling weights to high-probability paths. These high-probability paths are more likely to produce accurate answers, and their distribution reflects the model's ability to select more efficient inference paths. To obtain the distribution of high-probability paths, this invention defines node v based on node scores. i and v j The transition probability between them is denoted as:

[0172]

[0173] In the formula, S j Represents node v j The score is then calculated based on these high-probability path distributions, resulting in a weighted average KL divergence for the large model.

[0174] In this embodiment, as Figure 7 As shown, this demonstrates the evaluation of inference efficiency for large models using frequent patterns. Consider two frequent patterns, P1 and P2, extracted from the inference graphs of large models A and B, respectively. Since... and Larger model B exhibits more backtracking during the reasoning process. Meanwhile, L(P1) = 2 > L(P2) = 1, meaning B backtracks earlier. Therefore, larger model A is more efficient at reasoning than B.

[0175] In this embodiment, path aggregation is used to evaluate the reliability of multi-path reasoning in a large model. This method reflects the model's ability to integrate multiple reasoning paths and generate the correct answer. Path aggregation forms a comprehensive distribution by weighting the probabilities of multiple reasoning paths, reflecting the model's confidence level and preference for various potential reasoning paths.

[0176] First, traverse the mind map G. S And identify the path from input node v0 to output node v n Let the set of all reasoning paths be denoted as . Inspired by the classic path ranking algorithm (PRA), this invention defines the aggregation of selected paths as a weighted sum of the probabilities of all inference paths connecting two target nodes. More specifically, it focuses on the potential edge e(v0, v n The probability distribution P under ) w (v n |v0,e), which takes the following form:

[0177]

[0178] Among them, w s (h) is the weight assigned to path h, defined as the weight between this path and the successful inference path P. S degree of overlap:

[0179]

[0180] For edge e(v0,v) n Consider the reasoning path in conjunction form:

[0181] e(v0,v n )←e1(v0,v1)∧e2(v1,v2)∧…∧e n (v n-1 ,v n )

[0182] Among them, v i ∈V and e i ∈E. Such a path can be simplified as h=[e1,e2,...,e...]. n The set of all possible paths can be formally represented as:

[0183]

[0184] From node v1 to node v n Along the path The probability P(v) n |v0,h) can be represented as the sum of probabilities of all possible paths, where path h = [e1,e2,...,e...]. n ]:

[0185]

[0186] in, In mind map G S The path follows all subpaths of path h, while P(v) i |v i-1 ,e i Depends on node v i-1 The number of child nodes is defined as:

[0187]

[0188] In the formula, Represents node v i-1 The set of child nodes.

[0189] The path aggregation probability value represents the model's confidence and reliability in the reasoning process from question to answer. A higher value indicates that the model assigns greater probability weights to the main reasoning paths, reflecting stronger logical discrimination and information integration capabilities, thus making the reasoning process more focused and coherent. Conversely, a lower value indicates a more dispersed path distribution, higher uncertainty, or weaker adaptability to complex problems.

[0190] In this embodiment, as Figure 8 As shown, a mind map G is presented. S Path aggregation prediction P w (v 10 Examples of |v1,e).

[0191] From node v1 to node v 10 There are two paths, h1 = [e1, e2, e3] and h2 = [e1, e4, e5, e6]. Therefore, the probability of path aggregation is calculated as follows:

[0192]

[0193] Assume path h1 is completely consistent with the successful inference path P S They overlap, and path h2 has 3 edges that connect to P. S If they overlap, then... and Therefore, the probability of path aggregation is: P w (v 10 |v1,e)=0.208.

[0194] In this embodiment, in step S3, based on the evaluation and analysis results, frequent patterns are mined from the thinking data of success and failure, and explanatory rules are extracted from the frequent patterns; the probability distribution of reasoning paths is integrated through a path aggregation strategy; a prompt word optimization strategy is generated based on the explanatory rules and the probability distribution; the prompt words are optimized through the prompt word optimization strategy to generate optimized prompt words.

[0195] In this process, rules are extracted from frequent patterns mined from thought data and incorporated into prompt words.

[0196] Specifically, the reasoning data is divided into two categories: successful reasoning data that leads to the correct result and failed reasoning data that fails to produce the correct result. These data are then transformed into labeled mind maps G. c and G u By extracting frequent patterns from these two types of mind maps, we can identify G respectively. c Good frequent thinking patterns and G u This study identifies and extracts rules from frequent, undesirable thought patterns within the clusters. Based on the dominant semantic features of node clustering information for each pattern, GPT-4o is used to summarize the semantics of node clustering labels, thereby extracting rules. Rules derived from good patterns help large models learn how to arrive at 24 more quickly and concisely, while rules derived from undesirable patterns help large models avoid common errors and improve inference performance. Subsequently, by matching GPT-4o... c and G u This method identifies frequent good and bad thought patterns in thought data, obtaining a series of frequent subgraphs, which are then used as examples in the prompt words. These rules and their corresponding examples are further integrated into the prompt words to enhance the reasoning ability of the large model. By extracting rules from frequent patterns, the prompt words can be optimized for specific tasks, thereby improving accuracy and robustness. Furthermore, this method can automatically mine patterns in thought data and generate explanatory rules without human intervention.

[0197] In this embodiment, a path aggregation strategy is used to integrate the probability distribution of inference paths; a prompt word optimization strategy is generated based on the explanatory rules and the probability distribution; the prompt words are optimized using the prompt word optimization strategy to generate optimized prompt words.

[0198] Specifically, the core idea of ​​prompt word optimization through path aggregation is to improve the reasoning ability of a large model by integrating information from multiple reasoning paths. In this method, the same question is reasoned multiple times, generating different mind maps and extracting successful reasoning paths. Subsequently, the extracted paths are sorted and categorized according to probability, and one path and its probability are selected from each category as examples in the prompt words. High-probability (i.e., shorter) reasoning paths provide intuitive answers, while low-probability (i.e., longer) reasoning paths enhance the depth and logical rigor of the reasoning. By capturing the relationships between paths, the model can form a more comprehensive understanding of the reasoning process. Furthermore, this progressive path aggregation helps achieve incremental reasoning, enabling the model to strike a balance between efficiency and accuracy.

[0199] In this embodiment, an example of the rules extracted from the 24-point game is as follows: Figure 9 As shown.

[0200] In this embodiment, in step S4, the optimized prompt words are injected into the large model inference process, and the large model performs accurate inference through rule guidance and path balancing strategies.

[0201] Specifically, the optimized prompts from step S3 are injected into the large model's reasoning process. Through rule-guided and path balancing strategies, the large model can perform accurate reasoning.

[0202] In one possible embodiment, an experimental verification example is provided as follows:

[0203] The dataset used for experimental verification is as follows:

[0204] Six datasets were generated for the Game of 24, Blocksworld, and GSM8K tasks using breadth-first search (BFS) and depth-first search (DFS). An overview of these tasks is provided below. Figure 10 As shown. These tasks are complex, requiring multi-step reasoning and potentially having multiple solutions. In BFS reasoning, the large model generates 10 candidate sub-solutions for each subproblem and selects the 3 highest-scoring sub-solutions to continue exploring. In DFS reasoning, the reasoning process terminates when the large model finds the correct solution or when the number of reasoning nodes in the last layer reaches the maximum limit of 18. The temperature parameter is set to 0.95.

[0205] Models Used: This experiment was conducted on an NVIDIA A100-80G GPU. Each experiment was run three times, and the average results were reported. Six large API-call models were primarily tested on more complex tasks (such as the 24-point game and the Blocksworld task), while smaller open-source models showed near-zero accuracy on these tasks. For the simpler GSM8K task, large open-source models and GPT-3.5 were tested; other large API-call models performed equally well on this task, so they were not further evaluated. For detailed evaluation methods of mind maps for different tasks, please refer to [link to relevant documentation]. Figure 10 The number of clusters n = N v / 6, where N v This represents the number of node types in the mind graph, with a frequent pattern threshold of k=14 and a random walk round count of 100.

[0206] Reasoning ability assessment and analysis:

[0207] Evaluation results of different large models on different datasets under various metrics, as follows Figure 11 and Figure 12 As shown below, we will analyze in detail the advantages and disadvantages of large model reasoning capabilities for each indicator.

[0208] d1. Reasoning efficiency:

[0209] Figure 11 The |C| and L(P) values ​​reflect the inference efficiency of each model when using depth-first search in the "Blocksworld" and "Game of 24" tasks. The large model GPT-4o exhibits high inference efficiency in both tasks, with an average |C| value of 1.43, indicating that GPT-4o's inference path is more focused and has fewer branches, thus achieving higher inference efficiency. Other models exhibit varying degrees of backtracking during inference, with Glm-4-plus having a |C| value of 1.58; Qwen-ml 1.675; GPT-4 1.825; Claude-S3.5 1.87; and GPT-3.5 2.33, showing a gradual decrease in inference efficiency. For the L(P) value, larger models with stronger inference capabilities typically exhibit larger values, such as GPT-4o and GPT-4, indicating that these models can directly generate better solutions in the early stages of inference, thus avoiding additional backtracking overhead.

[0210] d2. Key node identification capability:

[0211] Figure 12 The NIS value measures the ability of each model to identify key nodes. In the Blocksworld and Game of 24 tasks, GPT-4 achieved the highest average NIS value of 0.711, demonstrating its significant advantage in identifying key nodes and showing superior performance in complex inference tasks. Qwen-ml and GPT-4o followed closely behind, while Glm-4-plus and Claude-S-3.5 scored 0.676 and 0.587 respectively, showing relatively low performance. However, GPT-3.5's NIS was only 0.350, and it showed a negative correlation in the Blocksworld task, indicating that the reasoning process of this large model could not be effectively carried out in this task. In the GSM8K task, GPT-3.5 and Qwen-1.5-7b had high NIS values, both exceeding 0.7, demonstrating strong ability to identify key nodes; while the NIS values ​​of other large models were around 0.5, showing only a moderate level of reasoning ability.

[0212] d3. Commonalities in reasoning:

[0213] Figure 12 D in KL and D wKL The values ​​are used to evaluate the inference commonality of each model on the Blocksworld and Game of 24 tasks. The GSM8K task, due to its inconsistent inference step size affecting path distribution, was not included in the evaluation. For D under random path sampling... KL The average value of GPT-4o was 0.4425, and the average value of Glm-4-plus was 0.45525. These two exhibited the most stable and widely distributed inference paths, indicating stronger inference capabilities. Subsequently, the performance of GPT-4, Qwen-ml, and Claude-S-3.5 decreased sequentially, with GPT-3.5 showing the greatest difference in average value at 0.691. For D under weighted path sampling... wKL Game of 24 had the lowest average score at 0.542, reflecting its greater efficiency in high-probability path selection. GPT-3.5 again demonstrated the weakest reasoning ability with the highest average score of 0.924. Furthermore, as... Figure 13 As shown, the KL divergence remains high in weighted path sampling, indicating that all large models have insufficient robustness and adaptability across different tasks.

[0214] d4. Reliability of multi-path reasoning:

[0215] Figure 12 P in w The reliability of multipath inference for different large models was evaluated. In the Blocksworld and Game of 24 tasks, the average P-value of GPT-4o was [value missing]. w The highest score was 0.480, followed by GPT-4 at 0.463, Glm-4-plus at 0.445, and Qwen-ml at 0.410. These results indicate that the aforementioned large models can more efficiently integrate information from multiple inference paths and select the most reliable inference path with the highest logical consistency. In contrast, GPT-3.5 and Claude-S-3.5 scored lower, with averages of 0.160 and 0.295, respectively, reflecting their weaker multi-path inference capabilities, more dispersed inference path distribution, and higher uncertainty. On the GSM8K dataset, GPT-3.5, Glm-4-9b, Llama-3-8b, and Qwen-1.5-7b achieved the highest P-value. w An average value above 0.5 indicates stronger multi-path inference reliability, resulting in more accurate results. Meanwhile, Mistral-7b's P... w The lowest value was 0.350, indicating that the model has significant difficulties in integrating information from multiple inference paths.

[0216] d5. Cumulative Hierarchical Reasoning:

[0217] Figure 12 The CTRS values ​​in the dataset assess the ability of each large model to maintain logical consistency and accuracy during multi-layered inference. In the Blocksworld and Game of 24 tasks, GPT-4 and GPT-4o achieved average scores of 0.3295 and 0.32925, respectively, demonstrating stable performance across multiple inference tasks. This indicates that their inference paths possess high overall matching quality and strong deep inference capabilities. In contrast, ClaudeS-3.5, Qwen-ml, and Glm-4-plus had lower average CTRS values ​​of 0.30725, 0.29775, and 0.25625, respectively. Their inference paths exhibited lower overall matching quality and more dispersed distribution characteristics, reflecting limitations in their robustness and optimization capabilities in complex tasks. GPT-3.5 had the lowest average CTRS at only 0.0765, highlighting its shortcomings in deep inference. On the GSM8K dataset, the average CTRS of GPT-3.5, Glm-4-9b, Llama-3-8b, Qwen-1.5-7b, and Mistral-7b, from high to low, are 0.6085, 0.600, 0.504, 0.3435, and 0.21, respectively, indicating that the deep reasoning ability of these large models in complex tasks gradually declines.

[0218] In this embodiment, a multi-dimensional evaluation is conducted:

[0219] Evaluating large models solely based on accuracy may mask potential problems in their reasoning process, while multi-dimensional evaluation can reveal the model's strengths and limitations, providing a more comprehensive basis for improvement. Figure 11 and Figure 12 The average values ​​of various indicators reveal that Qwen-ml and Qwen-1.5-7b excel in key node identification, but suffer from deficiencies in deep reasoning and multi-path integration. This results in weaker subsequent reasoning from key nodes and insufficient selection during candidate answer generation, exposing limitations in candidate generation. In contrast, GPT-4o exhibits the opposite trend. Although Glm-4-plus faces some difficulties in accumulating deep reasoning, its reasoning efficiency is high, and it has a strong dependence on cognitive data processing. Its general reasoning strategy effectively filters redundant paths, optimizes logical processing, and maintains high accuracy. Conversely, Claude-S-3.5 displays the opposite characteristics.

[0220] Improved reasoning ability in large models:

[0221] Based on the comprehensive evaluation above, the following insights are summarized regarding the improvement of large model reasoning capabilities:

[0222] Enhancing Inference Depth through External Frameworks: It is suggested that external frameworks such as Monte Carlo Tree Search (MCTS) be used for multi-step evaluation, or reinforcement learning (RL) methods be employed to extend the inference chain and improve the accuracy of self-evaluation. These methods can address the problem of insufficient inference depth in current models. Large models need to "think slowly" to handle complex inference tasks, and their performance improves as the inference chain lengthens. However, current large models struggle with key node evaluation, limiting further inference. For example, models like Claude-3.5-Sonnet tend to overestimate node scores, while Qwen-1.5-7b and Mistral-7b exhibit the opposite trend.

[0223] Mitigating the "Snowball Effect" through Constrained Decoding: It is recommended to introduce constrained decoding methods during the inference process. By imposing structural constraints in the early stages of decoding and implementing logical consistency checks at each step of inference, the model can mitigate the "snowball effect," the tendency for generated content to gradually deviate from the initial inference trajectory. Evaluation results show that the large model performs poorly in hierarchical inference, which limits its practical application value in complex scenarios.

[0224] In this embodiment, the prompt words are optimized:

[0225] The validation experiments compared the effectiveness of the two proposed suggestion optimization strategies with the classic suggestion method, specifically including:

[0226] Input-Output (IO): Question and direct output of answer;

[0227] Chain of Reasoning (CoT): The problem and the chain of reasoning used for random sampling;

[0228] PRs: Add additional rules extracted from frequent patterns based on CoT;

[0229] PA: Problems with different categories of multipaths with probability distributions;

[0230] PA exp : Additional descriptions added for path aggregation.

[0231] PRs were evaluated in the Game of 24, Blocksworld, and GSM8K tasks. Due to the higher complexity of Game of 24, five example evaluations were performed, while one example evaluation was performed each for Blocksworld and GSM8K. For PA and PA... exp It was evaluated only in the GSM8K task because the inference path lengths of the other two tasks are fixed, making path aggregation impossible.

[0232] Figure 14 The accuracy results for the three tasks are presented. Overall, the proposed PRs, PA, and PA are... exp The method significantly improves the performance of various large models across all tasks. Compared with IO and CoT, PRs achieve average improvements of 32.3% and 8.1%, respectively; PA improves accuracy by 39.3% compared to IO and 2.7% compared to CoT; while PA... exp Superior results were achieved, with a 43.8% improvement over IO and a 7.3% improvement over CoT. Among all large models, GPT-3.5 and Mistral-7b showed the most significant performance improvements after applying the optimization method of this invention. Specifically, GPT-3.5 achieved a 24.4% improvement over CoT using the PRs method on the Game of 24 task, while PA achieved a significant improvement on the GSM8K task. exp The method is 10.0% better than CoT.

[0233] Experimental results demonstrate the significant effectiveness of thought-based prompting optimization methods. Specifically, rule-based methods guide large models to learn efficient reasoning paths, avoid common errors, and thus improve the accuracy and robustness of reasoning. Path aggregation methods enable large models to integrate information, master efficient and intuitive solutions, and develop comprehensive reasoning abilities.

[0234] In summary, this invention constructs a data flow graph by dynamically monitoring variables in the domain-specific language LMCL runtime using a definition chain; extracts the logical variable LVAR from the data flow graph to generate an LVAR node set; removes indirect edges from the LVAR node set, retaining direct edges, to generate a preprocessed LVAR node set; eliminates redundant edges in the preprocessed LVAR node set using a breadth-first search and strongly connected component analysis strategy, thus constructing a mind graph; and based on the mind graph, designs a five-dimensional inference system that includes inference efficiency, key node identification capability, inference commonality, multi-path inference reliability, and cumulative hierarchical inference. A five-dimensional evaluation system is used to assess the reasoning ability of a large model and generate evaluation analysis results. Based on the evaluation analysis results, frequent patterns are mined from successful and failed thought data, and explanatory rules are extracted from these frequent patterns. A path aggregation strategy is used to integrate the probability distribution of reasoning paths. Based on the explanatory rules and the probability distribution, a prompt word optimization strategy is generated. The prompt words are optimized using the prompt word optimization strategy to generate optimized prompt words. The optimized prompt words are injected into the large model's reasoning process, and the large model performs accurate reasoning through rule guidance and path balancing strategies. This invention achieves standardized modeling of thought data by constructing a domain-specific language LMCL, solving the cross-task migration barrier caused by data heterogeneity in traditional methods. It dynamically monitors variable dependencies during program execution and automatically generates mind maps, ensuring the integrity and traceability of reasoning paths. Based on the five-dimensional evaluation system, this invention, for the first time, incorporates core capabilities such as reasoning efficiency, key node identification, and multi-path reliability into a quantitative analysis framework, breaking through the limitation of traditional evaluations that only focus on the correctness of results, and providing multi-dimensional improvement directions for model optimization. This invention mines frequent patterns and extracts semantic rules from successful and failed reasoning cases, and combines this with a path aggregation strategy to generate interpretable prompts, transforming the black-box operation of the reasoning process into transparent decision-making and improving the accuracy of model reasoning. The technical system of this invention forms a closed-loop chain of "data modeling → quantitative evaluation → strategy optimization → task enhancement," which not only accurately identifies weaknesses in model reasoning but also achieves continuous iterative improvement of reasoning capabilities through dynamic injection of optimization rules, providing a systematic solution for large-scale model optimization in complex task scenarios.

[0235] It should be noted that the method of this disclosure embodiment can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this disclosure embodiment, and the multiple devices will interact with each other to complete the method described.

[0236] It should be noted that the above description describes some embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0237] Example 2

[0238] See Figure 15 Embodiment 2 of the present invention also provides a large-scale model reasoning ability evaluation and analysis device based on thinking data, including:

[0239] The mind graph construction module 001 is used to construct a data flow graph by dynamically monitoring variables in the domain-specific language LMCL runtime using a -definition chain; extract the logical variable LVAR from the data flow graph to generate an LVAR node set; remove indirect edges from the LVAR node set and retain direct edges to generate a preprocessed LVAR node set; and eliminate redundant edges in the preprocessed LVAR node set through a breadth-first search and strongly connected component analysis strategy to construct a mind graph.

[0240] The large model reasoning ability assessment module 002 is used to design a five-dimensional assessment system based on the mind map, including reasoning efficiency, key node identification ability, reasoning commonality, multi-path reasoning reliability, and cumulative hierarchical reasoning; the reasoning ability of the large model is assessed through the five-dimensional assessment system, and assessment analysis results are generated.

[0241] The prompt word optimization module 003 is used to mine frequent patterns from the thinking data of success and failure based on the evaluation and analysis results, and extract explanatory rules from the frequent patterns; integrate the probability distribution of reasoning paths through a path aggregation strategy; generate a prompt word optimization strategy based on the explanatory rules and the probability distribution; and optimize the prompt words through the prompt word optimization strategy to generate optimized prompt words.

[0242] The large model inference optimization module 004 is used to inject the optimized prompt words into the large model inference process, and enable the large model to perform accurate inference through rule guidance and path balancing strategies.

[0243] In this embodiment, in the mind map construction module 001, during the process of constructing the data flow graph by dynamically monitoring the variables of the domain-specific language LMCL runtime using a -definition chain, the monitoring statements include: assignment statements, if / while statements, and query statements.

[0244] The expression for the data flow graph is:

[0245] G = (V, E)

[0246] In the formula, G is the data flow graph; V is the edge set; and E is the vertex set.

[0247] In this embodiment, in the mind map construction module 001, the expression of the mind map is:

[0248] G S =(S,E′)

[0249] In the formula, G S Let S be a mind map; S be the set of LVAR nodes; and E′ be the set of direct edges of S.

[0250] In this embodiment, during the evaluation of the reasoning ability of the large model using the five-dimensional evaluation system in the large model reasoning ability evaluation module 002:

[0251] Evaluation of inference efficiency for large models using frequent subgraph mining:

[0252] If max v∈P1 |C(v)|>max v∈P2 If |C(v)| or L(P1)>L(P2), then the reasoning efficiency is: E(M1)>E(M2);

[0253] In the formula, M1 and M2 are the large models; P1 and P2 are the frequent patterns mined by M1 and M2, respectively; C(v) is the set of child nodes of node v; L(P) is the inference level at which the frequent pattern P first appears.

[0254] The ability to identify key nodes in a large model is evaluated using node influence scores.

[0255] The formula for calculating the node influence score is as follows:

[0256]

[0257] In the formula, NIS is the node influence score; w1, w2, w3 are user-defined weights, satisfying w1 + w2 + w3 = 1; ρ I,S R is the Pearson correlation coefficient. local For local stability; The average score of each node;

[0258] The similarity of inference distributions among large models is evaluated using KL divergence:

[0259] The expression for the KL divergence is:

[0260]

[0261] In the formula, x is the inference path of the large model, and P(x) and Q(x) are the probability distributions of the inference paths of the two large models, respectively.

[0262] Reliability evaluation of multi-path inference for large models using path aggregation method:

[0263] The formula for calculating the path aggregation probability value is:

[0264]

[0265] In the formula, P(v n |v0,h) represents the distance from node v1 to node v n Along the path The probability of; For mind mapping G S All subpaths that follow path h; For node v i-1 The set of child nodes;

[0266] The cumulative hierarchical inference capability of large models is evaluated using cumulative hierarchical inference scores:

[0267] The expression for the cumulative hierarchical inference score is:

[0268]

[0269] In the formula, CTRS is the cumulative hierarchical reasoning score; The weights assigned to each layer i; L is the total number of layers in each inference path; N is the total number of inference paths; δ(·) is a correct node matching function; v j,i and v′ j,i These are nodes on the actual reasoning path and nodes on the reference reasoning path, respectively.

[0270] In this embodiment, the prompt word optimization module 003 performs several inferences on the same question to obtain several mind maps during the process of integrating the probability distribution of reasoning paths through the path aggregation strategy; extracts several successful reasoning paths from the several mind maps; sorts and classifies the several successful reasoning paths according to their probabilities, and selects one path and its probability from each category as an example of a prompt word.

[0271] It should be noted that the information interaction and execution process between the modules of the above system are based on the same concept as the method embodiment in Embodiment 1 of this application, and the resulting technical effects are the same as those in the method embodiment of this application. For details, please refer to the description in the method embodiment shown above in this application, and it will not be repeated here.

[0272] Example 3

[0273] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium storing program code for a method for evaluating and analyzing the reasoning ability of a large model based on thought data. The program code includes instructions for executing the method for evaluating and analyzing the reasoning ability of a large model based on thought data as described in Embodiment 1 or any possible implementation thereof.

[0274] Computer-readable storage media can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives, SSDs).

[0275] Example 4

[0276] Embodiment 4 of the present invention provides an electronic device, including: a memory and a processor;

[0277] The processor and the memory communicate with each other via a bus; the memory stores program instructions that can be executed by the processor, and the processor can execute the large model reasoning ability evaluation and analysis method based on thinking data according to Embodiment 1 or any possible implementation thereof by calling the program instructions.

[0278] Specifically, a processor can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor that reads software code stored in memory. This memory can be integrated into the processor or located outside the processor and exist independently.

[0279] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable system. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0280] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing systems. They can be centralized on a single computing system or distributed across a network of multiple computing systems. Optionally, they can be implemented using program code executable by a computing system, thereby storing them in a storage system for execution by the computing system. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0281] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A method for evaluating and analyzing reasoning ability based on large-scale models of cognitive data, characterized in that: include: A data flow graph is constructed by dynamically monitoring variables at runtime in the domain-specific language LMCL using a -definition chain. Extract the logical variable LVAR from the data flow graph to generate an LVAR node set; remove indirect edges from the LVAR node set and retain direct edges to generate a preprocessed LVAR node set; eliminate redundant edges in the preprocessed LVAR node set through breadth-first search and strong connected component analysis strategies to construct a mind graph. Based on the mind map, a five-dimensional evaluation system is designed, including reasoning efficiency, key node identification ability, reasoning commonality, multi-path reasoning reliability, and cumulative hierarchical reasoning. The reasoning ability of the large model is evaluated through the five-dimensional evaluation system, and evaluation analysis results are generated. Based on the evaluation and analysis results, frequent patterns are mined from the thinking data of success and failure, and explanatory rules are extracted from the frequent patterns; the probability distribution of reasoning paths is integrated through a path aggregation strategy. Based on the explanatory rules and the probability distribution, a prompt word optimization strategy is generated; The prompt words are optimized using the aforementioned prompt word optimization strategy to generate optimized prompt words; The optimized prompts are injected into the large model's inference process, and the large model performs accurate inference through rule guidance and path balancing strategies.

2. The method for evaluating and analyzing the reasoning ability of large models based on thinking data according to claim 1, characterized in that, In constructing the data flow graph by dynamically monitoring variables at runtime in the domain-specific language LMCL using a -definition chain, the monitoring statements include: assignment statements, if / while statements, and query statements; The expression for the data flow graph is: G = (V, E) In the formula, G is the data flow graph; V is the edge set; and E is the vertex set.

3. The method for evaluating and analyzing the reasoning ability of large models based on thinking data according to claim 2, characterized in that, The expression for the mind map is: G S =(S,E′) In the formula, G S Let S be a mind map; S be the set of LVAR nodes; and E′ be the set of direct edges of S.

4. The method for evaluating and analyzing reasoning ability based on large-scale models of thinking data according to claim 3, characterized in that, In the process of evaluating the reasoning ability of large models using the aforementioned five-dimensional evaluation system: Evaluation of inference efficiency for large models using frequent subgraph mining: If max v∈P1 |C(v)|>max v∈P2 If |C(v)| or L(P1)>L(P2), then the reasoning efficiency is: E(M1)>E(M2); In the formula, M1 and M2 are the large models; P1 and P2 are the frequent patterns mined by M1 and M2, respectively; C(v) is the set of child nodes of node v; L(P) is the inference level at which the frequent pattern P first appears. The ability to identify key nodes in a large model is evaluated using node influence scores. The formula for calculating the node influence score is as follows: In the formula, NIS is the node influence score; w1, w2, w3 are user-defined weights, satisfying w1 + w2 + w3 = 1; ρ I,S R is the Pearson correlation coefficient. local For local stability; The average score of each node; The similarity of inference distributions among large models is evaluated using KL divergence: The expression for the KL divergence is: In the formula, x is the inference path of the large model, and P(x) and Q(x) are the probability distributions of the inference paths of the two large models, respectively. Reliability evaluation of multi-path inference for large models using path aggregation method: The formula for calculating the path aggregation probability value is: In the formula, P(v n |v0,h) represents the distance from node v1 to node v n Along the path The probability of; For mind mapping G S All subpaths that follow path h; For node v i-1 The set of child nodes; The cumulative hierarchical inference capability of large models is evaluated using cumulative hierarchical inference scores: The expression for the cumulative hierarchical inference score is: In the formula, CTRS is the cumulative hierarchical reasoning score; The weights assigned to each layer i; L is the total number of layers in each inference path; N is the total number of inference paths; δ(·) is a correct node matching function; v j,i and v′ j,i These are nodes on the actual reasoning path and nodes on the reference reasoning path, respectively.

5. The method for evaluating and analyzing the reasoning ability of large models based on thinking data according to claim 4, characterized in that, In the process of integrating the probability distribution of reasoning paths through the path aggregation strategy, several reasoning operations are performed on the same question to obtain several mind maps; several successful reasoning paths are extracted from the several mind maps; the several successful reasoning paths are sorted and classified according to their probabilities, and one path and its probability are selected from each category as an example of a prompt word.

6. A device for evaluating and analyzing large-scale model reasoning ability based on thinking data, employing the method for evaluating and analyzing large-scale model reasoning ability based on thinking data as described in any one of claims 1-5, characterized in that, include: The mind mapping module is used to build data flow graphs by dynamically monitoring variables in the domain-specific language LMCL runtime using a -definition chain. Extract the logical variable LVAR from the data flow graph to generate an LVAR node set; remove indirect edges from the LVAR node set and retain direct edges to generate a preprocessed LVAR node set; eliminate redundant edges in the preprocessed LVAR node set through breadth-first search and strong connected component analysis strategies to construct a mind graph. The large model reasoning ability assessment module is used to design a five-dimensional assessment system based on the mind map, including reasoning efficiency, key node identification ability, reasoning commonality, multi-path reasoning reliability, and cumulative hierarchical reasoning; the reasoning ability of the large model is assessed through the five-dimensional assessment system, and assessment analysis results are generated. The prompt word optimization module is used to mine frequent patterns from successful and failed thought data based on the evaluation and analysis results, and extract explanatory rules from the frequent patterns; and to integrate the probability distribution of reasoning paths through a path aggregation strategy. Based on the explanatory rules and the probability distribution, a prompt word optimization strategy is generated; The prompt words are optimized using the aforementioned prompt word optimization strategy to generate optimized prompt words; The large model inference optimization module is used to inject the optimized prompts into the large model inference process, and enable the large model to perform accurate inference through rule guidance and path balancing strategies.

7. The large-scale model reasoning ability evaluation and analysis device based on thinking data according to claim 6, characterized in that, In the mind map construction module, during the process of constructing the data flow graph by dynamically monitoring the variables of the domain-specific language LMCL runtime using a -definition chain, the monitoring statements include: assignment statements, if / while statements, and query statements. The expression for the data flow graph is: G = (V, E) In the formula, G is the data flow graph; V is the edge set; and E is the vertex set.

8. The large-scale model reasoning ability evaluation and analysis device based on thinking data according to claim 7, characterized in that, In the mind map construction module, the expression of the mind map is: G S =(S,E′) In the formula, G S Let S be a mind map; S be the set of LVAR nodes; and E′ be the set of direct edges of S.

9. The large-scale model reasoning ability evaluation and analysis device based on thinking data according to claim 8, characterized in that, In the large model reasoning ability assessment module, during the process of assessing the reasoning ability of the large model through the five-dimensional assessment system: Evaluation of inference efficiency for large models using frequent subgraph mining: If max v∈P1 |C(v)|>max v∈P2 If |C(v)| or L(P1)>L(P2), then the reasoning efficiency is: E(M1)>E(M2); In the formula, M1 and M2 are the large models; P1 and P2 are the frequent patterns mined by M1 and M2, respectively; C(v) is the set of child nodes of node v; L(P) is the inference level at which the frequent pattern P first appears. The ability to identify key nodes in a large model is evaluated using node influence scores. The formula for calculating the node influence score is as follows: In the formula, NIS is the node influence score; w1, w2, w3 are user-defined weights, satisfying w1 + w2 + w3 = 1; ρ I,S R is the Pearson correlation coefficient. local For local stability; The average score of each node; The similarity of inference distributions among large models is evaluated using KL divergence: The expression for the KL divergence is: In the formula, x is the inference path of the large model, and P(x) and Q(x) are the probability distributions of the inference paths of the two large models, respectively. Reliability evaluation of multi-path inference for large models using path aggregation method: The formula for calculating the path aggregation probability value is: In the formula, P(v n |v0,h) represents the distance from node v1 to node v n Along the path The probability of; For mind mapping G S All subpaths that follow path h; For node v i-1 The set of child nodes; The cumulative hierarchical inference capability of large models is evaluated using cumulative hierarchical inference scores: The expression for the cumulative hierarchical inference score is: In the formula, CTRS is the cumulative hierarchical reasoning score; The weights assigned to each layer i; L is the total number of layers in each inference path; N is the total number of inference paths; δ(·) is a correct node matching function; v j,i and v′ j,i These are nodes on the actual reasoning path and nodes on the reference reasoning path, respectively.

10. The large-scale model reasoning ability evaluation and analysis device based on thinking data according to claim 9, characterized in that, In the prompt word optimization module, during the process of integrating the probability distribution of reasoning paths through the path aggregation strategy, several reasoning operations are performed on the same question to obtain several mind maps; several successful reasoning paths are extracted from the several mind maps; the several successful reasoning paths are sorted and classified according to their probabilities, and one path and its probability are selected from each category as an example of a prompt word.

Images