Machine Learning-Based Decision-Making Ability Assessment Method and Apparatus

Abstract

This invention provides a machine learning-based method and apparatus for evaluating decision-making ability. The method includes: acquiring language data of target participants and converting the language data into text data; extracting candidate keywords and target sentences from the text data, extracting nodes from the target sentences to obtain target word pairs; constructing a participant influence graph using the candidate keywords as nodes and the target word pairs as directed edges; mapping the participant influence graph to a pre-constructed reference answer influence graph; and generating a decision-making ability evaluation result based on the hit ratio of nodes in the participant influence graph to their corresponding nodes in the reference answer influence graph, and the hit ratio of directed edges in the participant influence graph to their corresponding directed edges in the reference answer influence graph. This solves the problems of existing decision-making ability evaluations being highly subjective, lacking quantitative evaluation, and unable to be applied to multi-objective decision-making scenarios.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and apparatus for evaluating decision-making ability based on machine learning. Background Technology

[0002] In multi-objective decision-making scenarios, participants, within a limited time, form a comprehensive judgment on the problem based on publicly available information and background knowledge, demonstrating their decision-making ability. The assessment of participants' decision-making ability is generated through a decision-making ability assessment system. However, existing technologies for such systems heavily rely on the subjective experience and judgment of experts, neglecting objective and systematic evaluation of participants' decision-making abilities. Specifically, these systems often lack the support of decision analysis theory or apply only a single framework (such as the Analytic Hierarchy Process), the effectiveness of which is not well-proven. The assessment dimensions included in such systems are often unsystematic and fragmented, lacking comprehensiveness and systematicity. For example, indicators such as emotional stability, language organization ability, and the number of options proposed are determined solely by the evaluator's subjectivity, and these indicators may not effectively measure decision-making ability. Furthermore, the assessment dimensions included in such systems may be difficult to quantify (such as comprehension, discernment, reasoning, and expression abilities).

[0003] In general, existing decision-making capability assessment systems rely on expert opinions to provide overall evaluations, which introduces significant subjectivity, and the evaluation indicators lack quantitative assessment. In particular, past assessment systems are difficult to apply to multi-value (including conflicting objectives), multi-information (including uncertain information), multi-option, multi-stakeholder, and logic-based scientific decision-making problems.

[0004] In view of this, the present invention provides a decision-making ability assessment method and apparatus based on machine learning to solve the problems of high subjectivity, lack of quantitative evaluation, and inability to be applied to multi-objective decision-making scenarios in the prior art. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a decision-making ability assessment method and apparatus based on machine learning, which solves the problems that the decision-making ability assessment in the prior art is highly subjective, lacks quantitative evaluation, and cannot be applied to multi-objective decision-making scenarios, thereby improving the accuracy and objectivity of decision-making ability assessment.

[0006] This invention provides a machine learning-based method for evaluating decision-making ability, the method comprising:

[0007] Obtain the language data of the target participants and convert the language data into text data;

[0008] Candidate keywords and target sentences are extracted from the text data, and node extraction is performed on the target sentences to obtain target word pairs;

[0009] Using the candidate keywords as nodes and the target word pairs as directed edges, construct a participant influence graph;

[0010] The participant influence graph is mapped to a pre-constructed reference answer influence graph. Based on the hit ratio of the nodes in the participant influence graph to the corresponding nodes in the reference answer influence graph, and the hit ratio of the directed edges in the participant influence graph to the corresponding directed edges in the reference answer influence graph, a decision-making ability assessment result is generated.

[0011] The reference answer influence graph is constructed by using natural language processing methods to perform preliminary information extraction and clustering, preliminary directed relation extraction from the target domain knowledge base, and verifying and evaluating the importance of nodes and directed edges through interaction with experts.

[0012] In some embodiments, the process of constructing the reference answer influence graph includes:

[0013] Based on the target domain knowledge base, a domain knowledge graph is generated using natural language processing methods; based on the domain knowledge graph and a manually supplemented domain keyword dictionary, nodes for reference answers are extracted.

[0014] Based on the target domain knowledge base, target sentences with causal meaning or conditions are extracted, and node-based pre-training matching is performed on the target sentences to obtain directed edges of the reference answer;

[0015] Construct the influence graph of the reference answer using the nodes and directed edges of the reference answer.

[0016] In some embodiments, an influence graph of the reference answer is constructed using the nodes and directed edges of the reference answer, and the method further includes:

[0017] Based on the local Markov property, the joint probability distribution represented by the influence graph of the reference answer is decomposed into several marginal probability distributions represented by nodes and conditional probability distributions represented by directed edges to obtain multiple subgraphs.

[0018] Based on the root cause nodes obtained from the subgraph decomposition, determine the important nodes in each subgraph; based on the conditional dependencies between mutually independent nodes obtained from the subgraph decomposition, determine the important node relationships in each subgraph.

[0019] In some embodiments, the nodes of the influence graph of the reference answer include at least:

[0020] Opportunity nodes, which are used to characterize each uncertainty to be modeled;

[0021] Decision nodes, which are used to represent each decision to be made;

[0022] Utility nodes are used to characterize each component corresponding to an additivity-separable utility function.

[0023] In some embodiments, the target word pair includes directed word pairs and undirected word pairs, and the target word pair is used as a directed edge, specifically including:

[0024] If the target word pair is a directed word pair, the target word pair is treated as a directed edge in the influence graph of the participants;

[0025] If the target word pair is an undirected word pair, after annotating the directional relationship of the target word pair, it is added as a directed edge in the influence graph of the participants.

[0026] In some embodiments, generating decision-making capability assessment results further includes:

[0027] The results of the decision-making ability assessment will be output and visualized.

[0028] The present invention also provides a machine learning-based decision-making ability assessment device, the device comprising:

[0029] A data acquisition unit is used to acquire the language data of the target participants and convert the language data into text data.

[0030] The data extraction unit is used to extract candidate keywords and target sentences from the text data, and to extract nodes from the target sentences to obtain target word pairs.

[0031] The influence graph construction unit is used to construct a participant influence graph by using the candidate keywords as nodes and the target word pairs as directed edges.

[0032] The result generation unit is used to map the participant influence graph to a pre-constructed reference answer influence graph, and generate decision-making ability assessment results based on the hit ratio of the nodes in the participant influence graph to the corresponding nodes in the reference answer influence graph, and the hit ratio of the directed edges in the participant influence graph to the corresponding directed edges in the reference answer influence graph.

[0033] The reference answer influence graph is constructed by using natural language processing methods to perform preliminary information extraction and clustering, preliminary directed relation extraction from the target domain knowledge base, and verifying and evaluating the importance of nodes and directed edges through interaction with experts.

[0034] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the machine learning-based decision-making ability assessment method as described above.

[0035] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the machine learning-based decision-making ability assessment method as described above.

[0036] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the machine learning-based decision-making ability assessment method as described above.

[0037] The present invention provides a machine learning-based decision-making ability assessment method and apparatus, which acquires language data of target participants and converts the language data into text data; extracts candidate keywords and target sentences from the text data, extracts nodes from the target sentences to obtain target word pairs; constructs a participant influence graph using the candidate keywords as nodes and the target word pairs as directed edges; maps the participant influence graph to a pre-constructed reference answer influence graph, and generates a decision-making ability assessment result based on the hit ratio of nodes in the participant influence graph to their corresponding nodes in the reference answer influence graph, and the hit ratio of directed edges in the participant influence graph to their corresponding directed edges in the reference answer influence graph; wherein, the reference answer influence graph is constructed by using natural language processing methods to perform preliminary information extraction and clustering, preliminary directed relation extraction from a target domain knowledge base, and verifying and evaluating the importance of nodes and directed edges through interaction with experts, resulting in an influence graph containing the importance of nodes and directed edges.

[0038] This method and apparatus are based on machine learning and decision analysis algorithms. By combining machine learning models with expert judgment, an influence map of the reference answer is constructed. The machine learning model processes the participants' statements or conversational texts to generate a participant influence map of the target participants. Based on decision analysis, the participants' decision-making ability is comprehensively evaluated, including value-based decision-making ability (hitting key utility nodes and decision nodes in the two influence maps) and information utilization ability (qualitative and quantitative hits on key opportunity nodes and directed relationships in the two influence maps). This achieves a quantitative evaluation of decision-making ability, thereby solving the problems of high subjectivity, lack of quantitative evaluation, and inability to be applied to multi-objective decision-making scenarios in existing technologies. It improves the accuracy and objectivity of decision-making ability evaluation. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0040] Figure 1 This is one of the flowcharts illustrating the machine learning-based decision-making ability assessment method provided by the present invention.

[0041] Figure 2 The second flowchart illustrates the machine learning-based decision-making ability assessment method provided by this invention.

[0042] Figure 3 A structural block diagram of the machine learning-based decision-making ability assessment device provided by the present invention;

[0043] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0045] The following is combined with Figures 1-4 This invention introduces a machine learning-based decision-making ability assessment method and apparatus.

[0046] In one specific embodiment, the present invention provides a decision-making ability assessment method based on machine learning, such as... Figure 1 As shown, the method includes the following steps:

[0047] S110: Obtain the language data of the target participant and convert the language data into text data; in the specific implementation process, the method relies on a decision-making ability assessment system, which can be loaded on a smart terminal. The system includes a text analysis module. In the text analysis module of statements or conversations, the text of statements or conversations obtained through shorthand or speech transcription is used as input and then converted into text data.

[0048] S120: Extract candidate keywords and target sentences from the text data, and extract nodes from the target sentences to obtain target word pairs.

[0049] S130: Using the candidate keywords as nodes and the target word pairs as directed edges, construct a participant influence graph; specifically, extract nodes from the text using natural language processing methods (e.g., pre-trained models) and fuzzy matching based on reference answers, extract directed edges from the text using natural language processing methods and rule-based extraction, and construct a participant influence graph based on nodes and directed edges.

[0050] In a specific use case, the text analysis process for statements or conversations involves converting the video or audio of the statement or conversation into text data using on-site shorthand and speech transcription. The original text undergoes preprocessing, including proofreading and removal of interjections and pauses. During node extraction, a mapping is constructed between text keywords and reference answer nodes. Based on the text keywords, fuzzy matching is first performed on the segmented text to determine candidate keywords. Then, a pre-trained model (SimBERT) is used to identify keywords most similar in meaning to the text to be matched. Mapping these keywords yields nodes in the reference answer. The nodes obtained through this method represent a subset of nodes in the reference answer influence graph; the hit rate reflects the participant's understanding of domain knowledge.

[0051] During directed edge extraction, sentences with causal meaning are identified using three methods (template-based extraction, classification based on pre-trained models, and manual verification and supplementation); sentences with conditional meaning are identified through template matching and manual judgment. Node extraction is performed on these two types of sentences to obtain directed (template-based and manual) or undirected (pre-trained model-based) word pairs. Directed word pairs directly become directed edges in the participant's influence graph, while undirected word pairs are added as directed edges after manual judgment of their directional relationships. The proportion of directed edges in the participant's influence graph that overlap with the reference answer reflects the participant's understanding of the underlying mechanisms of the domain.

[0052] S140: Map the participant influence graph to the pre-constructed reference answer influence graph, and generate a decision-making ability assessment result based on the hit ratio of the nodes in the participant influence graph to the corresponding nodes in the reference answer influence graph, and the hit ratio of the directed edges in the participant influence graph to the corresponding directed edges in the reference answer influence graph.

[0053] In the capability assessment module based on the decision analysis framework, this invention links value-based decision-making ideas with influence diagrams, assesses value-based decision-making capabilities by hitting key utility nodes and decision nodes, assesses the ability to capture key information by hitting opportunity nodes with observation incentives, and assesses the ability to recognize uncertainty by hitting root cause nodes and conditional dependencies.

[0054] In principle, a decision analysis framework refers to the six elements of decision-making (framework, value, information, options, logic, and commitment). The decision analysis framework includes six elements: framework (stakeholders, decision-making atmosphere, decision-making cycle, etc.); value (goals and trade-offs); information (key factors, uncertainties); options (alternative solutions); logic (scientific analytical methods); and commitment (commitment to action). In the influence diagram, opportunity nodes correspond to the grasp of information (information) in the decision analysis framework, decision nodes correspond to the consideration of alternative solutions (options), utility nodes correspond to the evaluation of value (value), and directed edges correspond to the analytical logic (logic) in the decision analysis framework.

[0055] Specifically, value-based decision-making involves first identifying key objectives (utility nodes) of concern, and then using these objectives to guide the organizational decision-making process (decision nodes). Objectives can be divided into two categories: fundamental objectives and instrumental objectives. Fundamental objectives refer to the basic reasons for focusing on the decision problem (judged by experts), and the identification of fundamental objectives reflects the participants' value judgment ability. Decision nodes that directly influence fundamental objectives are the nodes d-connected to the fundamental objectives in the influence diagram; the identification of such nodes reflects the participants' grasp of the importance of the decision.

[0056] The reference answer influence graph is constructed by using natural language processing methods to perform preliminary information extraction and clustering, preliminary directed relation extraction from the target domain knowledge base, and verifying and evaluating the importance of nodes and directed edges through interaction with experts.

[0057] To comprehensively, objectively, and quantitatively assess the decision-making ability of multi-objective decision-makers, this invention conducts decision analysis based on influence graphs. Three types of nodes (opportunity nodes, decision nodes, and utility nodes) in the influence graph represent the information involved in the reference answer and statements or conversational texts, and directed edges in the influence graph represent the directed relationships between information. The decision-making ability of participants is measured by the coverage of nodes and directed edges.

[0058] The influence graph of the reference answer is constructed in a semi-automatic manner. Specifically, based on domain knowledge, natural language processing methods are used to perform preliminary information extraction and clustering from the domain knowledge base, as well as preliminary directed relation extraction. Through interaction with experts, the nodes and directed edges are verified and their importance is evaluated, thus constructing an influence graph of the reference answer that includes the importance of nodes and directed edges.

[0059] In some embodiments, such as Figure 2 As shown, the process of constructing the influence diagram of the reference answer includes the following steps:

[0060] S210: Based on the target domain knowledge base, generate a domain knowledge graph using natural language processing methods; based on the domain knowledge graph and a manually supplemented domain keyword dictionary, extract nodes for the reference answer.

[0061] In a specific use case, for the influence graph of the reference answer, during node construction, authoritative guiding texts are selected as the domain knowledge base (taking the public health field as an example, industry guidelines, industry standards, and modeling articles published in authoritative journals such as *Nature*, *Science*, and *Proceedings of the National Academy of Sciences* are selected). This is combined with word segmentation (jieba) and syntactic analysis (spacy) methods to generate a domain knowledge graph. Based on the knowledge graph and a manually supplemented domain keyword dictionary, nodes are integrated and clustered according to their frequency of occurrence, similarity, and human judgment. Confidence is determined using indicators such as frequency, uncertainty, reputation, and score, and experts are given importance classifications (which can be divided into core, important, and other levels). The uncertainty level of nodes is determined based on literature or human judgment. Quantitative characterization of node distribution is achieved through differential equation modeling (based on literature or human judgment) or by converting qualitative descriptions input by experts into quantitative methods using heuristics.

[0062] S220: Based on the target domain knowledge base, extract target sentences with causal meaning or conditions, perform node-based pre-training matching on the target sentences, and obtain directed edges of the reference answer.

[0063] In a specific use case, for the influence graph of the reference answer, during the construction of directed edges, sentences with causal meaning are identified from the domain knowledge base (industry guidelines, industry standards, and authoritative literature) using three methods (template-based extraction, pre-trained model-based classification, and manual verification and supplementation); sentences with conditional relational meaning are identified through template matching and manual judgment. In the template-based extraction stage (keywords, sentence structure, etc.), existing templates are improved to capture explicit directed relational sentence pairs; in the pre-trained model-based classification stage (BERT, etc.), sentences are divided into two categories: those with potential causal relationships and those without, thus capturing implicit causal sentences; in the manual verification and supplementation stage, the directed relational sentences obtained from the first two stages are verified, and supplementation is made based on human knowledge.

[0064] Node-based pre-training (SimBERT) matching is performed on the two types of sentences mentioned above, and the resulting node pairs are directed edges. The automatically generated directed edges are then heuristically graded for importance (strength) (e.g., directed edges between core nodes are defined as high (core) importance, and directed edges between core nodes and important nodes are defined as medium (important) importance). The strength of directed relationships is then quantitatively characterized through differential equation modeling (based on literature or human judgment) or by converting qualitative descriptions input by experts into quantitative methods using heuristics.

[0065] Based on domain knowledge, we use natural language processing methods to perform preliminary information extraction and clustering, and preliminary directed relation extraction. Through interaction with experts, we verify and evaluate the importance of nodes and directed edges, and construct an influence graph of the reference answer that includes the importance of nodes and directed edges.

[0066] S230: Construct the influence graph of the reference answer using the nodes and directed edges of the reference answer.

[0067] Influence graphs can be viewed as a special type of Bayesian network, therefore, Bayesian network-based analysis can also be applied to influence graphs. In both types of influence graphs, the three types of nodes are considered variables, and the two types of relationships are considered conditional dependencies between variables. Under the causal Markov condition (defined as: for each node in a causal graph, given its parent node, the node's condition is independent of all other non-descendant nodes), directed edges in a causal Bayesian network can be understood as direct causes pointing to direct consequences. Specifically, head nodes pointing to opportunity and utility nodes are direct causes of tail nodes, and head nodes pointing to decision nodes provide information for decision-making, i.e., influencing the range of values ​​for the decision. This understanding aligns with directed edge extraction and expert supplementation based on explicit / implicit causal relationships. Due to the local Markov property of Bayesian networks (defined as: given a node's parent node, that node is conditionally independent of all its non-descendant nodes), directed edges in a Bayesian network can be understood as direct dependencies; in other words, given a set of nodes that directly point to a certain node, if the distribution of this set of nodes is known, the distribution of other nodes will not directly affect the distribution of that node. This understanding aligns with directed edge extraction based on foretalk (defined as a narrative relevance form with typical connectors) templates and expert supplementation. Due to the local Markov property of Bayesian networks, the joint probability distribution represented by a Bayesian network can be decomposed into marginal probability distributions represented by several nodes and conditional probability distributions represented by directed edges, with each decomposition being independent. Participants' capture of the marginal and conditional distributions in the reference answer's Bayesian network characterizes their perception of uncertainty regarding root cause nodes and relationships between nodes.

[0068] Based on the properties of the Bayesian network described above, an influence graph of the reference answer is constructed using the nodes and directed edges of the reference answer. This process then includes:

[0069] Based on the local Markov property, the joint probability distribution represented by the influence graph of the reference answer is decomposed into several marginal probability distributions represented by nodes and conditional probability distributions represented by directed edges to obtain multiple subgraphs.

[0070] Based on the root cause nodes obtained from the subgraph decomposition, determine the important nodes in each subgraph; based on the conditional dependencies between mutually independent nodes obtained from the subgraph decomposition, determine the important node relationships in each subgraph.

[0071] Specifically, during subgraph decomposition, after obtaining the reference answer influence graph through the above three steps, the joint probability distribution represented by the reference answer influence graph can be decomposed into several marginal probability distributions represented by nodes and conditional probability distributions represented by directed edges, based on the local Markov property of the influence graph / Bayesian network. Each decomposition is independent. The root cause nodes and relationships between nodes obtained from the decomposition contain all the quantitative uncertainty information in the reference answer influence graph.

[0072] In some embodiments, the nodes of the influence graph of the reference answer include at least:

[0073] Opportunity nodes, which are used to characterize each uncertainty to be modeled;

[0074] Decision nodes, which are used to represent each decision to be made;

[0075] Utility nodes are used to characterize each component corresponding to an additivity-separable utility function.

[0076] Specifically, this invention constructs two types of observer-perspective influence diagrams: the reference answer influence diagram and the participant influence diagram. The reference answer influence diagram contains three types of nodes: opportunity nodes, decision nodes, and utility nodes, which are considered as variables with known or unknown distributions. Opportunity nodes correspond to each uncertainty to be modeled, decision nodes correspond to each decision to be made, and utility nodes correspond to each component of the additivity-separable utility function. The three types of nodes in the reference answer influence diagram are manually defined, representing key information within the domain.

[0077] In some embodiments, the target word pair includes directed word pairs and undirected word pairs, and the target word pair is used as a directed edge, specifically including:

[0078] If the target word pair is a directed word pair, the target word pair is treated as a directed edge in the influence graph of the participants;

[0079] If the target word pair is an undirected word pair, after annotating the directional relationship of the target word pair, it is added as a directed edge in the influence graph of the participants.

[0080] Directed edges (conditional dependencies) represent directed relationships between key information within the domain; the three types of nodes on the influence graph of participants represent the participants' hits on key information within the domain, and directed edges represent the participants' hits on key directed relationships within the domain. Specifically, the directed relationships in the influence graph are divided into two categories: causal relationships and correlational relationships. Causal relationships are assumed to be determined in advance through randomized experiments (structural invariance), and both types of relationships can be represented by conditional probabilities.

[0081] When determining key nodes, expert judgment identifies important and core utility nodes, which reflect the core values ​​within the domain. Based on expert judgment, graph criteria are used to identify opportunity nodes with observable (interventional) incentives that influence important and core utilities. The information (control) value of these opportunity nodes is calculated based on the conditional probability of directed edges. These nodes reflect high information (control) value elements within the domain, i.e., information (control) that provides utility for decision-making and its quantitative impact on utility. Combining value-based decision-making principles, important decision nodes are identified, reflecting highly important decisions within the domain.

[0082] The root cause nodes are obtained from the subgraph decomposition. Understanding the uncertainty of the root cause nodes is a necessary condition for understanding the global uncertainty.

[0083] When determining the importance of relationships in a subgraph, the importance of directed relationships is judged based on heuristic importance ranking and literature and expert knowledge. The conditional dependencies between nodes are obtained from the subgraph decomposition; understanding the uncertainty and independence of these relationships is a necessary condition for recognizing global uncertainty.

[0084] Directed relations can be divided into two categories: causal relations and conditional relations. In causal directed relations, the pointing relation represents a direct cause; in conditional directed relations, the pointing relation represents a direct influence, that is, given a set of nodes that directly point to a certain node, if the distribution of this set of nodes is known, the distribution of other nodes will not directly affect the distribution of that node.

[0085] For explicit directed relationships, this invention employs a template-based extraction method. This invention utilizes and refines two types of extraction templates: causal templates and predictive statement templates. The causal templates cover nine categories of causal markers and sentence structures, while the predictive statement templates cover seven categories of predictive statement structures (single consequence, common consequence, alternative consequence, single cause, common cause, alternative cause, and reverse cause / optional statement). Based on these two types of templates, this invention directly captures explicit directed relationships through keyword matching.

[0086] For implicit directed relations, this invention employs a pre-trained model-based extraction method. Natural language text paragraphs are used as input, and the output is whether implicit causal / intervention semantics are present, sentence by sentence. An implicit causal representation recognition model is established by fine-tuning a pre-trained neural network. The pre-trained model BERT is used as the backbone model for causal representation extraction, with an additional feedforward network layer as a classifier. This classification model is fine-tuned and trained on a dataset containing implicit causal representations, and the model parameters are saved after convergence. In application, the natural text is first segmented into sentences and tokenized. Then, the sentences are input into the implicit causal representation recognition model, which outputs the causal classification result sentence by sentence (1 indicates the presence of implicit causal semantics, 0 indicates no presence).

[0087] In a (causal) influence graph, if the observed outcome of an opportunity node helps improve the expected utility of a decision, then that node has an observation incentive or positive information value. If an opportunity node is the object of an agent's intervention, then that node has an intervention incentive or positive control value. Observation incentives and intervention incentives are determined using graph criteria. The graph criterion for determining observation incentives is: given a decision and known distribution of observations, opportunity nodes connected to utility nodes d- have observation incentives. The graph criterion for determining intervention incentives is: after removing nodes without observation incentives and their directed edges to the decision, opportunity nodes connected to utility nodes have intervention incentives. Participants' choice of opportunity nodes with observation incentives reflects their consideration of information that provides utility.

[0088] For an influence graph with multiple utility nodes and decision nodes and a complex structure (not satisfying the sufficient recall condition), a heuristic approach is used to construct a local influence graph for each utility-decision node pair. In this local influence graph, there is only one utility node and one decision node; all other nodes are opportunity nodes. Using the graph criterion of observed incentives, a set of opportunity nodes connected to a given decision node and a utility node d- is searched. This set, after removing the actual decision nodes, represents the set of opportunity nodes with positive observed incentives. During this process, a dictionary data structure records the highest importance of the utility node with positive observed incentives for each opportunity node, representing the weight of the observed incentive for that opportunity node, used for subsequent evaluation.

[0089] A Bayesian network is a graph representation of a joint probability distribution. The probability distribution of nodes (variables) and the strength of edges (directed relationships) are the basic inputs for Bayesian networks in decision analysis. In the scenario of this invention, participants rarely provide direct quantitative statements. This invention employs two methods to estimate the probability distribution of nodes and the quantitative strength of edges: differential equation modeling of variables and directed relationships based on literature or human judgment; and discretized quantitative estimation based on qualitative strength (high, medium, and low levels) input by experts.

[0090] The causal strength of a set of edges to a node can be expressed as the distance between the distribution of the node with and without the set of edges. The distance between the two distributions is represented by the KL divergence.

[0091] Furthermore, the decision-making ability assessment results are generated, which then include:

[0092] The results of the decision-making ability assessment will be output and visualized.

[0093] In an influence graph, the information value of an opportunity node is defined as the difference between the expected utility of the influence graph containing that opportunity node (and its connected directed edges) and the expected utility of the influence graph not containing that opportunity node (and its connected directed edges). The complete information value of an opportunity node is defined as the difference between the expected utility of having complete information about that opportunity node and the expected utility of not having complete information about that opportunity node. In an influence graph, the control value of an opportunity node is defined as the expected utility improvement that optimally controlling that (unknown distribution) opportunity node would bring to the decision-maker.

[0094] In a specific use case, capability assessment based on a decision analysis framework can be statistically analyzed using the following five points to obtain key indicator data.

[0095] First, utility node coverage.

[0096] Based on different levels of importance, the number of utility nodes hit and their proportion of all utility nodes are counted as a measure of utility node coverage.

[0097] Secondly, the decision-making plan takes into account all aspects.

[0098] Based on different levels of importance, the number of decision nodes that are hit and their proportion of all decision nodes are statistically analyzed to measure the comprehensiveness of the decision-making scheme.

[0099] Third, the comprehensiveness of information consideration

[0100] Based on different levels of importance and their associated observation incentive weights, the number of hit opportunity nodes and their proportion of all opportunity nodes with positive observation incentives are statistically analyzed as a measure of the comprehensiveness of information consideration in decision analysis.

[0101] Fourth, the depth of consideration of the uncertainty of root cause nodes.

[0102] Based on importance, the number of key uncertain nodes hit is statistically graded as a measure of the depth of uncertainty consideration for nodes with no dependencies in the analysis.

[0103] Fifth, the depth of consideration regarding the uncertainty of conditional relationships.

[0104] Based on importance, the number of key uncertain relationships hit is statistically graded as a measure of the depth of uncertainty consideration regarding the dependencies between key nodes in the analysis. Sixth, the completeness of value-based decision-making considerations.

[0105] For each core utility node, identify decision nodes that have a direct impact on it (i.e., those d-connected to it in the influence graph) and opportunity nodes that have informational value (observational incentives), and together form a subgraph about that utility node. Statistically count the number of core utility node subgraphs covered (hitting any node in the subgraph is considered hitting the subgraph) and their proportion of all core utility node subgraphs as a measure of the completeness of value-based decision-making considerations.

[0106] During the visualization process, Python scripts and pre-written report templates (corresponding to the aforementioned indicators) can be used to output scores for each item and generate radar charts. Word embeddings (Wikipedia Chinese word vectors) are performed on the three types of nodes. Principal component analysis is used to transform the high-dimensional embedding vectors into two-dimensional vectors, which are then visualized in two-dimensional space. Node coverage is visualized based on node type, node importance, and whether a node is covered; the coverage of the core utility node subgraph is visualized based on the node's core utility node subgraph and whether the node is covered. Markdown files are output, and reports are batch-output in PDF format using Pandoc.

[0107] In the above specific embodiments, the machine learning-based decision-making ability assessment method provided by the present invention acquires the language data of the target participants and converts the language data into text data; extracts candidate keywords and target sentences from the text data, extracts nodes from the target sentences to obtain target word pairs; constructs a participant influence graph by using the candidate keywords as nodes and the target word pairs as directed edges; maps the participant influence graph to a pre-constructed reference answer influence graph, and generates a decision-making ability assessment result based on the hit ratio of the nodes in the participant influence graph to the corresponding nodes in the reference answer influence graph, and the hit ratio of the directed edges in the participant influence graph to the corresponding directed edges in the reference answer influence graph; wherein, the reference answer influence graph is constructed by using natural language processing methods to perform preliminary information extraction and clustering, preliminary directed relation extraction from the target domain knowledge base, and verifying and evaluating the importance of nodes and directed edges through interaction with experts, resulting in an influence graph containing the importance of nodes and directed edges.

[0108] This method, based on machine learning and decision analysis algorithms, constructs an influence map of reference answers by combining machine learning models with expert judgment. It uses machine learning models to process participants' statements or conversational texts to generate participant influence maps for target participants. Based on decision analysis, it comprehensively evaluates participants' decision-making abilities, including value-based decision-making abilities (hitting key utility nodes and decision nodes in the two influence maps) and information utilization abilities (qualitative and quantitative hits on key opportunity nodes and directed relationships in the two influence maps). This achieves a quantitative evaluation of decision-making abilities, thus solving the problems of high subjectivity, lack of quantitative evaluation, and inability to be applied to multi-objective decision-making scenarios in existing technologies, and improving the accuracy and objectivity of decision-making ability evaluation.

[0109] In addition to the methods described above, this invention also provides a decision-making ability assessment device based on machine learning, such as... Figure 3 As shown, the device includes:

[0110] Data acquisition unit 310 is used to acquire the language data of the target participant and convert the language data into text data;

[0111] Data extraction unit 320 is used to extract candidate keywords and target sentences from the text data, and to extract nodes from the target sentences to obtain target word pairs;

[0112] The influence graph construction unit 330 is used to construct a participant influence graph by using the candidate keywords as nodes and the target word pairs as directed edges.

[0113] The result generation unit 340 is used to map the participant influence graph to a pre-constructed reference answer influence graph, and generate a decision-making ability assessment result based on the hit ratio of the nodes in the participant influence graph to the corresponding nodes in the reference answer influence graph, and the hit ratio of the directed edges in the participant influence graph to the corresponding directed edges in the reference answer influence graph.

[0114] The reference answer influence graph is constructed by using natural language processing methods to perform preliminary information extraction and clustering, preliminary directed relation extraction from the target domain knowledge base, and verifying and evaluating the importance of nodes and directed edges through interaction with experts.

[0115] In some embodiments, the process of constructing the reference answer influence graph includes:

[0116] Based on the target domain knowledge base, a domain knowledge graph is generated using natural language processing methods; based on the domain knowledge graph and a manually supplemented domain keyword dictionary, nodes for reference answers are extracted.

[0117] Based on the target domain knowledge base, target sentences with causal meaning or conditions are extracted, and node-based pre-training matching is performed on the target sentences to obtain directed edges of the reference answer;

[0118] Construct the influence graph of the reference answer using the nodes and directed edges of the reference answer.

[0119] In some embodiments, an influence graph of the reference answer is constructed using the nodes and directed edges of the reference answer, and the method further includes:

[0120] Based on the local Markov property, the joint probability distribution represented by the influence graph of the reference answer is decomposed into several marginal probability distributions represented by nodes and conditional probability distributions represented by directed edges to obtain multiple subgraphs.

[0121] Based on the root cause nodes obtained from the subgraph decomposition, determine the important nodes in each subgraph; based on the conditional dependencies between mutually independent nodes obtained from the subgraph decomposition, determine the important node relationships in each subgraph.

[0122] In some embodiments, the nodes of the influence graph of the reference answer include at least:

[0123] Opportunity nodes, which are used to characterize each uncertainty to be modeled;

[0124] Decision nodes, which are used to represent each decision to be made;

[0125] Utility nodes are used to characterize each component corresponding to an additivity-separable utility function.

[0126] In some embodiments, the target word pair includes directed word pairs and undirected word pairs, and the target word pair is used as a directed edge, specifically including:

[0127] If the target word pair is a directed word pair, the target word pair is treated as a directed edge in the influence graph of the participants;

[0128] If the target word pair is an undirected word pair, after annotating the directional relationship of the target word pair, it is added as a directed edge in the influence graph of the participants.

[0129] In some embodiments, generating decision-making capability assessment results further includes:

[0130] The results of the decision-making ability assessment will be output and visualized.

[0131] In the above specific embodiments, the machine learning-based decision-making ability assessment device provided by the present invention acquires the language data of the target participants and converts the language data into text data; extracts candidate keywords and target sentences from the text data, extracts nodes from the target sentences to obtain target word pairs; constructs a participant influence graph by using the candidate keywords as nodes and the target word pairs as directed edges; maps the participant influence graph to a pre-constructed reference answer influence graph, and generates a decision-making ability assessment result based on the hit ratio of the nodes in the participant influence graph to the corresponding nodes in the reference answer influence graph, and the hit ratio of the directed edges in the participant influence graph to the corresponding directed edges in the reference answer influence graph; wherein, the reference answer influence graph is constructed by using natural language processing methods to perform preliminary information extraction and clustering, preliminary directed relation extraction from the target domain knowledge base, and verifying and evaluating the importance of nodes and directed edges through interaction with experts, resulting in an influence graph containing the importance of nodes and directed edges.

[0132] This device, based on machine learning and decision analysis algorithms, constructs an influence map of reference answers by combining machine learning models with expert judgments. It uses machine learning models to process participants' statements or conversational texts to generate participant influence maps for target participants. Based on decision analysis, it comprehensively evaluates participants' decision-making abilities, including value-based decision-making abilities (hitting key utility nodes and decision nodes in the two influence maps) and information utilization abilities (qualitative and quantitative hits on key opportunity nodes and directed relationships in the two influence maps). This achieves quantitative evaluation of decision-making abilities, thus solving the problems of high subjectivity, lack of quantitative evaluation, and inability to be applied to multi-objective decision-making scenarios in existing technologies, and improving the accuracy and objectivity of decision-making ability evaluation.

[0133] Figure 4An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute the above-described method.

[0134] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0135] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is able to perform the methods described above.

[0136] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the methods described above.

[0137] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0138] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A machine learning-based method for evaluating decision-making ability, characterized in that, The method includes: Obtain the language data of the target participants and convert the language data into text data; Candidate keywords and target sentences are extracted from the text data, and node extraction is performed on the target sentences to obtain target word pairs; Using the candidate keywords as nodes and the target word pairs as directed edges, construct a participant influence graph; The participant influence graph is mapped to a pre-constructed reference answer influence graph. Based on the hit ratio of the nodes in the participant influence graph to the corresponding nodes in the reference answer influence graph, and the hit ratio of the directed edges in the participant influence graph to the corresponding directed edges in the reference answer influence graph, a decision-making ability assessment result is generated. The reference answer influence graph is constructed by using natural language processing methods to perform preliminary information extraction and clustering, preliminary directed relation extraction from the target domain knowledge base, and verifying and evaluating the importance of nodes and directed edges through interaction with experts. The process of constructing the influence diagram of the reference answer includes: Based on the target domain knowledge base, a domain knowledge graph is generated using natural language processing methods; based on the domain knowledge graph and a manually supplemented domain keyword dictionary, nodes for reference answers are extracted. Based on the target domain knowledge base, target sentences with causal meaning or conditions are extracted, and node-based pre-training matching is performed on the target sentences to obtain directed edges of the reference answer; Construct the influence graph of the reference answer using the nodes and directed edges of the reference answer.

2. The machine learning-based decision-making ability assessment method according to claim 1, characterized in that, Using the nodes and directed edges of the reference answer, an influence graph of the reference answer is constructed, which then includes: Based on the local Markov property, the joint probability distribution represented by the influence graph of the reference answer is decomposed into several marginal probability distributions represented by nodes and conditional probability distributions represented by directed edges to obtain multiple subgraphs. Based on the root cause nodes obtained from the subgraph decomposition, determine the important nodes in each subgraph; based on the conditional dependencies between mutually independent nodes obtained from the subgraph decomposition, determine the important node relationships in each subgraph.

3. The decision-making ability assessment method based on machine learning according to claim 1, characterized in that, The nodes in the influence diagram of the reference answer include at least: Opportunity nodes, which are used to characterize each uncertainty to be modeled; Decision nodes, which are used to represent each decision to be made; Utility nodes are used to characterize each component corresponding to an additivity-separable utility function.

4. The machine learning-based decision-making ability assessment method according to claim 1, characterized in that, The target word pairs include directed word pairs and undirected word pairs. The target word pairs are used as directed edges, specifically including: If the target word pair is a directed word pair, the target word pair is treated as a directed edge in the influence graph of the participants; If the target word pair is an undirected word pair, after annotating the directional relationship of the target word pair, it is added as a directed edge in the influence graph of the participants.

5. The machine learning-based decision-making ability assessment method according to claim 1, characterized in that, The process of generating decision-making capability assessment results will then include: The decision-making ability assessment results will be output and visualized.

6. A decision-making ability assessment device based on machine learning, characterized in that, The device includes: A data acquisition unit is used to acquire the language data of the target participants and convert the language data into text data. The data extraction unit is used to extract candidate keywords and target sentences from the text data, and to extract nodes from the target sentences to obtain target word pairs. The influence graph construction unit is used to construct a participant influence graph by using the candidate keywords as nodes and the target word pairs as directed edges. The result generation unit is used to map the participant influence graph to a pre-constructed reference answer influence graph, and generate decision-making ability assessment results based on the hit ratio of the nodes in the participant influence graph to the corresponding nodes in the reference answer influence graph, and the hit ratio of the directed edges in the participant influence graph to the corresponding directed edges in the reference answer influence graph. The reference answer influence graph is constructed by using natural language processing methods to perform preliminary information extraction and clustering, preliminary directed relation extraction from the target domain knowledge base, and verifying and evaluating the importance of nodes and directed edges through interaction with experts. The process of constructing the influence diagram of the reference answer includes: Based on the target domain knowledge base, a domain knowledge graph is generated using natural language processing methods; based on the domain knowledge graph and a manually supplemented domain keyword dictionary, nodes for reference answers are extracted. Based on the target domain knowledge base, target sentences with causal meaning or conditions are extracted, and node-based pre-training matching is performed on the target sentences to obtain directed edges of the reference answer; Construct the influence graph of the reference answer using the nodes and directed edges of the reference answer.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the machine learning-based decision-making ability assessment method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the machine learning-based decision-making ability assessment method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the machine learning-based decision-making ability assessment method as described in any one of claims 1 to 5.

Images