A method and device for constructing an evaluation scenario set for an intelligent decision-making system based on a case-constrained large model

Abstract

The present invention relates to the field of electronic digital data processing, and in particular to a method and device for constructing an evaluation scenario set of an intelligent decision-making system based on a case-constrained large model. The method comprises the following steps: constructing a scenario tree framework based on domain knowledge and extracting key elements; traversing the scenario tree framework, calculating the complexity of subnodes and node weights; assembling prompt words, and generating a scenario description through a large model; determining the matching degree of a scenario template and a scenario description through a scenario template and scenario description matching measurement model; aggregating matching scenario templates and scenario descriptions to generate a scenario tree; deduplicating and calculating the balance degree of the scene tree, and outputting an evaluation scenario set if the scene tree is already balanced. The present invention effectively suppresses the illusion of a large model, automatically generates a high-quality evaluation scenario set, and realizes the precise matching measurement of the scenario template and the scene description, and also effectively solves the problem that it is difficult to achieve balanced and full coverage of the evaluation scenario.

Description

Technical Field

[0001] The present invention relates to the field of electronic digital data processing, and in particular to a method and device for constructing an evaluation scenario set of an intelligent decision-making system based on a case constraint large model. Background Art

[0002] In recent years, the evaluation task of intelligent decision-making systems requires the use of massive, high-quality, and comprehensive evaluation scenario data, so the construction of evaluation scenario sets has become crucial. An evaluation scenario refers to a comprehensive expression consisting of a series of interrelated events, activities, and situations in a specific time and space, which is used to describe a complete and internally logical situation. The evaluation scenario set construction technology for intelligent decision-making systems is to design, simulate, and generate a series of evaluation scenarios to evaluate the functions, performance, and reliability of intelligent decision-making systems in different situations.

[0003] At present, common methods in this field mainly include rule-based scenario generation, simulation-based scenario generation, expert knowledge-based construction, etc. With the rapid development of general large model technology, methods that use general large model technology to assist in generating evaluation scenarios have also emerged, and the quality of evaluation scenario data generation has been greatly improved.

[0004] However, the existing technology still has the following problems: First, the task of constructing an evaluation scene dataset faces challenges such as high implementation cost, easy omissions, and difficulty in collecting abnormal / scarce scene data. Secondly, the general large model technology still has the inherent deficiencies of content illusion and semantic drift that are difficult to overcome, resulting in low credibility of the generated scene data and insufficient for evaluation purposes. Thirdly, the existing methods lack the overall structural design and accurate measurement of the balance of the evaluation scene library, which can easily lead to an imbalance of scene elements. Finally, there is a lack of effective means to measure the semantic matching of structured scene templates assembled from scene elements and unstructured scene descriptions.

[0005] For example, Chinese patent application CN116050136A discloses a method for generating a test scenario library for an intelligent ship, which first obtains a test task, then determines a test scenario element hierarchical model based on the test task, generates static and dynamic test scenarios, and finally obtains a target test scenario library based on complexity and overlap. However, the invention only divides the constructed scenario library into dynamic and static hierarchies, and lacks an accurate measure for evaluating the overall structural design and balance of the scenario library.

[0006] For another example, Chinese patent application CN117667699A discloses a method for generating test scenarios for intelligent connected vehicles based on knowledge graphs. This method ensures comprehensive coverage and diversity of scenario elements by constructing a knowledge graph of ICV scenarios. However, the scenarios generated and defined by the invention through the domain knowledge of the knowledge graph only use scenario elements to assemble the structured scenario definition, and have not yet finally obtained an unstructured scenario description.

[0007] For another example, Chinese patent application CN118586281A discloses a virtual simulation test method for intelligent vehicles based on a large language model, which uses a large model to generate evaluation scene data. However, the invention does not consider the problem of suppressing content hallucinations and semantic drift, so the generated evaluation scene data is not credible. Summary of the invention

[0008] In order to reduce the workload of existing scenario construction methods and improve the credibility of generated data, so as to obtain a high-quality, factor-balanced, and fully covered evaluation scenario set, the present invention provides a method and device for constructing an evaluation scenario set for an intelligent decision-making system based on a case constraint large model.

[0009] The present invention provides a method for constructing an evaluation scenario set of an intelligent decision-making system based on a case constraint large model, which adopts the following technical solutions:

[0010] A method for constructing an evaluation scenario set of an intelligent decision-making system based on a case constraint large model comprises the following steps:

[0011] Build a scene tree framework based on domain knowledge, and extract key elements based on the knowledge graph;

[0012] Traverse the scene tree framework and calculate the complexity and node weight of child nodes;

[0013] Prompt word assembly, generating scene descriptions through large models;

[0014] The scene template and scene description matching measurement model is used to determine the matching degree between the scene template and the scene description. If the matching is not determined, it is used as a negative example for prompt word assembly.

[0015] Aggregate matching scene templates and scene descriptions to generate a scene tree;

[0016] The scene tree is deduplicated and balanced. If the scene tree is balanced, the evaluation scene set is output; otherwise, the scene template and scene description are aggregated, and the scene tree is deduplicated and balanced.

[0017] In a specific implementation scheme, the method for constructing a scene tree framework based on domain knowledge includes the following steps:

[0018] Build an empty scene tree framework, and add the initial scene as the root node to the scene tree framework;

[0019] Extract the key elements in the initial scene, assign values ​​to the key elements based on the knowledge graph, and generate child nodes;

[0020] Take the child node as the current node, extract the key elements of the current node, assign values ​​to the key elements, generate child nodes after the current node, and loop until all possible scenes are added to the scene tree framework;

[0021] Remove redundant nodes and invalid paths in the scene tree framework.

[0022] In a specific implementation scheme, the method for calculating the complexity and node weight of a subnode includes the following steps:

[0023] Calculate the complexity score of each key element of the child node and sum them up, which is the complexity of the child node;

[0024] Get the depth and branching factor of the child node, and calculate the position score of the child node based on the depth and branching factor;

[0025] The complexity and position scores of the child nodes are weighted and summed to obtain the node weight of the child node.

[0026] In a specific implementation scheme, the cue word assembly adopts the CRISPE framework.

[0027] In a specific possible implementation scheme, the method for determining the matching degree between the scene template and the scene description by the scene template and scene description matching measurement model comprises the following steps:

[0028] Convert the input <scene template, scene description> dataset into a high-dimensional sparse vector representation;

[0029] Convert high-dimensional sparse vectors to low-dimensional dense vectors;

[0030] The dense vector is transformed through a nonlinear activation function. Each layer maps the input dense vector to a new representation space and gradually extracts higher-level features.

[0031] Capture the deep semantic information of the input text, and the output is the final semantic feature representation after multi-layer nonlinear mapping;

[0032] The matching degree between the scene template semantic feature vector and the scene description semantic feature vector is calculated by KL divergence algorithm;

[0033] The matching degree is converted into a posterior probability, and further calculated to obtain the matching degree score;

[0034] The matching score is compared with the matching threshold. If the matching score ≥ the matching threshold, it indicates that the scene template and the scene description match; otherwise, it indicates that the scene template and the scene description do not match.

[0035] In a specific feasible implementation scheme, when the scene template and the scene description do not match, the scene template and the scene description are used as negative examples for prompt word assembly and become penalty examples for the large model.

[0036] In a specific implementation scheme, the method for calculating the balance of a scene tree includes the following steps:

[0037] Perform key element balance statistics on the current scene tree, including single-element statistics, two-element statistics, and three-element statistics;

[0038] Based on the calculation results of single-factor statistics, two-factor statistics, and three-factor statistics, the standard deviation is further calculated, and the standard deviation is used as the balance degree to be compared with the balance degree threshold;

[0039] If the balance degree is greater than or equal to the balance degree threshold, it indicates that the current scene tree is unbalanced; otherwise, it indicates that the current scene tree is balanced.

[0040] In a specific possible implementation scheme, before the scene template and scene description matching measurement model determines the matching degree between the scene template and the scene description, the scene template and scene description matching measurement model is trained by using a training scene set, and the method for generating the training scene set includes the following steps:

[0041] Extract features from case scenario descriptions and analyze semantic information and contextual relationships;

[0042] Match the extracted features with entities and relations in the knowledge graph to identify key elements in the case scenario description;

[0043] Combine key elements according to predefined templates to generate structured scene templates;

[0044] Create a training scenario set of <case scenario template, case scenario description>.

[0045] In a specific possible implementation, a breadth-first search method is used when traversing the scene tree framework.

[0046] The present invention also provides a device for constructing an evaluation scenario set of an intelligent decision-making system based on a case constraint large model, which adopts the following technical solution:

[0047] A device for constructing an evaluation scenario set of an intelligent decision-making system based on a case constraint large model, comprising one or more processors; a memory;

[0048] And one or more computer programs, wherein the one or more computer programs are stored in a memory, and the one or more computer programs include instructions, which, when executed by a processor, enable the device for constructing an evaluation scenario set of an intelligent decision system based on a case constraint big model to execute the above-mentioned method for constructing an evaluation scenario set of an intelligent decision system based on a case constraint big model.

[0049] In summary, the present invention has the following beneficial effects:

[0050] 1. When the scene template and scene description matching measurement model are used to determine the matching degree, the scene templates and scene descriptions that are determined to be mismatched are used as negative examples of the CRISPE framework, so that when the large model subsequently generates scene descriptions, it can avoid repeated generation of erroneous scene descriptions, effectively suppress the large model hallucination, and improve the accuracy of the generated scenes.

[0051] 2. After the scene tree is generated, the balance of key elements of the scene tree as a whole is calculated to effectively solve the problem of imbalance of scene elements caused by simply piling up evaluation data.

[0052] 3. In the matching layer of the DSSM model, the KL divergence algorithm is used instead of the original method of using cosine distance to calculate the vector, which improves the accuracy of the matching measurement between the scene template and the scene description.

[0053] 4. Through optimization methods such as scene deduplication and node weight re-evaluation, combined with the calculation and iterative optimization process of the balance of the evaluation scene tree, it ensures that the structure distribution of the final constructed evaluation scene set is reasonable, minimizing the influence of subjective factors. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a flowchart of the method for constructing an evaluation scenario set for an intelligent decision-making system based on a case-constrained large model.

[0055] Figure 2 It is a block diagram of the method for constructing an evaluation scenario set for an intelligent decision-making system based on a case-constrained large model. DETAILED DESCRIPTION

[0056] To facilitate understanding, some nouns that appear in the following text are explained first.

[0057] Scenario refers to a comprehensive expression of a series of interrelated events, activities and situations within a specific time and space. It includes elements such as physical environment, participants, interaction methods and background information, and is used to describe a complete situation with internal logic.

[0058] Scenario Elements refer to the basic components that make up a scene. Elements work together to form a complete situation with internal logic, usually including the physical environment, participants, interaction methods and background information.

[0059] Scenario relationships refer to the connection and interaction between different scenarios, describing how one scenario transitions to another, as well as the logical and causal relationships between scenarios. These relationships can be temporal sequence, causal relationship, conditional triggering, etc.

[0060] Scenario Tree refers to a tree structure formed by organizing multiple interrelated scenarios according to certain associations during the operation of the target object. Each node represents a specific scenario, and the edges between scenario nodes represent the associations between scenarios. Scenario Tree is used to describe and analyze the state and behavior changes of the target object under different conditions.

[0061] The following combination Figure 1-Figure 2 The present invention is described in further detail.

[0062] Reference Figure 1 ,The method for constructing evaluation scenario sets of the intelligent decision system based on ,case constraint large model includes the following steps:

[0063] S100, builds a scene tree framework based on domain knowledge.

[0064] Specifically, the steps include:

[0065] S110, construct an empty scene tree framework, and add the initial scene as a root node to the scene tree framework.

[0066] S120, analyzing the initial scene and extracting key elements from the initial scene. Based on the knowledge graph, further decompose and associate the key elements, assign values ​​to the key elements, and define a new scene. According to the scene association relationship of the new scene, add the new scene as a child node of the scene tree framework under the current node.

[0067] S130, when adding a child node, check whether the conditions of the association relationship are met, and only the scenes that meet the conditions are added as child nodes of the current node to the scene tree framework. In other words, the scene as a child node is a sub-scene of the current scene, and the sub-scene is an extension of the current scene.

[0068] S140, for each newly added child node, recursively execute steps S120 and S130 until all possible scenes are added to the scene tree framework.

[0069] Specifically, the degree of coverage of all possible permutations and combinations of key elements by the sub-nodes is judged. If the sub-nodes in the scene tree framework can completely cover all possible permutations and combinations of key elements, it indicates that all possible scenes have been added to the scene tree framework; otherwise, steps S120 and S130 are looped.

[0070] S150, optimize the generated scene tree framework to remove redundant nodes and invalid paths. The scene tree framework obtained after optimization is a semi-structured scene template. .

[0071] To facilitate understanding, the following is a further explanation using the example of user loan applications in financial risk control.

[0072] Root node: "User submits loan application", that is, the initial scenario is "User submits loan application". The key elements extracted are: user information and loan policy. User information can be complete or incomplete, and loan policies can be either compliant or non-compliant. At this point, four different scenarios can be generated: user information is complete and compliant with loan policies, user information is complete but not compliant with loan policies, user information is incomplete but compliant with loan policies, and user information is incomplete and not compliant with loan policies. These four scenarios can be used as child nodes of the root node to generate a scenario tree framework. By further extracting and analyzing the key elements of these four scenarios, more scenarios can be further generated, thereby generating more child nodes after the child nodes. Specifically as follows:

[0073] Subnode 1 (condition: user information is complete and meets loan policy requirements): "Loan approval passed"

[0074] Subnode 1.1 (condition: user repays on time): "Loan settled"

[0075] Subnode 1.2 (Condition: User overdue repayment): "Overdue collection"

[0076] Subnode 1.2.1 (Condition: User pays off overdue payments): "Resume normal repayments"

[0077] Subnode 1.2.2 (Condition: User is continuously overdue): "Loan default processing"

[0078] Subnode 2 (condition: user information is incomplete but meets loan policy requirements): "Loan approval rejected"

[0079] Sub-node 2.1 (condition: user completes the information and reapplies): "Re-enter the loan approval process"

[0080] Subnode 2.2 (Condition: User abandons loan application): "Application completed"

[0081] Subnode 3 (Condition: User information is complete but does not meet loan policy requirements): "Loan approval rejected"

[0082] Subnode 3.1 (Condition: User adjusts loan policy): "Re-enter the loan approval process"

[0083] Subnode 3.2 (Condition: User abandons loan application): "Application completed"

[0084] Subnode 4 (Condition: User information is incomplete and does not comply with loan policy): "Loan approval rejected".

[0085] Subnode 4.1 (condition: user adjusts loan strategy and completes information): "Re-enter the loan approval process"

[0086] Subnode 4.2 (Condition: User abandons loan application): “Application completed”.

[0087] It is easy to understand that different scenarios, as sub-scenarios of the current scenario, need to meet the conditions of the current scenario. For example, the "resumption of normal repayment" scenario needs to meet the condition that the user is overdue for repayment before it can serve as a sub-node of the "overdue collection" node.

[0088] S200, traverse the scene tree framework, traverse each sub-node in turn, read the key elements of each sub-node, and then use them for subsequent prompt word element assembly. At the same time, during the scene tree framework traversal process, the weight of the sub-node is calculated according to the complexity and position of the sub-node.

[0089] Specifically, the steps include:

[0090] S210, traversing the scene tree framework based on a breadth-first search method.

[0091] After the nodes in the scene tree framework are combined with the scene description, a complete scene tree is formed. Each node in the scene tree represents a specific evaluation scene, and the generated set is the evaluation scene set. The rhythm of traversing the scene tree framework determines the rhythm of building the evaluation scene set, and also has a significant impact on the comprehensiveness and coverage of the constructed evaluation scene set. Therefore, breadth-first search (BFS) is used.

[0092] Breadth-first search is a graph traversal algorithm that starts from the root node, first visits the root node, then visits all its child nodes in turn, then visits the child nodes of these child nodes, and so on, until all nodes are visited, ensuring that every child node in the scene tree frame can be traversed. Breadth-first search traversal of the scene tree frame includes the following steps:

[0093] Create a queue, add the root node to the queue, and mark the root node as visited;

[0094] Get a node from the queue as the current node, visit all unvisited child nodes of the current node, and analyze the logic of the current node, such as recording paths, analyzing scenarios, etc. Add all unvisited child nodes of the current node to the queue and mark them as visited;

[0095] Repeat the above steps until the queue is empty and the traversal ends.

[0096] S220, calculating the complexity and node weight of the child node.

[0097] The complexity of a subnode can be quantified by the number and complexity of key elements. Assuming that each key element (such as the physical environment, participants, interaction methods, and background information) has a complexity score, these scores can be summed up to get the complexity of the subnode. The specific formula is as follows:

[0098]

[0099] In the formula, Representative The complexity of the child nodes;

[0100] Representative The child node The complexity score of each key element;

[0101] Represents the number of key elements of the child node.

[0102] The position of a child node in the scene tree framework can be quantified by the depth of the node and the branching factor of the node to obtain a position score. The depth of a node indicates the level of the node in the tree. The depth of the root node is 0, the depth of the child node of the root node is 1, the depth of the child node of the current node with a depth of 1 is 2, and so on. The branching factor indicates the number of child nodes starting from the current node. The calculation formula for the position score is as follows:

[0103]

[0104] In the above formula, Representative Position score of each child node;

[0105] Indicates The depth of child nodes;

[0106] Representative The branching factor of the child nodes;

[0107] and is a weight used to adjust the impact of depth and branching factor on the location score.

[0108] By taking a weighted sum of the complexity and position scores of the child nodes, the node weight of the child node can be calculated. The specific calculation formula is as follows:

[0109]

[0110] In the above formula, It is The weight of the child nodes, and is a weight coefficient, which is used to measure the influence of complexity and position score on node weight. It is easy to understand that there is no order between step S210 and step S220, and the step number is only used to distinguish. Further, if step S220 is performed before step S210, different scenarios can be better evaluated and processed during the traversal and analysis process by calculating the weight of each child node.

[0111] S300, assembling prompt word elements.

[0112] The CRISPE framework can be used. The CRISPE framework is a tool for generating structured prompt words, which is designed to help users systematically construct complex scene descriptions. The CRISPE framework fully takes into account the capabilities of subsequent large language models: context understanding, role-playing, instruction following, topic focusing, default behavior setting, and exception handling. By combining these capabilities, high-quality prompt words can be generated in accordance with the CRISPE framework, thereby guiding subsequent large models to produce more accurate and relevant outputs.

[0113] The CRISPE framework has the following elements:

[0114] Context: Background information of the scene, including time, place, environment, etc.

[0115] Roles: The main characters involved in the scene and their characteristics.

[0116] Interactions: The interactions and behaviors between characters.

[0117] Scenarios: Specific events or activities that describe the dynamic changes in a scene.

[0118] Purpose: The goal or intention of the scene, explaining why the scene was created.

[0119] Elements: Specific components of a scene, such as the physical environment, participants, interaction methods, and background information.

[0120] Exception: Special circumstances or limitations of scene generation, such as various negative examples.

[0121] By adjusting the CRISPE framework, the elements in the CRISPE framework and the scenario template The key elements in the scene template correspond to All key elements in the CRISPE framework, such as evaluation object, evaluation intention, evaluation scenario elements, related scenarios, negative examples, and scenario generation instructions, are all incorporated into the CRISPE framework to obtain a complete scenario description. The prompt word.

[0122] S400, large model parameter configuration and call, generate scene description .

[0123] The large model generates a scenario description based on the written parameter configuration scheme, as well as specific parameters such as maximum generation length, temperature, top-level sampling, frequency penalty, existence penalty, stop sequence, batch size, and log probability. , thus forming a scene template , scene description >dataset.

[0124] S500, extract case scenario descriptions from the scenario case library, generate corresponding structured case scenario templates through a knowledge graph-based information extraction method, and then form a training scenario set.

[0125] Specifically, the steps include:

[0126] S510, the case scenario description is stored in a scenario case library in the form of text. The case scenario description is extracted from the scenario case library, and pre-processing operations such as word segmentation, part-of-speech tagging, and named entity recognition are performed on the case scenario description.

[0127] S520, extract features from the preprocessed case scenario description through a deep learning model, and analyze semantic information and contextual relationships. The deep learning model may be LSTM, Transformer, etc.

[0128] S530, matching the features extracted by the deep learning model with the entities and relationships in the knowledge graph to identify key elements in the case scenario description.

[0129] S540, combining key elements according to a predefined template to generate a structured scene template.

[0130] By combining deep learning models and knowledge graphs, it is possible to accurately extract features and key elements from text, such as physical environment, participants, interaction methods, and background information, and automatically generate structured scenario templates to facilitate better understanding and processing of case scenario descriptions in the future.

[0131] Based on the generated case scenario template and the case scenario description in the scenario case library, you can create a <Case Scenario Template , case scenario description >The training scene set serves as the training samples for the subsequent scene template and scene description matching measurement model.

[0132] S600: Perform model training on the scene template and scene description matching measurement model according to the training scene set. and scene description Matching determination.

[0133] The scene template and scene description matching measurement model is based on the deep structured semantic model DSSM (Deep Structured Semantic Models). The classic DSSM model is also called the dual-tower semantic matching measurement model, which aims to improve the performance of search engines by learning the semantic relevance between queries and documents. DSSM can capture deeper semantic information through deep learning technology, thereby helping search engines find documents that best match query keywords and improve the relevance of search results.

[0134] However, the traditional DSSM model can only perform scene template and scene description It is difficult to consider the information differences in different directions when judging the matching degree, which makes it difficult to meet the asymmetric scenario. In addition, in the matching degree judgment process, only the similarity judgment can be made based on whether the key elements exist, and it is difficult to accurately determine whether the key elements are correct in the context and word order, resulting in low accuracy of the judgment result. Whether to strictly follow the semi-structured scenario template The traditional DSSM model is difficult to meet the above requirements. Therefore, in the matching measurement layer of the DSSM model, the traditional cosine distance calculation is replaced by the Kullback-Leibler (KL) divergence algorithm. The KL divergence algorithm is used for more comprehensive analysis and more sophisticated matching measurement to perform scene template matching. and scene description The specific algorithm is as follows:

[0135] Term Vector Layer: Transform the input training scenario set < case scenario template , case scenario description >Converted to a high-dimensional sparse vector representation. The dimension of each vector usually corresponds to the case scenario template and case scenario description The key elements in the vector represent the occurrence frequency and weight of the key elements.

[0136] Word Embedding: Convert high-dimensional sparse vectors into low-dimensional dense vectors. Through pre-trained word embedding (such as Word2Vec or GloVe), each word is represented as a fixed-dimensional vector that captures the semantic relationship between words.

[0137] Multi-layer nonlinear projection: It consists of multiple fully connected layers (each layer has weights and biases) and is transformed by nonlinear activation functions (such as tanh). Each layer maps the input vector to a new representation space and gradually extracts higher-level features. The calculation formula is as follows.

[0138]

[0139] In the above formula, represents the input vector;

[0140] Indicates The output of the hidden layer is ;

[0141] Indicates Layer and Parameter matrix between layers;

[0142] Indicates A bias term.

[0143] Semantic feature layer: The output is the final semantic feature representation after multiple layers of nonlinear mapping. It captures the deep semantic information of the input text and is usually a vector of fixed dimension.

[0144] Relevance measured by KL: Calculates the degree of match between two semantic feature vectors. The calculation formula is as follows:

[0145]

[0146]

[0147] In the above formula, represents the KL divergence, It represents the result after KL divergence is standardized to the interval [0,1]. Presentation case scenario template and case scenario description The matching degree, The larger the case scenario template and case scenario description The higher the match.

[0148] Posterior probability computed by softmax: Convert the matching degree into posterior probability. Through the softmax function, the matching degree value is normalized into a probability distribution to represent the case scenario description. Belong to the case scenario description The final output is a case scenario description Belong to the case scenario description The posterior probability is calculated as follows.

[0149]

[0150] In the above formula, is a smoothing factor of a softmax function, Represents a set of candidate documents generated by the large model according to different parameter groups. express Furthermore, during the training phase, the loss function is minimized by maximum likelihood estimation, and the calculation formula is as follows:

[0151]

[0152] In the above formula, Represents the parameters of the neural network. The model is optimized through stochastic gradient descent (SGD) to obtain the parameters of each network layer. Finally, further calculation can obtain the case scenario template and case scenario description According to the preset matching threshold, if the matching score ≥ matching threshold, it means that the case scenario template and case scenario description Match; otherwise, it indicates that the case scenario template and case scenario description No match.

[0153] After the scene template and scene description matching measurement model completes the model training through the training scene set, the scene template , scene description >The dataset inputs the scene template and the scene description matching measurement model to perform scene template and scene description The method for determining the matching degree between the scene template and the scene description matching measurement model is the same as the method for processing the training scene set in the above text, and will not be repeated here.

[0154] If the scene template and scene description If the match is determined, the subsequent process will be carried out; if the scene template and scene description If it is determined that there is no match, the scene template The following scene description The negative example database of the CRISPE framework in step S300 is stored so that the case will no longer appear when the CRISPE framework generates prompt words. When The following scene description To penalize the sample, the large model will not appear in the scene template. Regenerate the scene description .

[0155] By adding a scene template and scene description Mismatched cases are used as negative examples in the CRISPE framework to optimize the generation of prompt words, making them more accurate and effectively avoiding the problems of semantic drift and content hallucination, thereby improving the accuracy of scene description generation, improving the accuracy and credibility of the generated scenes, and ultimately improving the accuracy of the evaluation results.

[0156] S700, describe the scene Aggregate to scene template In the process, a scene tree is generated and further analyzed and optimized.

[0157] Specifically, the steps include:

[0158] S710: Obtaining a scene description determined to be matched and scene templates , describe the scene Add it to the scene tree framework, use SimHash to remove data duplication, and re-evaluate the weight of the child nodes, and then describe the remaining scenes Add it to the corresponding child node of the scene tree frame to form a complete scene tree.

[0159] By deduplicating data, we can ensure that each scene in the scene tree is unique and avoid duplicate scenes, which can effectively reduce redundant data and improve evaluation efficiency and performance.

[0160] S720, calculating the balance of the scene tree.

[0161] Perform key element balance statistics on the current scene tree, including single element statistics, two element statistics, and three element statistics. When the key element balance statistics are less than the balance threshold, it means that the scene tree construction has achieved the expected goal, and the node traversal of the scene tree can be stopped without adding new evaluation scene descriptions.

[0162] Furthermore, single-element statistics refers to counting the frequency of each scene element in all scenes. The calculation formula is as follows:

[0163]

[0164] in, Indicates The weighted frequency of each key factor;

[0165] It is The weight of the child nodes

[0166] is the total number of scenes;

[0167] Indicates scenes;

[0168] is the indicator function, when Appear in In the middle of the day, ,otherwise, .

[0169] Two-factor statistics refers to the frequency of each pair of key factors appearing simultaneously in all scenarios. The calculation formula is as follows:

[0170]

[0171] In the above formula, Indicates The key elements and The weighted frequency of the simultaneous occurrence of the key elements.

[0172] Three-factor statistics refers to the frequency of each group of three key factors appearing simultaneously in all scenarios. The calculation formula is as follows:

[0173]

[0174] In the above formula, Indicates The key elements, The key elements and The weighted frequency of the simultaneous occurrence of the key elements.

[0175] By comparing the distribution of key elements in different scenes, the balance of the scene tree can be evaluated. Generally speaking, the statistical laws of the appearance of different key elements in all scenes satisfy the corresponding statistical distribution, indicating that the scene tree is well balanced. Scene tree balance means that it is hoped that all single elements appear in every scene of the scene tree, two elements appear in every scene as much as possible, and three elements appear in most scenes. Therefore, the standard deviation of single-element statistics, two-element statistics, and three-element statistics can be further calculated, and the balance of the distribution of key elements can be measured by the standard deviation. It can be understood that if a single element appears in every scene of the scene tree, it indicates that there is no fluctuation in the data, and the standard deviation is 0 at this time. In other words, the smaller the standard deviation, the higher the balance of the scene tree.

[0176] The standard deviation calculation formulas for single-factor statistics, two-factor statistics, and three-factor statistics are as follows:

[0177]

[0178]

[0179]

[0180] In the above formula, Represents the standard deviation of a single factor distribution;

[0181] Represents the standard deviation of the two-factor distribution;

[0182] represents the standard deviation of the three-factor distribution;

[0183] is the average value of the occurrence of a single element;

[0184] is the average value of the two elements;

[0185] is the average value of the three elements.

[0186] Different balance thresholds are set for single-factor statistics, two-factor statistics, and three-factor statistics. , and When it is greater than or equal to the balance threshold, it indicates that the scene tree is poorly balanced, and the process returns to step S710 to aggregate the scene description and the scene tree framework to generate more scenes; otherwise, it indicates that the scene tree has reached balance, and the evaluation scene set is output.

[0187] It should be noted that the generated scene can meet the scene tree balance requirement, that is, the scene generation can be stopped even if there are still child nodes with unconverged scene descriptions in the scene tree framework.

[0188] It is easy to understand that there is no order between steps S100 and S500, and the serial numbers are only used for distinction.

[0189] The present invention also discloses a device for constructing an evaluation scenario set for an intelligent decision system based on a case constraint big model, comprising one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the processor, enable the device for constructing an evaluation scenario set for an intelligent decision system based on a case constraint big model to execute the above-mentioned method for constructing an evaluation scenario set for an intelligent decision system based on a case constraint big model.

[0190] The above are all preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for constructing an evaluation scenario set for an intelligent decision-making system based on a case-constrained large model, characterized in that: The steps include: Build a scene tree framework based on domain knowledge, and extract key elements based on the knowledge graph; Traverse the scene tree framework and calculate the complexity and node weight of child nodes; Prompt word assembly, generating scene descriptions through large models; The scene template and scene description matching measurement model is used to determine the matching degree between the scene template and the scene description. If the matching is not determined, it is used as a negative example for prompt word assembly. Aggregate matching scene templates and scene descriptions to generate a scene tree; De-duplicate and balance the scene tree. If the scene tree is balanced, output the evaluation scene set. Otherwise, the aggregation of scene templates and scene descriptions, as well as scene tree deduplication and balance calculation are performed cyclically; The method for determining the matching degree between a scene template and a scene description by a scene template and scene description matching measurement model comprises the following steps: Convert the input <scene template, scene description> dataset into a high-dimensional sparse vector representation; Convert high-dimensional sparse vectors to low-dimensional dense vectors; The dense vector is transformed through a nonlinear activation function. Each layer maps the input dense vector to a new representation space and gradually extracts higher-level features. Capture the deep semantic information of the input text, and the output is the final semantic feature representation after multi-layer nonlinear mapping; The matching degree between the scene template semantic feature vector and the scene description semantic feature vector is calculated by KL divergence algorithm; The matching degree is converted into a posterior probability, and further calculated to obtain the matching degree score; The matching score is compared with the matching threshold. If the matching score ≥ the matching threshold, it indicates that the scene template and the scene description match; otherwise, it indicates that the scene template and the scene description do not match.

2. The method for constructing an evaluation scenario set for an intelligent decision-making system based on a case constraint large model according to claim 1 is characterized in that: The method of constructing a scene tree framework based on domain knowledge includes the following steps: Build an empty scene tree framework, and add the initial scene as the root node to the scene tree framework; Extract the key elements in the initial scene, assign values ​​to the key elements based on the knowledge graph, and generate child nodes; Take the child node as the current node, extract the key elements of the current node, assign values ​​to the key elements, generate child nodes after the current node, and loop until all possible scenes are added to the scene tree framework; Remove redundant nodes and invalid paths in the scene tree framework.

3. The method for constructing an evaluation scenario set for an intelligent decision-making system based on a case constraint large model according to claim 1 is characterized in that: The method for calculating the complexity and node weight of a child node includes the following steps: Calculate the complexity score of each key element of the child node and sum them up, which is the complexity of the child node; Get the depth and branching factor of the child node, and calculate the position score of the child node based on the depth and branching factor; The complexity and position scores of the child nodes are weighted and summed to obtain the node weight of the child node.

4. The method for constructing an evaluation scenario set for an intelligent decision-making system based on a case constraint large model according to claim 1 is characterized in that: The CRISPE framework was used for cue word assembly.

5. The method for constructing an evaluation scenario set for an intelligent decision-making system based on a case constraint large model according to claim 1 is characterized in that: When the scene template and the scene description do not match, the scene template and the scene description are used as negative examples for prompt word assembly and become penalty examples for the large model.

6. The method for constructing an evaluation scenario set for an intelligent decision-making system based on a case constraint large model according to claim 1 is characterized in that: The method for calculating the balance of a scene tree includes the following steps: Perform key element balance statistics on the current scene tree, including single-element statistics, two-element statistics, and three-element statistics; Based on the calculation results of single-factor statistics, two-factor statistics, and three-factor statistics, the standard deviation is further calculated, and the standard deviation is used as the balance degree to be compared with the balance degree threshold; If the balance degree is greater than or equal to the balance degree threshold, it indicates that the current scene tree is unbalanced; Otherwise it indicates that the current scene tree is balanced.

7. The method for constructing an evaluation scenario set for an intelligent decision-making system based on a case constraint large model according to claim 1 is characterized in that: Before the scene template and scene description matching measurement model determines the matching degree between the scene template and the scene description, the scene template and scene description matching measurement model is trained by using a training scene set. The method for generating the training scene set includes the following steps: Extract features from case scenario descriptions and analyze semantic information and contextual relationships; Match the extracted features with entities and relations in the knowledge graph to identify key elements in the case scenario description; Combine key elements according to predefined templates to generate structured scene templates; Create a training scenario set of <case scenario template, case scenario description>.

8. The method for constructing an evaluation scenario set for an intelligent decision-making system based on a case constraint large model according to claim 1 is characterized in that: When traversing the scene tree frame, a breadth-first search method is used.

9. A device for constructing an evaluation scenario set for an intelligent decision-making system based on a case constraint large model, characterized in that: comprising one or more processors; Memory; And one or more computer programs, wherein the one or more computer programs are stored in a memory, and the one or more computer programs include instructions, which, when executed by a processor, enable the device for constructing an evaluation scenario set of an intelligent decision system based on a case constraint big model to execute the method for constructing an evaluation scenario set of an intelligent decision system based on a case constraint big model as described in any one of claims 1-8.

Images