Llm-based state diagram model quality evaluation method and system

Abstract

The application provides a state diagram model quality evaluation method and system based on LLM, which comprises the following steps: constructing an SCBI quality index system for a state diagram model; calculating the original numerical value of each quality index in static structure data and dynamic behavior data according to the SCBI quality index system; constructing a multi-evaluation subject collaborative decision model, generating an evaluation result based on the original numerical value of the quality index and task context information under the constraint of the multi-evaluation subject and evaluation preference; calculating the difference of all evaluation results, and iteratively optimizing based on a consistency judgment condition until a final evaluation result is obtained by meeting a convergence condition; calculating the dimension score of each quality dimension according to the final evaluation result, and calculating a comprehensive quality score by weighting according to the dimension weight of each quality dimension; and generating a quality evaluation result according to the final evaluation result and the comprehensive quality score. The application can improve the automation degree and result consistency of quality evaluation.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for quality evaluation of state diagram models based on LLM. Background Technology

[0002] With the widespread application of Model-Based Systems Engineering (MBSE) in safety-critical fields such as aerospace, automotive electronics, and rail transportation, the model, as the core carrier of system design, analysis, and verification, directly determines the reliability, safety, and development efficiency of the final product. Rhapsody, as a mainstream MBSE modeling tool in the industry, supports multiple modeling languages ​​such as state machines, activity diagrams, and sequence diagrams, and is widely used for behavioral modeling and code generation of complex embedded systems. How to systematically and quantitatively evaluate the quality of Rhapsody models has become a crucial aspect of ensuring system quality.

[0003] Currently, research on model quality assessment mainly falls into two categories: one focuses on static structural analysis, identifying design flaws by examining the model's syntax, semantics, and design specifications (such as state machine integrity and interface consistency); the other focuses on dynamic simulation verification, confirming the model's correctness by executing the model and observing whether its behavior matches expectations. In dynamic simulation, co-simulation technology allows multiple heterogeneous models (such as the Rhapsody model and models from other simulation tools) to run together to verify system-level integrated behavior. However, most existing co-simulation methods focus on verifying the correctness of the overall system function, i.e., checking whether the expected system-level functions are achieved when multiple models run together, while neglecting quantitative assessments of dimensions such as the internal behavior quality of individual models, the interaction quality between models, and model maintainability. Furthermore, existing quality assessment methods often rely on human experience to set evaluation indicators and weights, resulting in strong subjectivity, poor repeatability, and difficulty in adapting to the automated evaluation needs of complex multi-model systems.

[0004] In recent years, large language models have demonstrated powerful capabilities in code analysis, test case generation, and defect prediction, but their application in model quality assessment is still in its infancy. How to leverage the semantic understanding and reasoning capabilities of large language models to construct an automated and intelligent model quality assessment system is a pressing technical problem to be solved in this field. Summary of the Invention

[0005] Therefore, the purpose of this invention is to provide a method and system for evaluating the quality of state diagram models based on LLM, so as to at least solve the shortcomings of the above-mentioned technologies.

[0006] This invention proposes a quality assessment method for state diagram models based on LLM, comprising: A quality index system for SCBI oriented state diagram models is constructed, which characterizes model quality from four dimensions: structural quality, coverage, behavioral correctness, and interaction performance. Obtain the static structural data of the state diagram model and the dynamic behavior data generated by the state diagram model during co-simulation, and calculate the original value of each quality indicator according to the SCBI quality indicator system. Construct a multi-evaluation subject collaborative decision-making model based on a large language model. The multi-evaluation subject includes at least two evaluation subjects with different preset evaluation roles, and each evaluation role is associated with a set of preset evaluation preference constraints. Under the constraints of multiple evaluation subjects and evaluation preferences, evaluation results for each evaluation subject are generated based on the original values ​​of the quality indicators and the task context information of the state graph model. The evaluation results include dimension weight allocation results, quality indicator weight allocation results, and quality indicator scoring results. The difference between all evaluation results is calculated, and iterative optimization is performed based on the preset consistency judgment conditions until the convergence condition is met to obtain a unified final evaluation result. The dimension score of each quality dimension is calculated based on the quality indicator weight allocation result and quality indicator scoring result in the final evaluation result, and a weighted calculation is performed according to the dimension weight of each quality dimension to calculate the comprehensive quality score of the state diagram model. The quality assessment result of the state diagram model is generated based on the final evaluation result and the comprehensive quality score.

[0007] Furthermore, in the SCBI quality indicator system, the structural quality dimensions include state integrity, connectivity, state nesting depth, and interface integrity. Coverage dimensions include state coverage, migration coverage, and event coverage; The behavioral correctness dimension includes state transition accuracy, response accuracy, and simulation pass rate; Interaction performance dimensions include synchronization error rate, resource utilization, and average response latency.

[0008] Furthermore, the static structural data includes a set of states, state transition relationships, and interface definition information; the dynamic behavioral data includes model response data, event interaction data, and time synchronization data; and the evaluation preference constraints include priority constraints, normalization constraints, and value constraints.

[0009] Furthermore, the steps of calculating the difference between all evaluation results and iteratively optimizing based on preset consistency criteria until convergence is met to obtain a unified final evaluation result include: The difference degree is calculated for all evaluation results to obtain the difference metric value, and the difference metric value is compared with the preset consistency threshold. If the difference metric is greater than the preset consistency threshold, the input information of each evaluation subject is updated and the evaluation result is regenerated through an iterative optimization mechanism until the difference metric converges to no greater than the consistency threshold, or the number of iterations reaches the preset upper limit, and a unified final evaluation result is obtained.

[0010] Furthermore, the steps for calculating the degree of difference of all evaluation results to obtain the difference measure value include: The corresponding dimension weight difference value is calculated based on the average of the weight ranges of all assessment subjects in the four dimensions of structural quality, coverage, behavioral correctness and interaction performance. Calculate the weight range and score range for all quality indicators respectively, and obtain the corresponding indicator weight difference value and indicator score difference value based on the average of the weight range and score range.

[0011] This invention also proposes a state diagram model quality evaluation system based on LLM, comprising: The system construction module is used to build the SCBI quality index system for state diagram models. This system characterizes the model quality from four dimensions: structural quality, coverage, behavioral correctness, and interaction performance. The data acquisition module is used to acquire the static structural data of the state diagram model and the dynamic behavior data generated by the state diagram model during the co-simulation process, and to calculate the original value of each quality indicator according to the SCBI quality indicator system. The model building module is used to build a multi-evaluation subject collaborative decision-making model based on a large language model. The multi-evaluation subject includes at least two evaluation subjects with different preset evaluation roles, and each evaluation role is associated with a set of preset evaluation preference constraints. An evaluation processing module is used to generate evaluation results for each of the evaluation subjects based on the original values ​​of the quality indicators and the task context information of the state graph model, under the constraints of the multiple evaluation subjects and the evaluation preferences. The evaluation results include dimension weight allocation results, quality indicator weight allocation results, and quality indicator scoring results. The iterative optimization module is used to calculate the difference of all evaluation results and perform iterative optimization based on the preset consistency judgment conditions until the convergence condition is met, so as to obtain a unified final evaluation result. The data calculation module is used to calculate the dimension score of each quality dimension based on the quality indicator weight allocation result and quality indicator scoring result in the final evaluation result, and to perform weighted calculation based on the dimension weight of each quality dimension to calculate the comprehensive quality score of the state diagram model. The quality assessment module is used to generate the quality assessment result of the state diagram model based on the final assessment result and the comprehensive quality score.

[0012] Furthermore, the iterative optimization module specifically includes: The difference calculation unit is used to calculate the difference of all evaluation results to obtain the difference measurement value, and compare the difference measurement value with the preset consistency threshold. The iterative optimization unit is used to update the input information of each evaluation subject and regenerate the evaluation result through an iterative optimization mechanism if the difference metric value is greater than a preset consistency threshold, until the difference metric value converges to no greater than the consistency threshold, or the number of iterations reaches a preset upper limit, so as to obtain a unified final evaluation result.

[0013] Furthermore, the difference calculation unit is specifically used for: The corresponding dimension weight difference value is calculated based on the average of the weight ranges of all assessment subjects in the four dimensions of structural quality, coverage, behavioral correctness and interaction performance. Calculate the weight range and score range for all quality indicators respectively, and obtain the corresponding indicator weight difference value and indicator score difference value based on the average of the weight range and score range.

[0014] The present invention also proposes a storage medium on which a computer program is stored, which, when executed by a processor, implements the above-described LLM-based state diagram model quality evaluation method.

[0015] The present invention also proposes a computer, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described LLM-based state diagram model quality evaluation method.

[0016] Compared with existing technologies, the LLM-based state graph model quality evaluation method and system of this invention, by constructing an SCBI quality index system including four dimensions of structural quality, coverage, behavioral correctness and interaction performance, achieves comprehensive evaluation of the state graph model from multiple dimensions, overcoming the shortcomings of existing technologies that rely on only a single dimension or subjective evaluation. By constructing a multi-evaluation subject collaborative decision-making model based on a large language model and utilizing the evaluation preference constraints corresponding to different evaluation roles, it is beneficial to reduce the subjective bias caused by traditional reliance on expert manual scoring, and improve the automation level and consistency of quality evaluation results. Attached Figure Description

[0017] Figure 1 This is a flowchart of the LLM-based state diagram model quality evaluation method in the first embodiment of the present invention; Figure 2 This is a structural block diagram of the state diagram model quality evaluation system based on LLM in the second embodiment of the present invention; Figure 3 This is a structural block diagram of the computer in the third embodiment of the present invention.

[0018] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0019] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0021] Example 1 Please see Figure 1 The figure shows a quality evaluation method for a state diagram model based on LLM in the first embodiment of the present invention. The method specifically includes steps S101 to S107: S101, Construct an SCBI quality index system for state diagram models. This system characterizes model quality from four dimensions: structural quality, coverage, behavioral correctness, and interaction performance. In practical implementation, the aircraft operation and control simulation system is used as the analysis object. This system includes three Rhapsody functional models: the Vehicle Management System (VMS), the Escape Survival Subsystem (ESS), and the Vehicle Control System (VCS).

[0022] Furthermore, an SCBI quality index system for state diagram models is constructed from four dimensions: structural quality, coverage, behavioral correctness, and interaction performance. Specifically, this embodiment addresses the quality evaluation requirements of state diagram models by constructing an SCBI quality index system from four dimensions: structure (S), coverage (C), behavior correctness (B), and interaction (I).

[0023] The structural quality dimension characterizes the completeness, connectivity, and hierarchical rationality of the static structure of the state diagram model; the coverage dimension characterizes the sufficiency of test coverage of model states, transitions, and events during co-simulation; the behavioral correctness dimension characterizes whether the model's state transitions and response behaviors during co-simulation execution meet expectations; and the interaction performance dimension characterizes the model's time synchronization performance, resource consumption, and response timeliness in the co-simulation environment. Please refer to Table 1 below. The SCBI quality indicator system includes the following dimensions and quality indicators: Table 1. Multidimensional Model Quality Index System

[0024] The raw values ​​of the above quality indicators serve as the input for the subsequent multi-assessment collaborative decision-making model, and are used to generate the dimension weight allocation results, quality indicator weight allocation results, and quality indicator scoring results.

[0025] Furthermore, the quality indicators for the structural quality dimension and their original numerical calculation methods are determined.

[0026] The structural quality dimensions include state integrity (S1), connectivity (S2), state nesting depth (S3), and interface integrity (S4).

[0027] State integrity (S1) is used to characterize the proportion of non-isolated states in a state diagram model. Let the set of states be... The migration set is For each state Determine whether there are no incoming edges (no migrations pointing to it) and no outgoing edges (no migrations originating from it). If both conditions are met, it is an isolated state. Count the number of isolated states. Then the original value of state integrity is:

[0028] The initial state is not considered an isolated state; the terminating state is considered a non-isolated state if it has an incoming edge.

[0029] Connectivity (S2) is used to characterize the proportion of reachable states from the initial state among all states. Unreachable states indicate redundancy or flawed design in the model. Let's assume we start from the initial state... Starting from this point, the set of reachable states obtained by traversing through the migration relationships is: The original value of connectivity is:

[0030] In this embodiment, a depth-first search or breadth-first search approach is used to determine the set of reachable states. The state nesting depth (S3) is used to characterize the hierarchical complexity of the state graph model. Excessive depth (typically exceeding 4 levels) significantly reduces the model's understandability and maintainability. Let... Representing state Given the nesting level, the original value of the state nesting depth is:

[0031] The initial state is denoted as level 0, its direct substates are denoted as level 1, and so on.

[0032] Interface integrity (S4) characterizes the degree of consistency between the actual defined interfaces in the model and the interfaces required by the design. Let the number of ports, flow attributes, and total operations defined in the model be... The total number of interfaces specified in the requirements document or design specification is The original value for interface integrity is:

[0033] when When this happens, the indicator is marked as not participating in the interface integrity evaluation of the current model, or a default value is assigned according to preset rules.

[0034] Furthermore, the quality metrics for the coverage dimension and their original numerical calculation methods are determined. The coverage dimension includes state coverage (C1), transition coverage (C2), and event coverage (C3). State coverage (C1) is used to characterize the proportion of states that are entered at least once during co-simulation out of the total number of states in the state machine. Let the number of states that are entered at least once during simulation be... The total number of states in the state machine is Then the original value of the state coverage is:

[0035] in, The initial state is not included.

[0036] Migration coverage (C2) characterizes the proportion of migrations that are triggered at least once during co-simulation out of the total number of migrations defined in the model. Let the number of migrations that are triggered at least once during simulation be... The total number of migrations defined in the model is The original value of the migration coverage is:

[0037] Event coverage (C3) characterizes the proportion of events that are triggered at least once in the model, out of the total number of events. These events include call events, signal events, time events, and change events. Let the number of events triggered at least once during the simulation be... The total number of events defined in the model is The original value of event coverage is:

[0038] Furthermore, the quality indicators and their original numerical calculation methods for the behavioral correctness dimension are determined. The behavioral correctness dimension includes state transition accuracy (B1), response accuracy (B2), and simulation pass rate (B3). State transition accuracy (B1) characterizes the proportion of correctly completed state transitions in triggered transitions during co-simulation. Let the number of transitions where the protection condition is met at trigger and the actual state reaches the design expectation be denoted as . The total number of migrations triggered is Then the original value of the state transition accuracy is:

[0039] Response accuracy (B²) characterizes the proportion of times a model correctly responds to an input stimulus or event. Let the number of responses in which the model output matches the expected output be denoted as B². The total number of responses during the simulation process is The original value of the response accuracy is:

[0040] The simulation pass rate (B3) characterizes the proportion of test cases that successfully pass during co-simulation. Let the number of test cases that pass assertion checks and terminate without errors or exceptions be denoted as . The total number of test cases executed is The original value of the simulation pass rate is:

[0041] Furthermore, the quality indicators for the interaction performance dimension and their original numerical calculation methods are determined.

[0042] Interaction performance dimensions include synchronization error rate (I1), resource utilization (I2), and average response latency (I3).

[0043] The synchronization error rate (I1) is used to characterize the degree of deviation between the sending and receiving times of synchronization messages during co-simulation. Let the first... The timestamp of each synchronized message is: The received timestamp is The total number of synchronization messages is N, and the maximum allowable synchronization error of the system is... The original value of the synchronization error rate is:

[0044] The synchronization error rate is calculated based on the same simulated clock, and the smaller the value, the higher the synchronization accuracy.

[0045] Resource utilization (I²) is used to characterize the degree of computational resource consumption by the state diagram model process during co-simulation. Let N be the number of samples taken within the total simulation duration, and the i-th... The CPU utilization rate of the state graph model process at the time of the next sampling was... The physical memory used is The total physical memory of the system is The original value of the resource utilization rate is:

[0046] This metric should be collected under the same simulation platform and resource configuration to ensure comparability between different state diagram models.

[0047] Average response latency (13) is used to characterize the average time interval between the sending of a request message and the receiving of the corresponding response message. Let the first... The timestamp of the request message is: The timestamp for receiving the corresponding response message is If the total number of request-response pairs is N, then the original value of the average response latency is:

[0048] The average response delay is also calculated based on a unified simulation clock; the smaller the value, the better the model's response performance.

[0049] S102, Obtain the static structural data of the state diagram model and the dynamic behavior data generated by the state diagram model during the co-simulation process, and calculate the original value of each quality indicator according to the SCBI quality indicator system. In practice, the static structure data and dynamic behavior data of the state diagram model are obtained, and the raw values ​​of each quality indicator are calculated.

[0050] Specifically, the state diagram model is analyzed to obtain static structure data.

[0051] Static structural data is obtained by parsing the state diagram model, including state sets, state transition relationships, and interface definition information. In this embodiment, the currently active Rhapsody project is accessed through an environment adapter, and the state machine objects in the project are traversed to extract the static structural data of each state diagram model. The static structural data includes: state set information, including state name, state type, parent-child hierarchy, entry action, exit action, and internal transition information; state transition relationship information, including the source state, target state, triggering event, and protection conditions for each transition; and interface definition information, including the ports, flow attributes, and operations defined in the model, as well as the corresponding data types, directions, and parameter lists. The above parsing results are stored as a structured data file as the basis for subsequent raw numerical calculations.

[0052] Furthermore, for the three Rhapsody functional models in this embodiment, namely the Aircraft Management System (VMS), the Cockpit System Escape Subsystem (ESS), and the Aircraft Control System (VCS), their respective static structural data are parsed and used to calculate some statistics in the dimensions of state integrity, connectivity, state nesting depth, interface integrity, and coverage.

[0053] Furthermore, dynamic behavior data is collected during the collaborative simulation process.

[0054] Dynamic behavioral data is collected in real time through observation nodes during the co-simulation process, including model response data, event interaction data, and time synchronization data. In this embodiment, the testing tool is configured as an observation node in the co-simulation environment, listening to event messages and state change messages published by the state graph model in real time during simulation execution. The dynamic behavioral data includes: model response data, including the excitation event sending time, model output time, and output content; event interaction data, including event trigger records, transition trigger records, state entry records, and state exit records; time synchronization data, including the sending timestamp, receiving timestamp, and timestamps corresponding to request and response messages of synchronization messages; and resource sampling data, including the CPU utilization and memory utilization of the state graph model process during simulation execution. All dynamic behavioral data carries a timestamp under a unified simulation clock and is written to a log file in chronological order to ensure the computability of raw values ​​such as state coverage, transition coverage, event coverage, state transition accuracy, response accuracy, simulation pass rate, synchronization error rate, resource utilization, and average response latency. In this embodiment, the collaborative simulation environment can adopt a multi-model linkage method based on the DDS soft bus. The DDS soft bus is only an optional implementation method and does not constitute a limitation on the scope of protection of this invention.

[0055] In this embodiment, the raw values ​​of each quality indicator are calculated based on static structural data and dynamic behavioral data. Based on the SCBI quality indicator system constructed above, the raw values ​​of indicators such as state integrity, connectivity, state nesting depth, and interface integrity in the structural quality dimension are calculated using the acquired static structural data; the raw values ​​of indicators such as state coverage, migration coverage, and event coverage in the coverage dimension are calculated using the aforementioned static structural data and dynamic behavioral data; and the raw values ​​of various indicators in the behavioral correctness dimension and interaction performance dimension are calculated using dynamic behavioral data.

[0056] Specifically: the original value of the structural quality dimension (S) is mainly calculated based on the state set, transition relationship and interface definition information; the original value of the coverage dimension (C) is calculated by the total number of states, transitions and events defined in the model and the number of states, transitions and events accessed and triggered during the co-simulation process; the original value of the behavior correctness dimension (B) is calculated based on the transition execution results, model response results and test case execution results during the co-simulation process; the original value of the interaction performance dimension (1) is calculated based on the synchronization message timestamp, request response timestamp and resource sampling data.

[0057] In this embodiment, taking a co-simulation test batch as an example, the original values ​​of various quality indicators of the three state diagram models VMS, ESS and VCS are calculated, and the results are shown in Table 2.

[0058] Table 2 Original values ​​of each model's quality index

[0059] S103, Construct a multi-evaluation subject collaborative decision-making model based on a large language model. The multi-evaluation subject includes at least two evaluation subjects with different preset evaluation roles. Each evaluation role is associated with a set of preset evaluation preference constraints. In practical implementation, a multi-evaluation subject collaborative decision-making model based on a large language model is constructed, and evaluation roles, evaluation preference constraints, and input information are configured.

[0060] Among them, a collaborative decision-making model for multiple assessment subjects is constructed, and pre-set assessment roles and evaluation preference constraints are configured for each assessment subject.

[0061] After obtaining the original values ​​of the quality indicators for each state diagram model, a multi-evaluation subject collaborative decision-making model based on a large language model is constructed. This multi-evaluation subject collaborative decision-making model includes multiple evaluation subjects, each corresponding to a different preset evaluation role, to evaluate the quality of the state diagram model from different technical perspectives.

[0062] In this embodiment, the multiple evaluation subjects include an architecture evaluation subject, used to evaluate the quality of the state diagram model from the perspective of system structure design, interface organization, and model collaboration; a requirements evaluation subject, used to evaluate the quality of the state diagram model from the perspective of requirements implementation, test coverage, and functional compliance; and a security evaluation subject, used to evaluate the quality of the state diagram model from the perspective of security-critical behaviors, abnormal state handling, and system reliability.

[0063] Furthermore, in another embodiment, the multi-evaluation subject collaborative decision-making model may also include a coordinating subject, which is used to manage the input and output processes of each evaluation subject, summarize the output results of each evaluation subject, and distribute updated input information to each evaluation subject, so as to achieve consistent convergence of results of multiple evaluation subjects.

[0064] To ensure that different assessment subjects reflect differentiated assessment perspectives during collaborative decision-making, evaluation preference constraints corresponding to their pre-defined assessment roles are configured for each assessment subject. These evaluation preference constraints are limited by pre-defined rules and take effect when the assessment subject generates dimensional weight allocation results, quality indicator weight allocation results, and quality indicator scoring results.

[0065] In this embodiment, the evaluation preference constraints include at least: quality dimension priority constraints, used to limit the order of attention of different evaluation subjects to the four dimensions of structural quality, coverage, behavioral correctness and interaction performance; quality indicator priority constraints, used to limit the degree of attention of different evaluation subjects to key quality indicators under each dimension; weight allocation sum constraints, used to limit the sum of dimension weights and the sum of quality indicator weights under each dimension to meet preset normalization conditions; and indicator score range constraints, used to limit the score results of each quality indicator to be within a preset score range, which in this embodiment is 0 to 100 points.

[0066] In another embodiment, the prompt template framework includes the following: "Role: {role_name);" Task: Evaluate the quality of the state graph model based on the original values ​​of the input state graph model quality indicators; Focus dimensions: {focus_dimensions}; Focus indicators: {focus_indicators}; Input content includes the original values ​​of quality indicators, the definition and value direction of each quality indicator, task context information, and the evaluation preference constraints corresponding to the current evaluation subject; The output requirements include: outputting the dimension weight allocation results for the four quality dimensions with a total of 1; outputting the quality indicator weight allocation results for each quality indicator under each dimension with a total weight of 1 within the same dimension; outputting the score results for each quality indicator with a score range of 0 to 100; and outputting the corresponding explanation information. The output uses a preset structured format. Furthermore, the prompt template may also include results analysis requirements to guide the assessment subject or large language model to generate weakness diagnosis descriptions and optimization suggestions based on quality assessment data. In one embodiment, the prompt template may include the following: "Based on the input state diagram model quality assessment data, please output a summary of the model's overall quality performance, key weak dimensions or weak quality indicators, diagnostic explanations for weaknesses, and corresponding optimization suggestions. The output should include a description of the model's functions, application scenario information, and task criticality, and should be consistent with the input data." Furthermore, we construct input and output data structures for multiple assessment subjects.

[0067] In this embodiment, the input content for each evaluation subject includes: the original values ​​of each quality indicator obtained above; the name, definition, value direction, and quality dimension information of each quality indicator; the task context information of the corresponding state diagram model; and the evaluation preference constraints corresponding to the current evaluation subject role. The value direction information is used to distinguish between positive indicators, negative indicators, and complexity representation indicators; the task context information includes functional description information, application scenario information, and task criticality information.

[0068] In this embodiment, the task context information constructed for the three state diagram models may include the following: the VMS model is used to execute aircraft management-related control functions, responsible for main power management and coordination of related control signals; the ESS model is used to execute cockpit escape-related functions, involving key processes such as canopy control, ejection procedures, and parachute deployment; and the VCS model is used to execute aircraft control and interaction-related functions, responsible for control command input and some human-machine interaction behavior processing. The above task context information, along with the original values ​​of each quality indicator, are input into each evaluation subject to guide the evaluation subject to conduct differentiated evaluations of each dimension and each quality indicator based on the model's function and application scenario.

[0069] In this embodiment, the output of each evaluation subject includes: the dimensional weight allocation results for each quality dimension; the quality indicator weight allocation results for each quality indicator under each dimension; the quality indicator score results for each quality indicator; and the explanatory information corresponding to the weight allocation results and score results.

[0070] S104, under the constraints of the multiple evaluation subjects and the evaluation preferences, the evaluation results of each of the evaluation subjects are generated based on the original values ​​of the quality indicators and the task context information of the state diagram model, wherein the evaluation results include dimension weight allocation results, quality indicator weight allocation results and quality indicator scoring results. In practice, the multi-assessment subject collaborative decision-making model generates dimensional weight allocation results, quality indicator weight allocation results, and quality indicator scoring results, and achieves consistency convergence through difference measurement.

[0071] This involves inputting the original values ​​of quality indicators and task context information into each assessment entity to generate initial assessment results.

[0072] After completing the construction of the collaborative decision-making model for multiple assessment subjects, the information is input into each assessment subject respectively.

[0073] Each assessment subject generates an initial assessment result based on the input information, under the constraints of its respective evaluation preferences.

[0074] In this embodiment, the initial evaluation results for each evaluation subject include: the dimension weight allocation results for the four quality dimensions; the quality indicator weight allocation results for each quality indicator under each dimension; the quality indicator score results for each quality indicator; and the explanatory information corresponding to the weight allocation results and the quality indicator score results.

[0075] Taking the dimension weights of the VMS model as an example, Table 3 shows an example of the dimension weight allocation for the output of the first round of evaluation of the VMS model: Table 3. Dimensional Weight Allocation of the VMS Model

[0076] S105, calculate the difference of all evaluation results, and perform iterative optimization based on the preset consistency judgment conditions until the convergence condition is met, so as to obtain a unified final evaluation result. Furthermore, step S105 specifically includes steps S1051 to S1052: S1051, calculate the degree of difference for all evaluation results to obtain a difference metric value, and compare the difference metric value with a preset consistency threshold; S1052, if the difference metric is greater than the preset consistency threshold, the input information of each evaluation subject is updated and the evaluation result is regenerated through an iterative optimization mechanism until the difference metric converges to no greater than the consistency threshold, or the number of iterations reaches the preset upper limit, and a unified final evaluation result is obtained.

[0077] Furthermore, step S1051 also includes the following steps: The corresponding dimension weight difference value is calculated based on the average of the weight ranges of all assessment subjects in the four dimensions of structural quality, coverage, behavioral correctness and interaction performance. Calculate the weight range and score range for all quality indicators respectively, and obtain the corresponding indicator weight difference value and indicator score difference value based on the average of the weight range and score range.

[0078] In practice, the differences in the output results of each evaluation subject are measured.

[0079] After obtaining the initial evaluation results from each evaluation subject, the differences in the dimension weight allocation results, quality indicator weight allocation results, and quality indicator scoring results output by each evaluation subject are measured to obtain the difference measurement value.

[0080] In this embodiment, the difference metric consists of the following three parts: dimension weight difference value. This is used to reflect the weighting deviations of different assessment subjects on the four quality dimensions; the weighting difference value of quality indicators. This is used to reflect the deviations in the weighting of various quality indicators by different assessment subjects; the difference in quality indicator scores. These values ​​are used to reflect the deviations in the scoring results of various quality indicators by different assessment subjects. The three types of difference values ​​are calculated as follows:

[0081] in, This indicates the dimensional weights assigned to each quality dimension by different assessment subjects;

[0082] In the formula, This indicates that different evaluation subjects have different opinions on the first. The quality indicators are given their weights;

[0083] In the formula, This indicates that different evaluation subjects have different opinions on the first. The scoring results given by each quality indicator.

[0084] Furthermore, the above deviations are synthesized according to a preset method to obtain an overall difference measure used to judge the degree of consistency. :

[0085] In this embodiment, taking the first round results of the VMS model as an example, according to Table 3, the ranges of the four dimensions are as follows: Dimension: 0.34 - 0.20 = 0.14; Dimension: 0.30 - 0.18 = 0.12; Dimension: 0.36 - 0.20 = 0.16; Dimension: 0.28 - 0.14 = 0.14; Therefore:

[0086] Similarly, calculate , Therefore, the first round of overall variance measurement value of the VMS model for:

[0087] When the difference metric is greater than a preset threshold and the number of iterations has not reached a preset upper limit, the inputs of each evaluation subject are updated, and the above steps are re-executed based on the updated inputs; when the difference metric is less than or equal to the preset threshold, or the number of iterations reaches a preset upper limit, the iteration ends.

[0088] In this embodiment, the preset threshold is set to 0.08, and the preset maximum number of iterations is set to 3. If a model satisfies the following condition before reaching the 3rd iteration: If the threshold condition is not met after three iterations, the final output result is used as the convergence result.

[0089] In one embodiment, the coordinating entity is responsible for collecting the output results of each assessment entity, calculating the difference metric, and sending input updates to each assessment entity when the iteration conditions are met. The input updates include at least one of the following methods: inputting the weight allocation results and quality indicator scores of other assessment entities as supplementary reference information into the current assessment entity; inputting quality dimensions or quality indicators with significant differences between different assessment entities as key review targets into the current assessment entity; and inputting the explanations given by each assessment entity in the previous round as supplementary explanation information into the current assessment entity.

[0090] After consensus convergence, the dimension weight allocation results, quality indicator weight allocation results, and quality indicator scoring results output by each evaluation subject are uniformly processed to obtain unified quality indicator weights and indicator scores. In this embodiment, the uniform processing adopts an arithmetic average method, that is, the average of the corresponding results output by each evaluation subject in the convergence round is taken as the final result. Through the above method, the final dimension weight allocation results, final quality indicator weight allocation results, and final quality indicator scoring results of the state diagram model to be evaluated are obtained, as shown in Tables 4, 5, and 6, respectively. These results serve as the input basis for subsequent calculation of the scores of each dimension and the overall quality score.

[0091] Table 4: Weights of Each Dimension in the State Diagram Model

[0092] Table 5: Weights of various quality indicators in the state diagram model

[0093] Table 6: Scoring Results of Quality Indicators for the State Diagram Model

[0094] S106. Calculate the dimension score of each quality dimension based on the quality indicator weight allocation result and quality indicator scoring result in the final evaluation result, and perform a weighted calculation based on the dimension weight of each quality dimension to calculate the comprehensive quality score of the state diagram model. In the specific implementation, based on the unified quality indicator weight allocation results and quality indicator scoring results obtained above, the dimension scores of structural quality dimension, coverage dimension, behavioral correctness dimension and interaction performance dimension are calculated respectively.

[0095] In this embodiment, let the first The first dimension The weight of each quality indicator is The corresponding quality indicator score is Then the first Dimensional scores of each dimension Calculate using the following formula:

[0096] in, Indicates the first The number of quality indicators included in each dimension. In this embodiment, the scores of each dimension of the three state diagram models, VMS, ESS, and VCS, are shown in Table 7: Table 7: Scores for Each Dimension

[0097] After obtaining the scores for each dimension, the overall quality score of the state diagram model is calculated by combining the unified dimension weight allocation results obtained above.

[0098] Let the first The dimensional weights of each dimension are The corresponding dimension score is The overall quality score of the state diagram model is then... Calculate using the following formula:

[0099] In this embodiment, based on the dimension weight allocation results in Table 4 and the scores of each dimension in Table 7, the comprehensive quality scores of the three state diagram models VMS, ESS, and VCS are further calculated to be 86.91, 96.02, and 89.96, respectively. The comprehensive quality scores serve as the input basis for generating subsequent quality assessment results.

[0100] S107, Generate the quality assessment result of the state diagram model based on the final evaluation result and the comprehensive quality score.

[0101] In practical implementation, after obtaining the quality index scores, scores for each dimension, and a comprehensive quality score for each state graph model, a quality assessment result for the state graph model is generated. The quality assessment result includes at least: a comprehensive quality score, used to characterize the overall quality level of the state graph model; model problem diagnostic information, used to identify weaknesses in the state graph model regarding structural quality, coverage, behavioral correctness, or interaction performance based on the quality index scores; and optimization suggestion information, used to generate corresponding model optimization suggestions for the identified weaknesses.

[0102] In this embodiment, the generated quality assessment results are as follows: "The ESS model has the highest overall quality score of 96.02, indicating a high overall quality level and a relatively balanced performance across all dimensions. In particular, the structural quality, coverage, and behavioral correctness are outstanding, suggesting that the model has a good quality foundation in terms of structural design, test coverage, and behavioral execution. However, there is still room for further optimization in its interaction performance dimension." The VCS model has a comprehensive quality score of 89.96, which is good overall. The structural quality and behavioral correctness are stable. However, the coverage dimension score is relatively low, and the scores for state nesting depth and resource utilization are not high. This indicates that there is still room for improvement in terms of test coverage sufficiency, hierarchical complexity control and running efficiency. The VMS model achieved a comprehensive quality score of 86.91, indicating a good overall quality level. However, compared to ESS and VCS, its coverage and interaction performance were relatively weaker, particularly in terms of state coverage, migration coverage, event coverage, resource utilization, and average response latency. This suggests that the model needs further improvement in quality through supplementing test scenarios, optimizing communication and scheduling strategies, and reducing resource consumption. In summary, the LLM-based state graph model quality evaluation method in the above embodiments of the present invention, by constructing an SCBI quality index system including four dimensions—structural quality, coverage, behavioral correctness, and interaction performance—enables a comprehensive evaluation of the state graph model from multiple dimensions, overcoming the shortcomings of existing technologies that rely solely on a single dimension or subjective evaluation. By constructing a multi-evaluation subject collaborative decision-making model based on a large language model and utilizing evaluation preference constraints corresponding to different evaluation roles, it helps to reduce the subjective bias caused by traditional reliance on expert manual scoring, and improves the automation level and consistency of quality evaluation results.

[0103] Example 2 In another aspect, this invention proposes a quality evaluation system based on an LLM-based state diagram model; please refer to [link / reference needed]. Figure 2 The figure shows a state diagram model quality evaluation system based on LLM according to the second embodiment of the present invention. The system includes: System construction module 11 is used to build an SCBI quality index system for state diagram models. This system characterizes the model quality from four dimensions: structural quality, coverage, behavioral correctness, and interaction performance. The data acquisition module 12 is used to acquire the static structural data of the state diagram model and the dynamic behavior data generated by the state diagram model during the co-simulation process, and to calculate the original value of each quality indicator according to the SCBI quality indicator system. Model building module 13 is used to build a multi-evaluation subject collaborative decision-making model based on a large language model. The multi-evaluation subject includes at least two evaluation subjects with different preset evaluation roles, and each evaluation role is associated with a set of preset evaluation preference constraints. The evaluation processing module 14 is used to generate evaluation results for each of the evaluation subjects based on the original values ​​of the quality indicators and the task context information of the state graph model under the constraints of the multiple evaluation subjects and the evaluation preferences. The evaluation results include dimension weight allocation results, quality indicator weight allocation results and quality indicator scoring results. The iterative optimization module 15 is used to calculate the difference of all evaluation results and perform iterative optimization based on the preset consistency judgment conditions until the convergence condition is met, so as to obtain a unified final evaluation result. The data calculation module 16 is used to calculate the dimension score of each quality dimension based on the quality indicator weight allocation result and quality indicator scoring result in the final evaluation result, and to perform weighted calculation based on the dimension weight of each quality dimension to calculate the comprehensive quality score of the state diagram model. The quality assessment module 17 is used to generate the quality assessment result of the state diagram model based on the final assessment result and the comprehensive quality score.

[0104] Furthermore, the iterative optimization module 15 specifically includes: The difference calculation unit is used to calculate the difference of all evaluation results to obtain the difference measurement value, and compare the difference measurement value with the preset consistency threshold. The iterative optimization unit is used to update the input information of each evaluation subject and regenerate the evaluation result through an iterative optimization mechanism if the difference metric value is greater than a preset consistency threshold, until the difference metric value converges to no greater than the consistency threshold, or the number of iterations reaches a preset upper limit, so as to obtain a unified final evaluation result.

[0105] Furthermore, the difference calculation unit is specifically used for: The corresponding dimension weight difference value is calculated based on the average of the weight ranges of all assessment subjects in the four dimensions of structural quality, coverage, behavioral correctness and interaction performance. Calculate the weight range and score range for all quality indicators respectively, and obtain the corresponding indicator weight difference value and indicator score difference value based on the average of the weight range and score range.

[0106] The functions or operation steps implemented by the above modules and units are largely the same as those in the above method embodiments, and will not be repeated here.

[0107] The LLM-based state diagram model quality evaluation system provided in this embodiment of the invention has the same implementation principle and technical effects as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the system embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0108] Example 3 This invention also proposes a computer, please refer to [link / reference]. Figure 3 The computer shown in the third embodiment of the present invention includes a memory 10, a processor 20, and a computer program 30 stored in the memory 10 and executable on the processor 20. When the processor 20 executes the computer program 30, it implements the above-described LLM-based state diagram model quality evaluation method.

[0109] The memory 10 includes at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10 can be an internal storage unit of a computer, such as the computer's hard disk. In other embodiments, the memory 10 can be an external storage device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Furthermore, the memory 10 can include both internal and external storage units of the computer. The memory 10 can be used not only to store application software and various types of data installed on the computer, but also to temporarily store data that has been output or will be output.

[0110] In some embodiments, the processor 20 may be an electronic control unit (ECU, also known as a vehicle computer), a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip, used to run program code stored in the memory 10 or process data, such as executing access restriction programs.

[0111] It should be pointed out that, Figure 3 The structure shown does not constitute a limitation on the computer. In other embodiments, the computer may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0112] This invention also proposes a storage medium storing a computer program that, when executed by a processor, implements the LLM-based state diagram model quality evaluation method described above.

[0113] Those skilled in the art will understand that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can mean any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0114] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0115] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0116] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0117] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A quality evaluation method for a state diagram model based on LLM, characterized in that, include: A quality index system for SCBI oriented state diagram models is constructed, which characterizes model quality from four dimensions: structural quality, coverage, behavioral correctness, and interaction performance. Obtain the static structural data of the state diagram model and the dynamic behavior data generated by the state diagram model during co-simulation, and calculate the original value of each quality indicator according to the SCBI quality indicator system. Construct a multi-evaluation subject collaborative decision-making model based on a large language model. The multi-evaluation subject includes at least two evaluation subjects with different preset evaluation roles, and each evaluation role is associated with a set of preset evaluation preference constraints. Under the constraints of multiple evaluation subjects and evaluation preferences, evaluation results for each evaluation subject are generated based on the original values ​​of the quality indicators and the task context information of the state graph model. The evaluation results include dimension weight allocation results, quality indicator weight allocation results, and quality indicator scoring results. The difference between all evaluation results is calculated, and iterative optimization is performed based on the preset consistency judgment conditions until the convergence condition is met to obtain a unified final evaluation result. The dimension score of each quality dimension is calculated based on the quality indicator weight allocation result and quality indicator scoring result in the final evaluation result, and a weighted calculation is performed according to the dimension weight of each quality dimension to calculate the comprehensive quality score of the state diagram model. The quality assessment result of the state diagram model is generated based on the final evaluation result and the comprehensive quality score.

2. The method for quality evaluation of state diagram models based on LLM according to claim 1, characterized in that, In the SCBI quality index system, the structural quality dimensions include state integrity, connectivity, state nesting depth, and interface integrity. Coverage dimensions include state coverage, migration coverage, and event coverage; The behavioral correctness dimension includes state transition accuracy, response accuracy, and simulation pass rate; Interaction performance dimensions include synchronization error rate, resource utilization, and average response latency.

3. The method for evaluating the quality of a state diagram model based on LLM according to claim 1, characterized in that, The static structural data includes a set of states, state transition relationships, and interface definition information. The dynamic behavioral data includes model response data, event interaction data, and time synchronization data. The evaluation preference constraints include priority constraints, normalization constraints, and value constraints.

4. The method for evaluating the quality of a state diagram model based on LLM according to claim 1, characterized in that, The steps for calculating the difference between all evaluation results and iteratively optimizing based on preset consistency criteria until convergence is met to obtain a unified final evaluation result include: The difference degree is calculated for all evaluation results to obtain the difference metric value, and the difference metric value is compared with the preset consistency threshold. If the difference metric is greater than the preset consistency threshold, the input information of each evaluation subject is updated and the evaluation result is regenerated through an iterative optimization mechanism until the difference metric converges to no greater than the consistency threshold, or the number of iterations reaches the preset upper limit, and a unified final evaluation result is obtained.

5. The method for evaluating the quality of a state diagram model based on LLM according to claim 4, characterized in that, The steps for calculating the degree of difference for all evaluation results to obtain the measure of difference include: The corresponding dimension weight difference value is calculated based on the average of the weight ranges of all assessment subjects in the four dimensions of structural quality, coverage, behavioral correctness and interaction performance. Calculate the weight range and score range for all quality indicators respectively, and obtain the corresponding indicator weight difference value and indicator score difference value based on the average of the weight range and score range.

6. A quality evaluation system based on an LLM-based state diagram model, characterized in that, include: The system construction module is used to build the SCBI quality index system for state diagram models. This system characterizes the model quality from four dimensions: structural quality, coverage, behavioral correctness, and interaction performance. The data acquisition module is used to acquire the static structural data of the state diagram model and the dynamic behavior data generated by the state diagram model during the co-simulation process, and to calculate the original value of each quality indicator according to the SCBI quality indicator system. The model building module is used to build a multi-evaluation subject collaborative decision-making model based on a large language model. The multi-evaluation subject includes at least two evaluation subjects with different preset evaluation roles, and each evaluation role is associated with a set of preset evaluation preference constraints. An evaluation processing module is used to generate evaluation results for each of the evaluation subjects based on the original values ​​of the quality indicators and the task context information of the state graph model, under the constraints of the multiple evaluation subjects and the evaluation preferences. The evaluation results include dimension weight allocation results, quality indicator weight allocation results, and quality indicator scoring results. The iterative optimization module is used to calculate the difference of all evaluation results and perform iterative optimization based on the preset consistency judgment conditions until the convergence condition is met, so as to obtain a unified final evaluation result. The data calculation module is used to calculate the dimension score of each quality dimension based on the quality indicator weight allocation result and quality indicator scoring result in the final evaluation result, and to perform weighted calculation based on the dimension weight of each quality dimension to calculate the comprehensive quality score of the state diagram model. The quality assessment module is used to generate the quality assessment result of the state diagram model based on the final assessment result and the comprehensive quality score.

7. The state diagram model quality evaluation system based on LLM according to claim 6, characterized in that, The iterative optimization module specifically includes: The difference calculation unit is used to calculate the difference of all evaluation results to obtain the difference measurement value, and compare the difference measurement value with the preset consistency threshold. The iterative optimization unit is used to update the input information of each evaluation subject and regenerate the evaluation result through an iterative optimization mechanism if the difference metric value is greater than a preset consistency threshold, until the difference metric value converges to no greater than the consistency threshold, or the number of iterations reaches a preset upper limit, so as to obtain a unified final evaluation result.

8. The state diagram model quality evaluation system based on LLM according to claim 7, characterized in that, The difference calculation unit is specifically used for: The corresponding dimension weight difference value is calculated based on the average of the weight ranges of all assessment subjects in the four dimensions of structural quality, coverage, behavioral correctness and interaction performance. Calculate the weight range and score range for all quality indicators respectively, and obtain the corresponding indicator weight difference value and indicator score difference value based on the average of the weight range and score range.

9. A readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the LLM-based state diagram model quality evaluation method as described in any one of claims 1 to 5.

10. A computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the LLM-based state diagram model quality evaluation method as described in any one of claims 1 to 5.

Images