Method and device for evaluating engineering training data, electronic equipment and storage medium

Abstract

The application provides an evaluation method and device for engineering training data, electronic equipment and storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: receiving multi-source heterogeneous interaction logs and uniformly converting the multi-source heterogeneous interaction logs into standardized training records; traversing the records to extract user subject operation data to generate a candidate evidence set; and obtaining a dynamic evaluation standard configuration. In the first stage, the candidate evidence is mapped to the ability dimension and the quality characteristic value is calculated, and after being sorted, truncated and compressed, an evidence mapping list is output. In the second stage, the evidence mapping list and the evaluation standard configuration are assembled into prompt words and sent to a large language model, and the dimension scores and evidence row number anchor points are returned. Finally, the row number anchor points and the training records are bidirectionally bound to generate an evaluation report. Through the truncation and compression of the first stage code layer, the application reduces the Token input quantity and response time of the large model, eliminates the calculation illusion, realizes traceability and reviewability, supports various modes such as single person, group chat and debate, and eliminates data islands.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device, and storage medium for evaluating engineering training data. Background Technology

[0002] With the rapid development of Large Language Models (LLMs) and multi-agent collaborative technologies, various intelligent interactive systems are widely used in engineering training, online education, and competency assessment scenarios. For example, learners can complete complex engineering tasks such as system architecture design and technical solution demonstration through single-person process simulation, multi-person concurrent group chat collaboration, or multi-agent discussion and debate. Competency assessment during the learning process can improve learning efficiency. Existing assessment methods include result comparison and end-to-end large model generation. Result comparison involves collecting learners' final submitted system architecture diagrams, code, test reports, or design documents, comparing them with preset reference answers or standard templates for text similarity, feature matching, or rule verification to derive an evaluation score or level. End-to-end large model generation directly inputs the learner's complete raw logs generated during the interaction process, including dialogue records, operation sequences, and agent responses, into the large language model, which then directly outputs a comprehensive score, scores for each dimension, or a natural language comment. However, in complex task scenarios deeply empowered by Artificial Intelligence Generated Content (AIGC), AIGC tools can easily generate final documents, code, or test reports. Evaluating only the final "outcome" cannot distinguish whether the learned content reflects the learner's genuine abilities developed through independent decision-making, trade-offs, and trial and error, or simply copied or modified AI-generated suggestions. In practice, the system struggles to capture the learner's genuine high-order thinking processes, such as design trade-offs under multiple constraints, reasons and justifications for rejecting AI solutions, and the evolution of their stance in multi-role games. This results in evaluation results that fail to reflect the learner's actual engineering capabilities. Furthermore, directly inputting massive amounts of raw group chat logs or multi-agent interaction logs into a large language model for processing leads to severe token consumption, high computational costs, long system response latency, and low concurrent processing capabilities. More importantly, large language models are prone to producing unrealistic results when handling deterministic logical calculation tasks such as extracting the top N valid data points, accurately calculating specific scores for a particular dimension, and truncating data chronologically. This can lead to outputs that do not conform to reality or contain logical errors, resulting in unreliable and unreproducible evaluation values. Furthermore, the underlying data structures of one-on-one dialogues and system group chats are different and fragmented, making it impossible to form a unified baseline for ability evaluation and hindering the tracking of learners' longitudinal growth across different tasks and time periods. Summary of the Invention

[0003] This invention provides a method, apparatus, electronic device, and storage medium for evaluating engineering training data, which addresses the shortcomings of existing technologies, such as ambiguous attribution of results making it impossible to distinguish between real ability and AI-generated content, wasted computing power and easy numerical illusion when large models process raw logs, and data structure fragmentation between different interaction modes making it impossible to uniformly track growth.

[0004] This invention provides a method for evaluating engineering training data, comprising: Receive multi-source heterogeneous interaction logs from at least one front-end interaction mode, and convert the multi-source heterogeneous interaction logs into a standardized training record set; each record in the training record set includes at least: an interaction subject identifier and a behavior type label; Traverse the training record set, extract user subject operation data based on the interaction subject identifier and the behavior type label, and generate a candidate evidence set; Obtain dynamic evaluation standard configuration, which includes preset engineering capability dimensions and matching keywords corresponding to each dimension; In the first stage, each piece of evidence in the candidate evidence set is mapped to one or more capability dimensions based on the matching keywords, and the quality feature value of each piece of evidence under each capability dimension is calculated. The evidence under each capability dimension is sorted and truncated and compressed according to the quality feature value, and an evidence mapping list is output. In the second stage, the evidence mapping list and the dynamic evaluation standard configuration are assembled into a standardized prompt word template and sent to the large language model. The large language model then returns an evaluation result containing scores for each capability dimension and anchor points for supporting evidence lines. The row number anchor in the evaluation result is bidirectionally bound to the training record set to generate an engineering training data evaluation report.

[0005] According to the evaluation method for engineering training data provided by the present invention, the interaction subject identifier is used to distinguish between real users, internal executive agents, and externally constrained agents; the behavior type label includes at least one of ordinary dialogue, position change, document adoption, and conflict compromise.

[0006] According to the evaluation method for engineering training data provided by the present invention, the step of extracting user-subjective operation data and generating a candidate evidence set based on the interaction subject identifier and the behavior type label includes: Based on the interaction subject identifier, system prompts, internal discussion segments between agents, and background noise are removed; Based on the behavior type label, subjective operational data reflecting the user's cognitive activities are extracted from the remaining records. The subjective operational data includes at least one of the following: questioning instructions with constraints, explicit reasons for rejection or modification of the agent generation scheme, records of compromise decisions in multi-party conflicts, and dialogue records before and after the change of stance. The extracted subjectivity operation data is used as a candidate evidence set.

[0007] According to the evaluation method for engineering training data provided by the present invention, the calculation of the quality feature value of each piece of evidence under each capability dimension includes: Calculate the quality feature value of each piece of evidence under each capability dimension it is mapped to using one or more of the following calculation methods: Textual specificity length calculation: Calculate the specificity score based on the character length or number of tokens in the evidence text; Relevance TF-IDF scoring: Calculate the TF-IDF similarity between the evidence text and the matching keywords corresponding to the mapped capability dimension; Cross-role interaction weighted score: The interaction weighted score is calculated based on the number of interacting entities involved in the evidence and the depth of the interaction; The quality characteristic value of each piece of evidence under each capability dimension can be obtained by calculating the result of any one of the above calculation methods, or by weighted fusion of the results of at least two of the above calculation methods to obtain the comprehensive quality characteristic value of each piece of evidence under each capability dimension.

[0008] According to the evaluation method for engineering training data provided by the present invention, the step of sorting and truncating / compressing the evidence under each capability dimension according to the quality feature values ​​and outputting an evidence mapping list includes: For each capability dimension, the quality feature values ​​of each piece of evidence are sorted in descending order from high to low. The arranged evidence is compressed according to the preset truncation rules: a preset number of evidence items are retained from the top of the order in each capability dimension, and the total number of characters of the evidence retained in each capability dimension does not exceed the preset character limit. The compressed and retained evidence is used as valid evidence for the corresponding capability dimension, and an evidence mapping list is generated.

[0009] According to the evaluation method for engineering training data provided by the present invention, after sorting the quality feature values ​​of each piece of evidence in descending order from high to low for each capability dimension, the method further includes: For capability dimensions that do not have any mappable evidence in the candidate evidence set, or for capability dimensions where no evidence is retained after compression, the code layer directly marks them as zero points or low confidence.

[0010] According to the evaluation method for engineering training data provided by the present invention, the step of assembling the evidence mapping list and the dynamic evaluation standard configuration into a standardized prompt word template, sending it to a large language model, and having the large language model return an evaluation result containing scores for each capability dimension and anchor points for supporting evidence line numbers includes: The valid evidence, the capability dimensions and scoring rules in the dynamic evaluation standard configuration are assembled according to the preset prompt word template format to generate standardized prompt words; The standardized prompt words are sent to the large language model, instructing the large language model to perform text understanding and logical reasoning based on the valid evidence; The evaluation results returned by the large language model are received. The evaluation results include scores for each ability dimension, summary comments for each ability dimension, and row number anchors in the evidence mapping list that support the evidence on which each score is based.

[0011] The present invention also provides an evaluation device for engineering training data, comprising: The receiving module is used to receive multi-source heterogeneous interaction logs from at least one front-end interaction mode, and to uniformly convert the multi-source heterogeneous interaction logs into a standardized training record set; each record in the training record set includes at least: an interaction subject identifier and a behavior type label; The extraction module is used to traverse the training record set, extract user subject operation data based on the interaction subject identifier and the behavior type label, and generate a candidate evidence set. The acquisition module is used to acquire the dynamic evaluation standard configuration, which includes preset engineering capability dimensions and matching keywords corresponding to each dimension; The calculation module is used in the first stage to map each piece of evidence in the candidate evidence set to one or more capability dimensions based on the matching keywords, calculate the quality feature value of each piece of evidence under each capability dimension, sort and truncate and compress the evidence under each capability dimension according to the quality feature value, and output the evidence mapping list. The sending module is used in the second stage to assemble the evidence mapping list and the dynamic evaluation standard configuration into a standardized prompt word template, and send it to the large language model. The large language model returns the evaluation results containing the scores of each capability dimension and the anchor points of the supporting evidence line numbers. The generation module is used to bidirectionally bind the row number anchor point in the evaluation result to the training record set to generate an engineering training data evaluation report.

[0012] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the evaluation method for engineering training data as described in any of the preceding claims.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the evaluation method for engineering training data described in any of the preceding claims.

[0014] The present invention provides an evaluation method, apparatus, electronic device, and storage medium for engineering training data. This involves receiving multi-source heterogeneous interaction logs from at least one front-end interaction mode and uniformly converting these logs into a standardized training record set. Each record in the training record set includes at least an interaction subject identifier and a behavior type label. The method iterates through the training record set, extracting user-subjective operation data based on the interaction subject identifier and the behavior type label to generate a candidate evidence set. It then obtains a dynamic evaluation standard configuration, which includes preset engineering capability dimensions and corresponding matching keywords for each dimension. In a first stage, each piece of evidence in the candidate evidence set is mapped to one or more capability dimensions based on the matching keywords. The quality feature value of each piece of evidence under each capability dimension is calculated, and the evidence under each capability dimension is sorted, truncated, and compressed according to the quality feature value, outputting an evidence mapping list. In a second stage, the evidence is mapped... The list of targets and the dynamic evaluation standard configuration are assembled into a standardized prompt word template and sent to the large language model. The large language model returns an evaluation result containing scores for each capability dimension and anchor points for supporting evidence line numbers. The anchor points for line numbers in the evaluation result are bidirectionally bound to the training record set to generate an engineering training data evaluation report. Through the truncation and compression of the code layer in the first stage, the amount of token input passed to the large language model is significantly reduced, the system response time is greatly shortened, and the mathematical errors and illusions that are prone to occur when the large language model calculates dimension scores are completely eliminated. Any capability score output in the evaluation report can be accurately traced back to the specific dialogue line number in the interaction process, realizing the absolute traceability and verifiability of the evaluation process. It can serve as a unified foundation to support system forms of different complexities, such as single-person training, multi-person concurrent collaboration, and multi-agent automatic debate, eliminating the problem of data silos between different systems and supporting cross-task and cross-cycle vertical capability tracking. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 This is a flowchart of the evaluation method for engineering training data provided in an embodiment of the present invention; Figure 2 This is a functional structure diagram of the evaluation device for engineering training data provided in an embodiment of the present invention; Figure 3 This is a functional structure diagram of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0018] Figure 1 A flowchart of the evaluation method for engineering training data provided in the embodiments of the present invention is shown below. Figure 1 As shown, the evaluation method for engineering training data provided in this embodiment of the invention includes: Step 101: Receive multi-source heterogeneous interaction logs from at least one front-end interaction mode, and convert the multi-source heterogeneous interaction logs into a standardized training record set; each record in the training record set includes at least: interaction subject identifier and behavior type label; In this embodiment of the invention, the engineering training data includes data such as interaction logs, behavior records, and dialogue content generated by the user during the engineering training process.

[0019] The embodiments of the present invention employ a unified data receiving interface (such as a RESTful API or message queue) to receive raw interaction logs from different front-end business modules, including but not limited to single-person process simulation, multi-person concurrent group chat, and multi-agent seminar hall.

[0020] The evaluation engine converts the aforementioned heterogeneous interaction logs into a standardized training record array. Each record includes, but is not limited to: interaction subject identifier (distinguishing between real users, internal executing agents, and externally constrained agents), micro-scene identifier (Micro-scene ID), behavior type label (e.g., ordinary dialogue, change of stance, document adoption, conflict compromise), timestamp, and payload data. This embodiment of the invention does not handle group chat concurrency logic or discussion rules; it only receives structured results, completely decoupling front-end business logic. This embodiment of the invention supports seamless horizontal scaling of the system platform. As a unified foundation, it successfully supports system configurations of varying complexity, such as single-person training, multi-person concurrent collaboration, and multi-agent automatic debate, eliminating the problem of data silos between different modules.

[0021] Step 102: Traverse the training record set, extract user subjectivity operation data based on the interaction subject identifier and the behavior type label, and generate a candidate evidence set; In this embodiment of the invention, the interaction subject identifier is used to distinguish between real users, internal execution agents, and externally constrained agents; the behavior type label includes at least one of ordinary dialogue, position change, document adoption, and conflict compromise.

[0022] Step 103: Obtain the dynamic evaluation standard configuration, which includes preset engineering capability dimensions and matching keywords corresponding to each dimension; Step 104: In the first stage, each piece of evidence in the candidate evidence set is mapped to one or more capability dimensions based on the matching keywords, and the quality feature value of each piece of evidence under each capability dimension is calculated. The evidence under each capability dimension is sorted and truncated and compressed according to the quality feature value, and an evidence mapping list is output. Step 105: In the second stage, the evidence mapping list and the dynamic evaluation standard configuration are assembled into a standardized prompt word template and sent to the large language model. The large language model returns the evaluation results containing the scores of each capability dimension and the anchor points of the supporting evidence line numbers. Step 106: Bind the row number anchor point in the evaluation result to the training record set in both directions to generate an engineering training data evaluation report.

[0023] In this embodiment of the invention, any capability score output in the evaluation report can be accurately traced back to the specific dialogue line number and micro-scene in the interaction process, realizing the absolute traceability and verifiability of the evaluation process.

[0024] In complex tasks deeply empowered by generative artificial intelligence (AIGC), such as engineering lifecycle simulation, traditional methods for evaluating engineering training data suffer from the following serious shortcomings: (1) The “black box effect” leads to ambiguity in the attribution of results: AIGC can easily write the final document. Only evaluating the “result” makes it impossible to distinguish whether the result is the result of the user’s independent decision-making and weighing or a direct copy of the AI’s suggestions. The system has difficulty capturing the user’s true high-order thinking process, such as the weighing of multiple constraints, the reasons for rejecting the AI ​​solution, and the evolution of the position in multi-role games.

[0025] (2) Waste of computing power in large models and defects in deterministic numerical calculation (illusion): Directly inputting massive amounts of raw group chat logs or multi-agent interaction logs into large models will lead to serious token consumption, resulting in high costs and long delays. More fatally, large language models are prone to illusions when performing deterministic logical calculations such as "extracting the first N valid data" and "precisely calculating the specific score of a certain dimension", resulting in unreliable numerical evaluation results.

[0026] (3) Lack of a decoupled evaluation foundation makes it difficult to reuse across modes: Existing evaluation logic is usually strongly bound to specific interactive business (such as one-person dialogue and multi-person chat). When the system expands to different interactive modes such as "group chat collaboration" and "multi-agent discussion and debate", the underlying data structure is broken, making it impossible to form a unified capability baseline and support vertical growth tracking across time periods.

[0027] The engineering training data evaluation method provided in this embodiment of the invention receives multi-source heterogeneous interaction logs from at least one front-end interaction mode and converts the multi-source heterogeneous interaction logs into a standardized training record set. Each record in the training record set includes at least: an interaction subject identifier and a behavior type label. The method iterates through the training record set, extracts user subject-based operation data based on the interaction subject identifier and the behavior type label, and generates a candidate evidence set. It then obtains a dynamic evaluation standard configuration, which includes preset engineering capability dimensions and matching keywords corresponding to each dimension. In the first stage, each piece of evidence in the candidate evidence set is mapped to one or more capability dimensions based on the matching keywords, and the quality feature value of each piece of evidence under each capability dimension is calculated. The evidence under each capability dimension is sorted and truncated and compressed according to the quality feature values, and an evidence mapping list is output. In the second stage, the evidence mapping list is compared with the... The dynamic evaluation standard configuration is assembled into a standardized prompt word template and sent to the large language model. The large language model returns an evaluation result containing scores for each capability dimension and anchor points for supporting evidence line numbers. The anchor points in the evaluation result are bidirectionally bound to the training record set to generate an engineering training data evaluation report. Through the truncation and compression of the code layer in the first stage, the amount of token input passed to the large language model is significantly reduced, the system response time is greatly shortened, and the mathematical errors and illusions that are prone to occur when the large language model calculates dimension scores are completely eliminated. Any capability score output in the evaluation report can be accurately traced back to the specific dialogue line number in the interaction process, realizing the absolute traceability and reproducibility of the evaluation process. It can serve as a unified platform to support system forms of different complexities, such as single-person training, multi-person concurrent collaboration, and multi-agent automatic debate, eliminating the problem of data silos between different systems and supporting cross-task and cross-cycle vertical capability tracking.

[0028] Based on any of the above embodiments, the step of extracting user-subjective operation data and generating a candidate evidence set according to the interaction subject identifier and the behavior type tag includes: Step 201: Based on the interaction subject identifier, remove system prompts, internal discussion segments between agents, and background noise; Step 202: Based on the behavior type label, extract the subjective operation data reflecting the user's cognitive activities from the remaining records. The subjective operation data includes at least one of the following: question instructions with constraints, explicit reasons for rejection or modification of the intelligent agent generation scheme, records of compromise decisions in multi-party conflicts, and dialogue records before and after the change of stance. Step 203: Use the extracted subjectivity operation data as a candidate evidence set.

[0029] Based on any of the above embodiments, calculating the quality feature value of each piece of evidence under each capability dimension includes: Calculate the quality feature value of each piece of evidence under each capability dimension it is mapped to using one or more of the following calculation methods: Textual specificity length calculation: Calculate the specificity score based on the character length or number of tokens in the evidence text; Relevance TF-IDF scoring: Calculate the TF-IDF similarity between the evidence text and the matching keywords corresponding to the mapped capability dimension; Cross-role interaction weighted score: The interaction weighted score is calculated based on the number of interacting entities involved in the evidence and the depth of the interaction; The quality characteristic value of each piece of evidence under each capability dimension can be obtained by calculating the result of any one of the above calculation methods, or by weighted fusion of the results of at least two of the above calculation methods to obtain the comprehensive quality characteristic value of each piece of evidence under each capability dimension.

[0030] Based on any of the above embodiments, the step of sorting and truncating / compressing the evidence according to the quality feature values ​​for each capability dimension, and outputting an evidence mapping list, includes: Step 301: Sort the quality feature values ​​of each piece of evidence in descending order from high to low for each capability dimension; Step 302: Compress the arranged evidence according to the preset truncation rules: retain the preset number of evidence items that are ranked first in each capability dimension, and the total number of characters of the evidence retained in each capability dimension does not exceed the preset character limit; Step 303: Use the compressed and retained evidence as valid evidence for the corresponding capability dimension to generate an evidence mapping list.

[0031] This invention parses the externally input "Dynamic Evaluation Standard Configuration JSON," which contains preset engineering capability dimensions, such as system design and compliance verification. It iterates through the candidate evidence set and maps each candidate piece of evidence to one or more capability dimensions using regular expression matching and a bag-of-words model. The code logic calculates quality feature values ​​for each mapped piece of evidence, including: text specificity length calculation, TF-IDF relevance score to the mapped dimension's keywords, and cross-role interaction weighted score. The evidence is sorted in descending order of quality feature values, and the evidence for each dimension is truncated to a preset character limit (e.g., retaining the top-3 evidence for a single dimension, with a total not exceeding 1000 characters), eliminating low-value evidence. A purely structured "Evidence Mapping List" is output. For dimensions where no evidence can be extracted, the code layer directly marks them as zero points or "low credibility."

[0032] Through the first-stage code layer truncation and compression, the amount of token input passed to the large model was reduced by about 80%-90%, the system response time was shortened, and mathematical errors in calculating dimensional scores were completely eliminated.

[0033] In this embodiment of the invention, after sorting the quality feature values ​​of each piece of evidence in descending order for each capability dimension, the method further includes: For capability dimensions that do not have any mappable evidence in the candidate evidence set, or for capability dimensions where no evidence is retained after compression, the code layer directly marks them as zero points or low confidence.

[0034] Based on any of the above embodiments, the step of assembling the evidence mapping list and the dynamic evaluation standard configuration into a standardized prompt word template, sending it to the large language model, and having the large language model return an evaluation result containing scores for each capability dimension and anchor points for supporting evidence line numbers includes: Step 401: Assemble the valid evidence, the capability dimensions and scoring rules in the dynamic evaluation standard configuration according to the preset prompt word template format to generate standardized prompt words; Step 402: Send the standardized prompt words to the large language model, instructing the large language model to perform text understanding and logical reasoning based on the valid evidence; Step 403: Receive the evaluation results returned by the large language model. The evaluation results include scores for each capability dimension, summary comments for each capability dimension, and row number anchors in the evidence mapping list that support each score.

[0035] The "Evidence Mapping List" and the "Dynamic Evaluation Standard Configuration JSON" are assembled into a standardized prompt word template and sent to the large language model. The large language model performs text understanding and logical reasoning based on the condensed factual evidence, outputting: dimension scores based on evidence quality assessment, summary comments, and line number anchor references to supporting evidence. The engine receives the JSON results returned by the large model, binds them bidirectionally with the original training records of each step using line numbers, and generates a "Verifiable Capability Evaluation Report" that allows for clickable backtracking.

[0036] This invention proposes an algorithm for determining capability stages based on a "total evidence pool" and a "historical highest score." The "total evidence pool" refers to an evidence pool that each user possesses, which only increases and never decreases. Valid process evidence extracted from each training task is added to that user's evidence pool and will never be deleted due to a low score on a particular task. The "historical highest score" refers to the system recording the highest overall score a user has ever achieved in history for each capability dimension, rather than only considering the score of the current task, thus overcoming the problem of score oscillation in single-task evaluations. This invention retrieves multiple historical evaluation reports from the same user, employing exact matching of the same dimension and mapping algorithms for similar dimensions. New evidence generated in the current evaluation is added to the user's "cumulative evidence pool" (reflecting the principle that process evidence only increases and never decreases); and the user's capability advancement status is recalculated based on the historical "highest overall score" for that dimension.

[0037] The system obtains the evaluation results of all users in the same batch, calculates the average score and standard deviation of the class dimension directly by the code layer, filters out abnormal objects with a single dimension below the set threshold, and outputs a group learning dashboard and peer learning recommendation pairs that complement each other's strengths and weaknesses.

[0038] This invention overturns the industry practice of directly using large models to "calculate scores," clearly defining boundaries. The code layer is responsible for deterministic extraction, sorting, filtering, and dimensionality reduction; the large model is only responsible for semantic induction based on high signal-to-noise ratio summaries. This solves the problems of wasted computational power and numerical illusion in LLM. The evaluation object shifts from the final generated document appearance to higher-order behaviors such as "weighing, rejecting, and questioning" by users under multiple constraints. By capturing cognitive conflict points through labeled feature words, it breaks the evaluation black box problem brought about by AIGC writing services.

[0039] The following describes the specific content of this invention in different scenarios: Example 1: Evaluation and processing of the "Dynamic Debate in the Seminar" includes: Scenario Description: In a debate seminar room containing five agents with different perspectives, such as a technology expert, an ethics expert, an economic analyst, a legal advisor, and a project manager, a user completed a 30-minute interactive debate, engaging in multiple rounds of questioning and response around a certain technical solution.

[0040] Execution Process: After the front-end seminar engine completes dynamic speech scheduling, it encapsulates the interaction logs into the Records Array format required by this embodiment of the invention, which is specifically marked with a position change tag. Upon recognizing this tag, it automatically extracts the dialogue fragments before and after the user's position change and uses them as high-weight process evidence; the code layer assigns high-quality feature values ​​of the "critical thinking" ability dimension to this evidence; in the second stage, the large language model generates the following comment based on this evidence: "When faced with questioning from an ethics expert agent, the user proactively adjusted the original technical solution, demonstrating good openness and reflective ability, and possessing excellent engineering ethics judgment." This example demonstrates the application capability of the present invention in complex debate scenarios. By capturing and weighting the higher-order cognitive behavior of "position shift", it effectively identifies the user's thinking evolution process in the collision of multiple viewpoints, and solves the problem that traditional evaluation methods cannot capture the user's critical thinking ability.

[0041] Example 2: Integrating group evaluations into "multi-person collaborative chats," specifically including: Scenario Description: Four users and three constrained agents collaborate in a concurrent group chat environment to complete a system architecture design task. Users engage in multiple rounds of discussions, solution comparisons, and decisions, as well as between users and agents.

[0042] Execution Process: After the group chat scheduling engine completes concurrent message processing, it transmits the merged interaction logs to the evaluation system of this invention according to user identifiers. The system initiates independent evaluation pipelines for the four users in parallel, generating individual capability evaluation reports for each. Subsequently, the code layer automatically aggregates the evaluation results of the four users and performs group analysis: it finds that user A scored the highest in the "System Design" dimension, while user B scored less than 2.0 in this dimension. Based on this, the system automatically generates a mutual assistance strategy: "It is recommended that user A serve as user B's technical mentor in the next stage to promote the overall improvement of the team's capabilities." This example demonstrates the group evaluation and intelligent grouping capabilities of the present invention in multi-user collaborative scenarios. It can not only conduct independent ability assessments for each user, but also achieve peer learning recommendations through aggregation analysis, providing data-driven teaching intervention strategies for team collaboration training.

[0043] Example 3: Receiving dynamic configurations for "AI Scene Orchestration", specifically including: Scenario Description: Teachers use an external AI scenario arrangement system to automatically generate a set of customized evaluation criteria for a specific engineering training task (such as "smart home system design").

[0044] Execution Process: The evaluation criteria are output in JSON format, containing seven custom capability dimensions (such as "requirements analysis capability," "module interface design capability," and "power consumption optimization capability") and corresponding matching keywords for each dimension. This JSON is passed as input parameters to the evaluation engine of this invention. After receiving this configuration, the code layer automatically and dynamically reconstructs the regular expressions and the bag-of-words model, accurately mapping user behavioral evidence to these seven new dimensions without modifying the underlying code of the evaluation engine.

[0045] This example demonstrates the dynamic configurability of the evaluation criteria of this invention. Through an external JSON injection mechanism, the evaluation engine can flexibly adapt to the evaluation requirements of different tasks and courses, achieving complete decoupling between the evaluation logic and business scenarios, and greatly improving the scalability and reusability of the system.

[0046] The evaluation method for engineering training data provided in this invention establishes a general-purpose underlying evaluation engine independent of specific front-end interaction modes and decoupled from individual training, group chat collaboration, and discussion / debate. By constructing a three-stage pipeline of "heterogeneous data unified bus → code-level precise evidence extraction and compression → large language model semantic reasoning," it significantly reduces the computational cost of large model inference while solving the problem of capability assessment distortion caused by AIGC (AI garbage collection) writing services. It also provides a unified and traceable evaluation interface for various complex collaborative systems at the upper layer. The evaluation method provided in this invention does not concern itself with whether the front-end is individual, group chat, or debate / discussion; as long as the front-end inputs the interaction according to the training record set format (stage-micro-scene-role-position tag), and the evaluation criteria are dynamically injected through external JSON, it connects to the AI ​​orchestration system, overcoming the score oscillation problem of single-task evaluation.

[0047] Figure 2 This is a schematic diagram of the structure of the evaluation device for engineering training data provided in an embodiment of the present invention, as shown below. Figure 2 As shown, the evaluation device for engineering training data provided in this embodiment of the invention includes: The receiving module 201 is used to receive multi-source heterogeneous interaction logs from at least one front-end interaction mode, and to uniformly convert the multi-source heterogeneous interaction logs into a standardized training record set; each record in the training record set includes at least: an interaction subject identifier and a behavior type label; Extraction module 202 is used to traverse the training record set, extract user subject operation data according to the interaction subject identifier and the behavior type label, and generate a candidate evidence set; The acquisition module 203 is used to acquire the dynamic evaluation standard configuration, which includes preset engineering capability dimensions and matching keywords corresponding to each dimension; The calculation module 204 is used in the first stage to map each piece of evidence in the candidate evidence set to one or more capability dimensions based on the matching keywords, calculate the quality feature value of each piece of evidence under each capability dimension, sort and truncate and compress the evidence under each capability dimension according to the quality feature value, and output the evidence mapping list. The sending module 205 is used in the second stage to assemble the evidence mapping list and the dynamic evaluation standard configuration into a standardized prompt word template, and send it to the large language model, which then returns the evaluation results containing scores for each capability dimension and anchor points for supporting evidence lines. The generation module 206 is used to bidirectionally bind the row number anchor point in the evaluation result to the training record set to generate an engineering training data evaluation report.

[0048] The engineering training data evaluation device provided in this embodiment of the invention receives multi-source heterogeneous interaction logs from at least one front-end interaction mode and converts the multi-source heterogeneous interaction logs into a standardized training record set. Each record in the training record set includes at least: an interaction subject identifier and a behavior type label. The device iterates through the training record set, extracts user subject-based operation data based on the interaction subject identifier and the behavior type label, and generates a candidate evidence set. It obtains a dynamic evaluation standard configuration, which includes preset engineering capability dimensions and matching keywords corresponding to each dimension. In a first stage, each piece of evidence in the candidate evidence set is mapped to one or more capability dimensions based on the matching keywords, and the quality feature value of each piece of evidence under each capability dimension is calculated. The evidence under each capability dimension is sorted and truncated and compressed according to the quality feature values, and an evidence mapping list is output. In a second stage, the evidence mapping list is compared with the... The dynamic evaluation standard configuration is assembled into a standardized prompt word template and sent to the large language model. The large language model returns an evaluation result containing scores for each capability dimension and anchor points for supporting evidence line numbers. The anchor points in the evaluation result are bidirectionally bound to the training record set to generate an engineering training data evaluation report. Through the truncation and compression of the code layer in the first stage, the amount of token input passed to the large language model is significantly reduced, the system response time is greatly shortened, and the mathematical errors and illusions that are prone to occur when the large language model calculates dimension scores are completely eliminated. Any capability score output in the evaluation report can be accurately traced back to the specific dialogue line number in the interaction process, realizing the absolute traceability and reproducibility of the evaluation process. It can serve as a unified platform to support system forms of different complexities, such as single-person training, multi-person concurrent collaboration, and multi-agent automatic debate, eliminating the problem of data silos between different systems and supporting cross-task and cross-cycle vertical capability tracking.

[0049] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The memory 330 includes computer programs, an operating system, and acquired data. The processor 310 can call logical instructions in the memory 330 to execute an evaluation method for engineering training data. This method includes: receiving multi-source heterogeneous interaction logs from at least one front-end interaction mode, and uniformly converting the multi-source heterogeneous interaction logs into a standardized training record set; each record in the training record set includes at least: an interaction subject identifier and a behavior type label; traversing the training record set, extracting user subjectivity operation data based on the interaction subject identifier and the behavior type label, and generating a candidate evidence set; and obtaining a dynamic evaluation standard configuration, which includes preset engineering capability dimensions and various... The matching keywords correspond to the dimensions; in the first stage, each piece of evidence in the candidate evidence set is mapped to one or more capability dimensions based on the matching keywords, and the quality feature value of each piece of evidence under each capability dimension is calculated. The evidence under each capability dimension is sorted and truncated and compressed according to the quality feature value, and an evidence mapping list is output; in the second stage, the evidence mapping list and the dynamic evaluation standard configuration are assembled into a standardized prompt word template and sent to the large language model. The large language model returns an evaluation result containing the scores of each capability dimension and the row number anchors of supporting evidence; the row number anchors in the evaluation result are bidirectionally bound to the training record set to generate an engineering training data evaluation report.

[0050] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0051] On the other hand, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements an evaluation method for performing the engineering training data provided by the methods described above. This method includes: receiving multi-source heterogeneous interaction logs from at least one front-end interaction mode, and uniformly converting the multi-source heterogeneous interaction logs into a standardized training record set; each record in the training record set includes at least: an interaction subject identifier and a behavior type label; traversing the training record set, extracting user subjectivity operation data based on the interaction subject identifier and the behavior type label, and generating a candidate evidence set; and obtaining a dynamic evaluation standard configuration, the dynamic evaluation standard configuration including preset engineering performance... The system identifies capability dimensions and their corresponding matching keywords. In the first stage, each piece of evidence in the candidate evidence set is mapped to one or more capability dimensions based on the matching keywords. The quality feature value of each piece of evidence under each capability dimension is calculated, and the evidence under each capability dimension is sorted, truncated, and compressed according to the quality feature value, outputting an evidence mapping list. In the second stage, the evidence mapping list and the dynamic evaluation standard configuration are assembled into a standardized prompt word template and sent to a large language model. The large language model returns an evaluation result containing scores for each capability dimension and row number anchors for supporting evidence. The row number anchors in the evaluation result are bidirectionally bound to the training record set to generate an engineering training data evaluation report.

[0052] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0053] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0054] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for evaluating engineering training data, characterized in that, include: Receive multi-source heterogeneous interaction logs from at least one front-end interaction mode, and convert the multi-source heterogeneous interaction logs into a standardized training record set; Each record in the training record set includes at least: an interaction subject identifier and a behavior type label; Traverse the training record set, extract user subject operation data based on the interaction subject identifier and the behavior type label, and generate a candidate evidence set; Obtain dynamic evaluation standard configuration, which includes preset engineering capability dimensions and matching keywords corresponding to each dimension; In the first stage, each piece of evidence in the candidate evidence set is mapped to one or more capability dimensions based on the matching keywords, and the quality feature value of each piece of evidence under each capability dimension is calculated. The evidence under each capability dimension is sorted and truncated and compressed according to the quality feature value, and an evidence mapping list is output. In the second stage, the evidence mapping list and the dynamic evaluation standard configuration are assembled into a standardized prompt word template and sent to the large language model. The large language model then returns an evaluation result containing scores for each capability dimension and anchor points for supporting evidence lines. The row number anchor in the evaluation result is bidirectionally bound to the training record set to generate an engineering training data evaluation report.

2. The evaluation method for engineering training data according to claim 1, characterized in that, The interaction subject identifier is used to distinguish between real users, internal execution agents, and externally constrained agents; the behavior type label includes at least one of ordinary dialogue, position change, document adoption, and conflict compromise.

3. The evaluation method for engineering training data according to claim 1, characterized in that, The step of extracting user-subjective operation data and generating a candidate evidence set based on the interaction subject identifier and the behavior type label includes: Based on the interaction subject identifier, system prompts, internal discussion segments between agents, and background noise are removed; Based on the behavior type label, subjective operational data reflecting the user's cognitive activities are extracted from the remaining records. The subjective operational data includes at least one of the following: questioning instructions with constraints, explicit reasons for rejection or modification of the agent generation scheme, records of compromise decisions in multi-party conflicts, and dialogue records before and after the change of stance. The extracted subjectivity operation data is used as a candidate evidence set.

4. The evaluation method for engineering training data according to claim 1, characterized in that, The calculation of the quality characteristic value of each piece of evidence under each capability dimension includes: Calculate the quality feature value of each piece of evidence under each capability dimension it is mapped to using one or more of the following calculation methods: Textual specificity length calculation: Calculate the specificity score based on the character length or number of tokens in the evidence text; Relevance TF-IDF scoring: Calculate the TF-IDF similarity between the evidence text and the matching keywords corresponding to the mapped capability dimension; Cross-role interaction weighted score: The interaction weighted score is calculated based on the number of interacting entities involved in the evidence and the depth of the interaction; The quality characteristic value of each piece of evidence under each capability dimension can be obtained by calculating the result of any one of the above calculation methods, or by weighted fusion of the results of at least two of the above calculation methods to obtain the comprehensive quality characteristic value of each piece of evidence under each capability dimension.

5. The evaluation method for engineering training data according to claim 1, characterized in that, The process of sorting and truncating / compressing the evidence according to the quality feature values ​​for each capability dimension, and outputting an evidence mapping list, includes: For each capability dimension, the quality feature values ​​of each piece of evidence are sorted in descending order from high to low. The arranged evidence is compressed according to the preset truncation rules: a preset number of evidence items are retained from the top of the order in each capability dimension, and the total number of characters of the evidence retained in each capability dimension does not exceed the preset character limit. The compressed and retained evidence is used as valid evidence for the corresponding capability dimension, and an evidence mapping list is generated.

6. The evaluation method for engineering training data according to claim 5, characterized in that, After sorting the quality feature values ​​of each piece of evidence in descending order for each capability dimension, the method further includes: For capability dimensions that do not have any mappable evidence in the candidate evidence set, or for capability dimensions where no evidence is retained after compression, the code layer directly marks them as zero points or low confidence.

7. The evaluation method for engineering training data according to claim 6, characterized in that, The process involves assembling the evidence mapping list and the dynamic evaluation standard configuration into a standardized prompt word template, which is then sent to the large language model. The large language model returns an evaluation result containing scores for each capability dimension and anchor points for supporting evidence line numbers, including: The valid evidence, the capability dimensions and scoring rules in the dynamic evaluation standard configuration are assembled according to the preset prompt word template format to generate standardized prompt words; The standardized prompt words are sent to the large language model, instructing the large language model to perform text understanding and logical reasoning based on the valid evidence; The evaluation results returned by the large language model are received. The evaluation results include scores for each ability dimension, summary comments for each ability dimension, and row number anchors in the evidence mapping list that support the evidence on which each score is based.

8. An evaluation device for engineering training data, characterized in that, include: The receiving module is used to receive multi-source heterogeneous interaction logs from at least one front-end interaction mode, and to uniformly convert the multi-source heterogeneous interaction logs into a standardized training record set. Each record in the training record set includes at least: an interaction subject identifier and a behavior type label; The extraction module is used to traverse the training record set, extract user subject operation data based on the interaction subject identifier and the behavior type label, and generate a candidate evidence set. The acquisition module is used to acquire the dynamic evaluation standard configuration, which includes preset engineering capability dimensions and matching keywords corresponding to each dimension; The calculation module is used in the first stage to map each piece of evidence in the candidate evidence set to one or more capability dimensions based on the matching keywords, calculate the quality feature value of each piece of evidence under each capability dimension, sort and truncate and compress the evidence under each capability dimension according to the quality feature value, and output the evidence mapping list. The sending module is used in the second stage to assemble the evidence mapping list and the dynamic evaluation standard configuration into a standardized prompt word template, and send it to the large language model. The large language model returns the evaluation results containing the scores of each capability dimension and the anchor points of the supporting evidence line numbers. The generation module is used to bidirectionally bind the row number anchor point in the evaluation result to the training record set to generate an engineering training data evaluation report.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the evaluation method for engineering training data as described in any one of claims 1 to 7.

10. A non-transitory readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the evaluation method for engineering training data as described in any one of claims 1 to 7.

Images