Training evaluation method, device, equipment and program product

Abstract

The invention discloses a training evaluation method, device, equipment and program product. According to the scheme, the method comprises the steps of performing semantic analysis on training requirement description input by a user based on a natural language processing technology, and constructing an evaluation dimension system comprising a plurality of evaluation dimensions and logic relationships among the evaluation dimensions; for each evaluation dimension in the evaluation dimension system, selecting an adaptive target evaluation model from a plurality of corresponding candidate evaluation models to generate a model scheduling strategy for mapping each evaluation dimension to the corresponding target evaluation model; wherein the candidate evaluation model is used for executing evaluation calculation of the evaluation dimension to which the candidate evaluation model belongs. And according to the model scheduling strategy, calling a corresponding target evaluation model to evaluate the training process data, and obtaining an evaluation result corresponding to each evaluation dimension. And according to the logic relationship in the evaluation dimension system, integrating the evaluation result of each evaluation dimension, and generating a target evaluation result for the training process data.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to a training assessment method, apparatus, device, and program product. Background Technology

[0002] Current mainstream training evaluation systems typically employ static dimensional analysis methods, relying on predefined sets of feature dimensions and fixed weight allocation mechanisms to make evaluation decisions. While this method can achieve basic evaluation functions, its fixed dimensional space and lack of dynamic expansion and correlation capabilities make it difficult to adapt to the differentiated needs of different training scenarios, and it also cannot automatically generate or adjust evaluation dimensions based on real-time semantic understanding during system operation. Summary of the Invention

[0003] This application proposes a training assessment method, apparatus, equipment, and program product, aiming to solve the problems of poor assessment flexibility, low resource utilization, and insufficient result accuracy caused by rigid assessment dimensions, inefficient model scheduling, and a single result fusion mechanism in existing technologies. Accordingly, the technical solution of this application is as follows: Firstly, a training assessment method is provided, including: Based on natural language processing technology, semantic parsing is performed on the training needs description input by users to construct an evaluation dimension system that includes multiple evaluation dimensions and logical relationships between each evaluation dimension; For each evaluation dimension in the evaluation dimension system, a suitable target evaluation model is selected from multiple candidate evaluation models to generate a model scheduling strategy that maps each evaluation dimension to the corresponding target evaluation model; wherein, the candidate evaluation model is used to perform the evaluation calculation of its respective evaluation dimension; According to the model scheduling strategy, the corresponding target evaluation model is invoked to evaluate the training process data, and the evaluation results corresponding to each evaluation dimension are obtained. Based on the logical relationships within the aforementioned evaluation dimension system, and by synthesizing the evaluation results of each evaluation dimension, a target evaluation result is generated for the training process data.

[0004] Secondly, a training assessment device is provided, comprising: The system construction module uses natural language processing technology to semantically parse the training needs descriptions input by users and constructs an evaluation dimension system that includes multiple evaluation dimensions and the logical relationships between them. The model selection module selects a suitable target evaluation model from multiple candidate evaluation models for each evaluation dimension in the evaluation dimension system, so as to generate a model scheduling strategy that maps each evaluation dimension to the corresponding target evaluation model; wherein, the candidate evaluation model is used to perform the evaluation calculation of its respective evaluation dimension; The evaluation execution module, according to the model scheduling strategy, calls the corresponding target evaluation model to evaluate the training process data and obtain the evaluation results corresponding to each evaluation dimension. The evaluation processing module, based on the logical relationships in the evaluation dimension system, integrates the evaluation results of each evaluation dimension to generate a target evaluation result for the training process data.

[0005] Thirdly, embodiments of this application provide an electronic device, including: a processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the method described in the first aspect.

[0006] Fourthly, a computer program product is provided, the computer program product including a computer-readable storage medium storing a computer program operable to cause a computer to perform the method described in the first aspect.

[0007] This application first uses natural language processing technology to semantically parse the training requirement description input by the user, automatically constructing a dynamic evaluation dimension system containing multiple evaluation dimensions and their logical relationships. This fundamentally breaks the limitation of relying on a predefined static dimension library, realizing the dynamic response and expansion of evaluation dimensions to complex semantic requirements. Subsequently, for each evaluation dimension in the evaluation dimension system, an intelligent decision-making module integrating reinforcement learning and attention mechanisms dynamically selects the most suitable target evaluation model from multiple candidate evaluation models and generates an optimal model scheduling strategy. This model scheduling strategy can comprehensively consider evaluation accuracy, response latency, and real-time system load, thereby overcoming the problems of simple binding between models and dimensions and rigid resource allocation. Then, according to the model scheduling strategy, the corresponding target evaluation model is called to perform parallel or cascade evaluation of the multimodal training process data, and a confidence calibration mechanism is introduced to ensure the reliability of the evaluation results of each dimension. Finally, based on the logical relationships inherent in the evaluation dimension system, the calibrated evaluation results of each dimension are comprehensively analyzed to generate accurate, interpretable, and scenario-appropriate final evaluation results. Through the synergy and progression of the above-mentioned steps, this solution achieves a comprehensive improvement in the evaluation system in terms of dimensional scalability, intelligent scheduling, and adaptive result fusion. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 This is a schematic diagram of the first flowchart of the training and evaluation method according to an embodiment of this application.

[0010] Figure 2 This is a schematic diagram of the second process of the training and evaluation method according to an embodiment of this application.

[0011] Figure 3 This is a schematic diagram of the structure of the training and evaluation device according to an embodiment of this application.

[0012] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0013] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0014] Current mainstream training assessment systems typically employ static dimensional analysis methods, relying on predefined sets of feature dimensions and fixed weight allocation mechanisms to make assessment decisions. While this method can achieve basic evaluation functions, its rigid dimensional space and lack of dynamic expansion and correlation capabilities make it difficult to adapt to the differentiated needs of different training scenarios, and it cannot automatically generate or adjust assessment dimensions based on real-time semantic understanding during system operation. Furthermore, existing solutions employing advanced technologies such as multimodal fusion still generally suffer from several key problems: First, assessment dimensions are rigidly bound to the model, requiring manual configuration and development for adding or adjusting dimensions, resulting in slow response and high maintenance costs; second, model scheduling strategies are simplistic, failing to fully consider real-time computational load and model performance differences, leading to low resource utilization efficiency; and third, result fusion mechanisms are rigid, often using fixed weighting methods, failing to dynamically calibrate based on the confidence level of the model output, affecting the scenario adaptability and accuracy of the assessment results.

[0015] In view of this, embodiments of this application propose a training assessment method, apparatus, equipment, and program product, aiming to solve the problems of poor assessment flexibility, low resource utilization, and insufficient result accuracy caused by rigid assessment dimensions, inefficient model scheduling, and a single result fusion mechanism in the prior art. The technical solutions provided by the various embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0016] One embodiment of this application provides a training assessment method. Figure 1 This is a flowchart illustrating the training assessment method, including: S101 uses natural language processing technology to semantically parse the training needs description input by the user and constructs an evaluation dimension system that includes multiple evaluation dimensions and the logical relationships between them.

[0017] The reason this embodiment uses natural language processing technology to semantically parse the user's input training needs and constructs an evaluation dimension system containing multiple evaluation dimensions and logical relationships between them is fundamentally to overcome the limitations of traditional evaluation systems that rely on fixed, predefined sets of dimensions, thus achieving flexible generation and dynamic adaptation of the evaluation framework. In traditional methods, evaluation dimensions need to be manually set in advance and are difficult to adjust, failing to accurately respond to the complex and ever-changing evaluation intentions in different training scenarios. Through this step, the system can directly understand the personalized needs expressed by the user in natural language and automatically transform them into a structured evaluation framework, thereby achieving customization and intelligence in the evaluation process from the source. Here, evaluation dimensions refer to specific aspects or characteristics used to measure training effectiveness, such as "depth of knowledge mastery," "communication fluency," or "problem-solving efficiency"; the logical relationships between evaluation dimensions describe the dependencies, associations, or hierarchical structures between different dimensions, for example, "teamwork ability" may depend on two sub-dimensions: "communication effectiveness" and "conflict resolution ability"; the evaluation dimension system is the overall structured representation of all these dimensions and their logical relationships, serving as the blueprint for the subsequent evaluation process. Taking a specific application scenario as an example, when a user inputs "needs to evaluate the trainees' on-the-spot response, product familiarity, and interactive persuasiveness during a sales skills training session," the evaluation dimension system constructed through this step may include three main evaluation dimensions: "on-the-spot response," "product familiarity," and "interactive persuasiveness." Furthermore, it is analyzed that "on-the-spot response" depends on two sub-dimensions: "timeliness of response" and "appropriateness of response," thus forming a structured evaluation network with hierarchical and dependent relationships.

[0018] In practical implementation, to construct the aforementioned evaluation dimension system, this embodiment can sequentially execute the following processes. First, based on natural language processing technology, semantic parsing is performed on the user-input training evaluation needs description to extract core evaluation intent elements. These elements reflect the user's initial intentions regarding evaluation goals, focus areas, and evaluation criteria. Subsequently, using abstract semantic representation technology, the extracted evaluation intent elements are further parsed into a graph structure that can represent entities and their semantic relationships. This graph structure breaks free from the surface grammatical constraints of the original text, directly revealing the core concepts and their connections involved in the needs description. Then, based on semantic role labeling technology, the entities constituting specific evaluation dimensions and their relationships are more precisely identified and extracted from the graph structure. For example, the entity representing "evaluation object" in the graph is combined with the relationship representing "evaluation attribute" to form a structured evaluation dimension definition. Finally, by analyzing the semantic dependencies, hierarchical or causal relationships of each structured evaluation dimension, a directed acyclic graph representing their logical relationships is constructed. This graph clarifies the calculation or evaluation order between dimensions, ultimately forming the evaluation dimension system defined in this embodiment, which can be used to guide the entire subsequent evaluation process.

[0019] S102, for each evaluation dimension in the evaluation dimension system, select an appropriate target evaluation model from multiple candidate evaluation models to generate a model scheduling strategy that maps each evaluation dimension to the corresponding target evaluation model; wherein, the candidate evaluation model is used to perform the evaluation calculation of its respective evaluation dimension.

[0020] In this embodiment, after constructing the evaluation dimension system, the purpose of step S102 is to dynamically select the most suitable target evaluation model from multiple candidate evaluation models for each evaluation dimension in the system, and generate an optimized model scheduling strategy accordingly. In traditional evaluation systems, an evaluation dimension is often fixedly bound to a single evaluation model. This rigid one-to-one mapping cannot adapt to different data conditions, real-time system load, and diverse accuracy and efficiency requirements. By configuring multiple candidate evaluation models with the same or similar functions but different performance characteristics (such as computational complexity, accuracy, and the types of data they are good at processing) for each dimension, this embodiment enables the system to intelligently select and schedule models based on the real-time context when a specific evaluation task is executed. The core significance of this is to achieve optimal allocation of evaluation resources. It can comprehensively balance multiple objectives such as evaluation accuracy, response speed, and system energy consumption, thereby significantly improving the overall system's resource utilization efficiency and response performance while meeting evaluation quality requirements. The generated model scheduling strategy is essentially a clear "evaluation dimension - target evaluation model" mapping scheme, which guides the system to call the correct model to calculate the data in subsequent steps.

[0021] In practical implementation, to achieve the aforementioned intelligent scheduling, this embodiment employs a decision-making framework that integrates reinforcement learning and attention mechanisms. First, the system takes currently available resource information (such as CPU / memory utilization, network bandwidth), all evaluation dimension attributes in the evaluation dimension system, and the performance characteristic profiles of each candidate evaluation model as input. Through the reinforcement learning mechanism, the system initially selects a target evaluation model for each evaluation dimension, forming a preliminary model scheduling strategy. In this mechanism, the aforementioned input information is constructed as a state representation of the decision-making environment in which the reinforcement learning agent is located, and the process of selecting a candidate model for a certain dimension is defined as an action. The agent continuously optimizes its mapping strategy from environmental state to selected actions by constantly trying and receiving feedback effects (e.g., the quality of the comprehensive evaluation results, the overall task completion time, and resource consumption) after executing historical scheduling strategies, thereby learning which model to choose for which dimension in which situation will yield better long-term returns. Subsequently, to further optimize the preliminary strategy and consider the mutual influence between cross-dimensional model selections, this embodiment introduces a self-attention mechanism based on the Transformer architecture. This mechanism can analyze and determine the potential dependencies between the evaluation models selected for different evaluation dimensions (e.g., the output of model A may be the input of model B, or the two models may compete for the same type of computing resources). Based on these dependencies, and combined with a pre-set multi-objective optimization function (which aims to jointly optimize one or more of the indicators such as evaluation accuracy, evaluation response latency, and evaluation energy consumption), the system coordinates and fine-tunes the target evaluation models selected in the initial model scheduling strategy, ultimately generating a globally better and more coordinated final model scheduling strategy.

[0022] Furthermore, in practical applications, the training process data to be evaluated often includes at least one modality of data, such as text, audio, and video. Therefore, the candidate evaluation model library in this embodiment includes specialized evaluation models that match these different data modalities. For example, for the "language fluency" dimension, the candidate models may include sequence models for processing text, speech recognition and analysis models for processing audio, and lip-sync and rhythm analysis models for processing video. The scheduling strategy generated in step S102 is a comprehensive optimal decision made based on a full understanding of the evaluation requirements (dimensional system), the composition of the data modalities, and the performance of the dedicated models for each modality.

[0023] S103, according to the model scheduling strategy, call the corresponding target evaluation model to evaluate the training process data and obtain the evaluation results corresponding to each evaluation dimension.

[0024] This embodiment, based on the model scheduling strategy generated in the previous step, calls the corresponding target evaluation model to evaluate the training process data. The aim is to transform intelligent scheduling planning into concrete, reliable, and efficient evaluation calculations, thereby obtaining evaluation results for each evaluation dimension. This step is a crucial leap from "knowing how to evaluate" to "completing the evaluation calculation," ensuring that the evaluation task can be executed based on the optimal resource and model configuration. By strictly adhering to the model scheduling strategy, the system not only delivers on the intelligent decisions made in the dimension construction and model scheduling stages, enabling the evaluation process to balance multiple objectives such as accuracy, efficiency, and resource constraints, but more importantly, it provides an accurate and reliable foundation for the subsequent generation of the fused, final target evaluation results.

[0025] In practical implementation, to further improve the reliability of the evaluation results, the model scheduling strategy generated in this embodiment can configure one primary evaluation model and at least one alternative evaluation model for at least one evaluation dimension. During evaluation, this embodiment first calls the primary evaluation model to calculate the training process data, obtaining the initial evaluation result and its confidence level for that dimension. Subsequently, the system determines whether the confidence level is lower than a preset threshold. If the confidence level is lower than the preset threshold, it indicates that the primary evaluation model lacks certainty regarding its output, and the result may have a significant risk of error. In this case, the system will trigger the use of an alternative evaluation model to re-evaluate the same training process data, obtaining a reference evaluation result. The initial evaluation result and the reference evaluation result are then combined to determine the final evaluation result for that evaluation dimension, for example, through a weighted average or a confidence-based fusion strategy. If the confidence level is greater than or equal to the preset threshold, it indicates that the output of the primary evaluation model has high reliability, and the system directly determines the initial evaluation result as the final evaluation result for that evaluation dimension. It is worth noting that in this design, the number of parameters in the alternative evaluation model is usually higher than that in the main evaluation model. This means that the alternative model usually has stronger representational capabilities and potentially higher accuracy, but it also comes with greater computational overhead. The core significance of this cascaded evaluation mechanism, which involves the collaborative work of the main and alternative models, lies in constructing an adaptive balance system between accuracy and efficiency. It allows the system to prioritize the use of the lightweight and fast main evaluation model in most cases to ensure efficiency, and only calls upon the more powerful but resource-intensive alternative model for verification and calibration when the main model itself lacks confidence. This effectively avoids the huge resource consumption caused by indiscriminately using a large model in all scenarios in pursuit of absolute accuracy, and prevents the output of low-quality evaluation results through a confidence monitoring mechanism, thus achieving an organic unity between high-reliability evaluation and reasonable resource consumption as a whole.

[0026] Furthermore, to achieve collaborative performance improvement across multiple evaluation models while protecting data privacy, this embodiment employs a federated learning architecture to build and optimize the evaluation model network. Specifically, this framework allows multiple primary and alternative evaluation models deployed in different environments or serving different evaluation tasks to co-evolve without sharing original training data. Its core technical mechanism is embodied in the following differential privacy-preserving parameter update formula: ,in In this formula, This represents the parameter update amount generated by an evaluation model (such as a primary evaluation model) after training locally using its private data; N(0, σ²) is the added Gaussian noise satisfying a specific variance; the intensity of the noise is strictly controlled by the privacy parameters ε (privacy budget) and δ (privacy leakage probability). Only after this noise-added parameter update ΔW is it securely uploaded to a central parameter sharing pool. The technical effect of this mechanism forms a deep synergy and closed loop with the aforementioned model scheduling and confidence evaluation stages. First, it directly supports the reliability of the cascaded verification mechanism in step S103: through continuous evolution via federated learning, the capabilities of both the primary evaluation model and the candidate evaluation models can be improved, making the confidence-based switching decision itself more accurate and reliable. Second, it provides the entire evaluation system with continuous self-evolution capability: each model aggregates privacy-preserved global knowledge from the shared pool, continuously optimizes its own parameters, and thus can be selected with higher accuracy or lower resource consumption in future scheduling decisions. Ultimately, this co-evolutionary framework based on federated learning and differential privacy, while ensuring strict compliance with data privacy regulations, drives the continuous and secure growth of the performance of all candidate evaluation models, fundamentally enhancing the long-term effectiveness of the model scheduling strategy and the overall reliability of the evaluation results under the dynamic evaluation dimension system.

[0027] S104, based on the logical relationships in the evaluation dimension system, integrates the evaluation results of each evaluation dimension to generate target evaluation results for the training process data.

[0028] This embodiment, based on the logical relationships within the evaluation dimension system, integrates the evaluation results of each dimension. Its aim is to consolidate the scattered evaluation results, which target different aspects, into a comprehensive, coordinated final conclusion that aligns with the initial evaluation intent—that is, a target evaluation result for the training process data. The significance of this step lies in its transcendence of simple score summation or averaging; rather, it involves structured fusion based on the inherent dependencies, hierarchical relationships, or causal connections between dimensions. This ensures that the final evaluation result not only reflects the performance of each sub-dimension but also embodies the intrinsic connections and overall effect between these performances. This generates a logically consistent, highly interpretable, and truly user-relevant comprehensive evaluation report, enhancing the depth and decision-support value of the evaluation results.

[0029] In practical implementation, this embodiment guides the synthesis process based on the logical relationship structure, such as the directed acyclic graph, defined in the evaluation dimension system constructed in step S101. For example, for dimensions with hierarchical relationships (such as "teamwork ability" depending on "communication effectiveness" and "conflict resolution ability"), the system first calculates the parent dimension's result based on the evaluation results of the sub-dimensions, according to predefined or dynamically learned aggregation rules (such as weighting or nonlinear functions). For dimensions with dependency or sequential relationships, the synthesis of their evaluation results may consider the influence weight of preceding dimension results on subsequent dimensions. In this way, synthesis is no longer a simple numerical merging, but a structured calculation process that follows a predetermined evaluation logic blueprint, enabling the final target evaluation result to systematically reflect the overall picture of training effectiveness across multiple dimensions and levels.

[0030] Furthermore, to construct a continuously optimized closed-loop system, this embodiment may also include a feedback learning mechanism in practical applications. Specifically, after the system generates and outputs the target evaluation result, if it receives confirmation of the validity of the result from the user or regulator—for example, the score is recognized or the result is used for successful decision-making—this embodiment can associate and store two key pieces of information from this task: the original training requirements description and the model scheduling strategy that has been proven effective in practice. The core significance of this approach is to accumulate valuable labeled data for the system's self-evolution. These associated records constitute a continuously growing experiential knowledge base, which can be used for subsequent reinforcement learning training to optimize the model scheduling strategy, adjust dimensional construction rules, or calibrate confidence thresholds. This allows the entire evaluation system to continuously adapt and improve with increased usage, achieving the goal of becoming increasingly intelligent and continuously improving performance.

[0031] in Figure 2The core technical architecture and data flow corresponding to steps S101 to S104 of this embodiment are presented. The entire process begins with user input, which is received by the dynamic model building module. The integrated AMR+LLM semantic parser performs deep semantic understanding and structured parsing of the natural language requirements, thereby completing the construction of the evaluation dimension system in S101. The output of this module drives two parallel computing branches: a small-model parallel model cluster focuses on incremental knowledge integration, parsability analysis, and user feedback optimization, corresponding to the efficient main evaluation model in S103; simultaneously, the large-model modeling part relies on big data modeling, high-precision solutions, asynchronous decision groups, and knowledge model distillation techniques to constitute the high-precision alternative evaluation model in S103. The collaboration of the two branches is uniformly scheduled by a multi-intelligent decision layer. This layer captures system load in real time through state awareness and dynamically optimizes resource allocation and model call paths based on the DQN policy generation mechanism, combined with the Transformer architecture and multi-objective routing decision, thereby realizing the generation of intelligent model scheduling strategies in S102. The task coordination and state evolution of the entire system are maintained by the incremental decision-making layer management system, and the application strategies and parameters are continuously optimized through the underlying genetic algorithm, etc., ultimately forming a complete closed loop from semantic parsing, dimension construction, intelligent scheduling, model evaluation to result fusion (corresponding to S104), supporting the entire evaluation system to achieve self-iteration and collaborative evolution during operation.

[0032] In summary, the method of this embodiment first performs semantic parsing on the user-input training requirement description based on natural language processing technology, automatically constructing a dynamic evaluation dimension system containing multiple evaluation dimensions and their logical relationships. This fundamentally breaks the limitation of relying on a predefined static dimension library, realizing the dynamic response and expansion of evaluation dimensions to complex semantic requirements. Subsequently, for each evaluation dimension in the evaluation dimension system, an intelligent decision-making module integrating reinforcement learning and attention mechanisms dynamically selects the most suitable target evaluation model from its multiple candidate evaluation models and generates an optimal model scheduling strategy. This model scheduling strategy can comprehensively consider evaluation accuracy, response latency, and real-time system load, thereby overcoming the problems of simple binding between models and dimensions and rigid resource allocation. Then, according to the model scheduling strategy, the corresponding target evaluation model is called to perform parallel or cascade evaluation of the multimodal training process data, and a confidence calibration mechanism is introduced to ensure the reliability of the evaluation results of each dimension. Finally, based on the logical relationships inherent in the evaluation dimension system, the calibrated evaluation results of each dimension are comprehensively analyzed to generate accurate, interpretable, and scenario-appropriate final evaluation results. Through the synergy and progression of the above-mentioned steps, this solution achieves a comprehensive improvement in the evaluation system in terms of dimensional scalability, intelligent scheduling, and adaptive result fusion.

[0033] In addition, corresponding to Figure 1In addition to the method shown, another embodiment of this example also provides a training assessment device. Figure 3 This is a schematic diagram of the training assessment device 300, including: System construction module 310 uses natural language processing technology to perform semantic parsing on the training needs description input by the user, and constructs an evaluation dimension system that includes multiple evaluation dimensions and the logical relationships between each evaluation dimension.

[0034] The model selection module 320 selects a suitable target evaluation model from multiple candidate evaluation models for each evaluation dimension in the evaluation dimension system, so as to generate a model scheduling strategy that maps each evaluation dimension to the corresponding target evaluation model; wherein, the candidate evaluation model is used to perform the evaluation calculation of its respective evaluation dimension.

[0035] The evaluation execution module 330, according to the model scheduling strategy, calls the corresponding target evaluation model to evaluate the training process data and obtain the evaluation results corresponding to each evaluation dimension.

[0036] The evaluation processing module 340, based on the logical relationships in the evaluation dimension system, integrates the evaluation results of each evaluation dimension to generate a target evaluation result for the training process data.

[0037] Optionally, the system construction module 310 performs semantic parsing on the user-input training needs description based on natural language processing technology to construct an evaluation dimension system containing at least one evaluation dimension and the logical relationships between each evaluation dimension. This includes: performing semantic parsing on the training evaluation needs description based on natural language processing technology to extract evaluation intent elements; parsing the evaluation intent elements into a graph structure representing entities and their semantic relationships based on abstract semantic representation technology; identifying and extracting the entities and relationships constituting the evaluation dimensions from the graph structure based on semantic role labeling technology to form structured evaluation dimensions; and constructing a directed acyclic graph representing the logical relationships between each structured evaluation dimension to form the evaluation dimension system.

[0038] Optionally, the model selection module 320 selects a suitable target evaluation model from all candidate evaluation models for each evaluation dimension in the evaluation dimension system, obtaining a model scheduling strategy that maps each evaluation dimension to the corresponding target evaluation model. This includes: using currently available resource information, all evaluation dimensions in the evaluation dimension system, and the performance characteristics of each candidate evaluation model as input, selecting a target evaluation model for each evaluation dimension in the evaluation dimension system through a reinforcement learning mechanism to obtain a preliminary model scheduling strategy; wherein, the reinforcement learning mechanism is used to: construct the input information as a state representation of the decision environment, define the evaluation model selection process for each evaluation dimension as an action, optimize the mapping strategy from state to action based on the feedback effect of historical model scheduling strategies; determine the dependency relationship between the evaluation models selected for different evaluation dimensions using a self-attention mechanism based on the Transformer architecture, and adjust the target evaluation model selected by the preliminary model scheduling strategy based on the dependency relationship and a pre-set target optimization function to obtain the final model scheduling strategy; wherein, the target optimization function is used to optimize at least one of the evaluation accuracy, evaluation response latency, and evaluation energy consumption.

[0039] Optionally, the model scheduling strategy configures at least one evaluation dimension with a primary evaluation model and at least one alternative evaluation model. The evaluation execution module 330, according to the model scheduling strategy, calls the corresponding target evaluation model to evaluate the training process data, obtaining evaluation results for each evaluation dimension. This includes: calling the primary evaluation model to evaluate the training process data, obtaining an initial evaluation result and its confidence level; if the confidence level is lower than a preset threshold, calling the alternative evaluation model to evaluate the training process data, obtaining a reference evaluation result, and determining the final evaluation result for the evaluation dimension based on the initial evaluation result and the reference evaluation result; if the confidence level is greater than or equal to the preset threshold, determining the initial evaluation result as the final evaluation result for the evaluation dimension.

[0040] Optionally, the alternative evaluation model has a higher number of model parameters than the main evaluation model.

[0041] Optionally, the training process data includes at least one modality of data, such as text, audio, and video; the candidate evaluation model includes evaluation models that match different data modalities.

[0042] Optionally, the device in this embodiment further includes a recording module, used to associate and store the training requirement description with the model scheduling strategy after receiving confirmation of the validity of the evaluation result for the target.

[0043] It should be noted that the training and assessment device in this embodiment can be used as... Figure 1The execution body of the method shown is therefore able to achieve... Figure 1 The steps and functions of the method shown are illustrated.

[0044] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Please refer to it. Figure 4 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.

[0045] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0046] Memory is used to store computer programs. Specifically, a computer program may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides the computer program to the processor.

[0047] Specifically, the processor reads the corresponding computer program from non-volatile memory into memory and then runs it, forming the above-mentioned logical structure. Figure 3 The training assessment device shown. Correspondingly, the processor executes the program stored in the memory, and specifically performs the following operations: Based on natural language processing technology, semantic parsing is performed on the training needs description input by users to construct an evaluation dimension system that includes multiple evaluation dimensions and the logical relationships between them.

[0048] For each evaluation dimension in the evaluation dimension system, a suitable target evaluation model is selected from multiple candidate evaluation models to generate a model scheduling strategy that maps each evaluation dimension to the corresponding target evaluation model; wherein, the candidate evaluation model is used to perform the evaluation calculation of its respective evaluation dimension.

[0049] According to the model scheduling strategy, the corresponding target evaluation model is invoked to evaluate the training process data, and the evaluation results corresponding to each evaluation dimension are obtained.

[0050] Based on the logical relationships within the aforementioned evaluation dimension system, and by synthesizing the evaluation results of each evaluation dimension, a target evaluation result is generated for the training process data.

[0051] The above is as described in this instruction manual. Figure 1 The training and assessment method disclosed in the illustrated embodiments can be applied to a processor and implemented by the processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit in the processor or by instructions in the form of software. The processor mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0052] Of course, in addition to software implementation, the electronic device described in this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0053] Furthermore, embodiments of this application also propose a computer program product, including a computer-readable storage medium storing one or more computer programs, the one or more computer programs including instructions.

[0054] When the aforementioned instructions are executed by a portable electronic device that includes multiple applications, they enable the portable electronic device to perform... Figure 1 The steps in the method shown include: Based on natural language processing technology, semantic parsing is performed on the training needs description input by users to construct an evaluation dimension system that includes multiple evaluation dimensions and the logical relationships between them.

[0055] For each evaluation dimension in the evaluation dimension system, a suitable target evaluation model is selected from multiple candidate evaluation models to generate a model scheduling strategy that maps each evaluation dimension to the corresponding target evaluation model; wherein, the candidate evaluation model is used to perform the evaluation calculation of its respective evaluation dimension.

[0056] According to the model scheduling strategy, the corresponding target evaluation model is invoked to evaluate the training process data, and the evaluation results corresponding to each evaluation dimension are obtained.

[0057] Based on the logical relationships within the aforementioned evaluation dimension system, and by synthesizing the evaluation results of each evaluation dimension, a target evaluation result is generated for the training process data.

[0058] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0059] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0060] The above are merely embodiments of this specification and are not intended to limit the scope of this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification. Furthermore, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of this document.

Claims

1. A training assessment method, characterized in that, include: Based on natural language processing technology, semantic parsing is performed on the training needs description input by users to construct an evaluation dimension system that includes multiple evaluation dimensions and logical relationships between each evaluation dimension; For each evaluation dimension in the evaluation dimension system, a suitable target evaluation model is selected from multiple candidate evaluation models to generate a model scheduling strategy that maps each evaluation dimension to the corresponding target evaluation model; wherein, the candidate evaluation model is used to perform the evaluation calculation of its respective evaluation dimension; According to the model scheduling strategy, the corresponding target evaluation model is invoked to evaluate the training process data, and the evaluation results corresponding to each evaluation dimension are obtained. Based on the logical relationships within the aforementioned evaluation dimension system, and by synthesizing the evaluation results of each evaluation dimension, a target evaluation result is generated for the training process data.

2. The method according to claim 1, characterized in that, Based on natural language processing technology, semantic parsing of user-inputted training needs descriptions is performed to construct an evaluation dimension system that includes at least one evaluation dimension and the logical relationships between these dimensions, including: The training assessment needs description is semantically parsed using natural language processing technology to extract assessment intent elements; Based on abstract semantic representation technology, the evaluation intent elements are parsed into graph structures that represent entities and their semantic relationships; Based on semantic role labeling technology, entities and relationships constituting the evaluation dimensions are identified and extracted from the graph structure to form a structured evaluation dimension; Construct a directed acyclic graph representing the logical relationships between the structured evaluation dimensions to form the evaluation dimension system.

3. The method according to claim 1, characterized in that, For each evaluation dimension in the evaluation dimension system, a suitable target evaluation model is selected from multiple candidate evaluation models to generate a model scheduling strategy that maps each evaluation dimension to the corresponding target evaluation model, including: Using currently available resource information, all evaluation dimensions in the evaluation dimension system, and the performance characteristics of each candidate evaluation model as input, a target evaluation model is selected for each evaluation dimension in the evaluation dimension system through a reinforcement learning mechanism to obtain a preliminary model scheduling strategy; wherein, the reinforcement learning mechanism is used to: construct the input information as a state representation of the decision environment, define the evaluation model selection process for each evaluation dimension as an action, and optimize the mapping strategy from state to action based on the feedback effect of the historical model scheduling strategy; The self-attention mechanism based on the Transformer architecture determines the dependencies between the evaluation models selected for different evaluation dimensions, and adjusts the target evaluation model selected by the initial model scheduling strategy based on the dependencies and the pre-set objective optimization function to obtain the final model scheduling strategy; wherein, the objective optimization function is used to optimize at least one of the evaluation accuracy, evaluation response latency and evaluation energy consumption.

4. The method according to claim 1, characterized in that, The model scheduling strategy is that at least one evaluation dimension is configured with a primary evaluation model and at least one alternative evaluation model; According to the model scheduling strategy, the corresponding target evaluation model is invoked to evaluate the training process data, and the evaluation results corresponding to each evaluation dimension are obtained, including: The main evaluation model is invoked to evaluate the training process data, and initial evaluation results and their confidence levels are obtained. If the confidence level is lower than a preset threshold, the alternative evaluation model is invoked to evaluate the training process data to obtain a reference evaluation result, and the final evaluation result of the evaluation dimension is determined based on the initial evaluation result and the reference evaluation result. If the confidence level is greater than or equal to the preset threshold, then the initial evaluation result is determined as the final evaluation result of the corresponding evaluation dimension.

5. The method according to claim 4, characterized in that, The alternative evaluation model has a higher number of model parameters than the main evaluation model.

6. The method according to claim 1, characterized in that, The training process data includes at least one modality of data, such as text, audio, and video; the candidate evaluation models include evaluation models that are matched with different data modalities.

7. The method according to claim 1, characterized in that, Also includes: After receiving confirmation of the validity of the evaluation results for the target, the training requirement description is associated with and stored in relation to the model scheduling strategy.

8. A training assessment device, characterized in that, include: The system construction module uses natural language processing technology to semantically parse the training needs descriptions input by users and constructs an evaluation dimension system that includes multiple evaluation dimensions and the logical relationships between them. The model selection module selects a suitable target evaluation model from multiple candidate evaluation models for each evaluation dimension in the evaluation dimension system, so as to generate a model scheduling strategy that maps each evaluation dimension to the corresponding target evaluation model; wherein, the candidate evaluation model is used to perform the evaluation calculation of its respective evaluation dimension; The evaluation execution module, according to the model scheduling strategy, calls the corresponding target evaluation model to evaluate the training process data and obtain the evaluation results corresponding to each evaluation dimension. The evaluation processing module, based on the logical relationships in the evaluation dimension system, integrates the evaluation results of each evaluation dimension to generate a target evaluation result for the training process data.

9. An electronic device, comprising: processor; And a memory arranged to store computer-executable instructions, characterized in that, when executed, the executable instructions cause the processor to perform the method as described in any one of claims 1 to 7.

10. A computer program product, the computer program product comprising a computer-readable storage medium storing a computer program, characterized in that, The computer program is operable to cause the computer to perform the method as described in any one of claims 1 to 7.

Images