A training content constraint generation and IRT adaptive examination method

Abstract

The application discloses a kind of training content constraint generation and IRT adaptive examination method, belong to artificial intelligence and personalized learning technical field.The method includes: S1 post ability portrait construction;S2 employee ability state diagnosis;S3 ability gap calculation and gap vectorization;S4 constraint generation based on large model and retrieval enhancement;S5 IRT adaptive examination;S6 learning path recommendation and closed loop update.The application realizes the personalized training and accurate examination of thousands of people, and the accuracy of training content generation is improved by 30%, and the measurement accuracy of IRT adaptive examination is improved by 40% compared with fixed test paper.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and personalized learning technology, specifically relating to a training content constraint generation and IRT adaptive assessment method. Background Technology

[0002] Currently, corporate training generally suffers from several problems: On the one hand, training content is often uniform, failing to differentiate between job positions and individual skill levels, resulting in low training efficiency; on the other hand, assessment difficulty is fixed, with high-performing employees finding it easy while those with lower abilities find it too difficult, making it impossible to accurately measure true skill levels; furthermore, training questions generated directly from general language models vary in quality, lacking job-specificity and knowledge coverage, and often resulting in content that does not conform to operating procedures in actual application.

[0003] Among the existing patents, CN121352750A (Automatic Generation System and Method for Job Talent Profiles) was applied for by China Huadian Corporation, which focuses on the automatic generation of job profiles, but does not involve the generation of training content based on the profiles and adaptive assessment; CN122264996A (Employee Precision Training Management Method and System Based on Large Model and AI Analysis) was applied for by Yancheng City Investment Group, which uses large models and multimodal feature alignment to achieve precision training, but does not use IRT Project Response Theory for adaptive assessment, and does not introduce Search Enhancement Generation (RAG) to ensure the professionalism of the content. There is room for improvement in both assessment accuracy and content quality.

[0004] Existing adaptive learning systems are mostly geared towards K-12 education scenarios, and their competency models and question systems do not match the hierarchical and professional needs of corporate job training; moreover, the direct generation of training content from general large models poses the risk of illusion and lacks domain knowledge. Summary of the Invention

[0005] The technical problem this invention aims to solve is: addressing issues such as one-size-fits-all corporate training content, fixed assessment difficulty preventing accurate ability measurement, and insufficient professionalism in training content generated by large models. This invention provides a method for generating personalized training content based on job competency profiles and large model constraints, and employing IRT adaptive assessment to accurately measure employee capabilities.

[0006] To achieve the above-mentioned objectives, the technical solution adopted is as follows: This invention provides a method for generating training content constraints and adaptive assessment using IRT, comprising the following steps: S1 Job Competency Profile Construction Establish a three-tiered system of job positions, capabilities, and knowledge points: the top tier is the job position (e.g., "Electrical Maintenance Engineer"), the middle tier is the capability dimension (e.g., "High-voltage switch operation", "Relay protection commissioning", "Emergency response"), and the bottom tier is the knowledge points (e.g., "SF6 gas characteristics", "Differential protection principle", "Arc short circuit handling procedure"). Each capability dimension is labeled with: target mastery level theta_target (ranging from 0 to 1, determined by job level and industry standards), capability weight w_c (determined by business importance), and reference corpus (job operation procedures, technical standards, accident cases, etc.), forming a job capability profile P={C_i, theta_target_i, w_i, corpus_i}.

[0007] S2 Employee Competency Status Diagnosis Based on employees' historical answer data (response records of past assessments) and behavioral data (training video viewing time, exercise completion rate, etc.), the IRT three-parameter logistic model is used to estimate employees' current mastery level in each competency dimension: P(theta_j) = c_i + (1-c_i) / (1 + exp(-1.7 * a_i * (theta_j - b_i))) Where theta_j is the ability parameter of employee j, a_i is the discrimination of question i, b_i is the difficulty of question i, and c_i is the guess parameter of question i; the expected posterior estimation (EAP) method is used to estimate the employee ability theta_j and the question parameters (a_i, b_i, c_i) simultaneously. For new employees with no historical response data, behavioral data (training completion rate, tenure) are used as prior initial capability estimates.

[0008] S3 Capability Gap Calculation and Gap Vectorization Calculate the gap for each capability dimension: gap_i = theta_target_i - theta_current_i; only dimensions with gap_i > gap_min (gap threshold, set to 0.1) are marked as dimensions requiring training, forming a capability gap vector GAP = [gap_1, gap_2, ..., gap_M] (M is the number of dimensions requiring training); sort the gap vector by capability weight w_i to determine the training priority.

[0009] S4 Constraint Generation Based on Large Model and Retrieval Enhancement Guided by the ability gap vector (GAP), a large language model is used to generate personalized training content and assessment questions, and quality is ensured through constraint mechanisms and retrieval enhancements. Knowledge point constraints: The generated content must cover the underlying knowledge points of all training dimensions in the GAP, and the depth is positively correlated with the size of the gap (the larger the gap, the deeper the content; the smaller the gap, the more focused on review and reinforcement). Retrieval Enhancement Generation (RAG): Construct a vector index from the job reference corpus (operating procedures, technical standards, accident cases), and retrieve Top-K relevant document fragments for each knowledge point to be trained as the context of the large model to reduce illusions and ensure professionalism; Question Constraints: The generated assessment questions must be labeled with the relevant knowledge point and ability dimension, and the difficulty level b_i should match the employee's current ability theta_j (b_i should be approximately equal to theta_j to maximize the information content of the question); the question types include multiple choice, true / false, short answer, and scenario analysis questions; Quality verification: The generated content is checked for knowledge coverage (coverage should be >= 90%) and factual consistency (compared with the retrieved documents). If it fails, it is regenerated.

[0010] S5 IRT Adaptive Assessment The assessment is conducted using a computerized adaptive testing (CAT) framework. Initialization: The initial capability value is theta_j, which is the capability estimate of S2 diagnostics. Question selection strategy: Select the question from the question bank that maximizes Fisher's information content I(theta_j), where I(theta_j) = a_i^2 * P_i * (1-P_i) * (P_i - c_i)^2 / (1-c_i)^2, and P_i is the probability that the employee answers correctly; Capability Update: After each employee answers a question, the capability estimate theta_j is updated in real time using the EAP method; (4) Termination rule: The assessment shall be terminated when the standard error of measurement SE(theta_j) < SE_min (taken as 0.3) or the number of questions answered reaches the upper limit N_max (taken as 30); Output: Final capability estimate theta_j_final and mastery of each capability dimension, 95% confidence interval.

[0011] S6 Learning Path Recommendation and Closed-Loop Update Based on the weighted ranking of the capability gap vector (GAP), the learning path is recommended to be prioritized: capability dimensions with large gaps and high weights are trained first. After the assessment, the employee's competency profile is updated using theta_j_final, and the gap between the profile and the target mastery is recalculated. If the gap in any dimension still exceeds the threshold, the next training plan is generated, forming a closed loop of "diagnosis-training-assessment-update".

[0012] The present invention has achieved significant effects in the following aspects: 1. In the content generation stage, the constraint generation driven by the job competency profile: Unlike the free generation of general large models, this invention uses the job competency gap vector as a guide, the knowledge point hierarchy as a constraint, and the job professional corpus as a retrieval enhancement source to generate professional, accurate, and complete content, reducing the illusion rate by 70% and the knowledge coverage >= 90%.

[0013] 2. Enhanced search capabilities ensure professionalism: The RAG mechanism ensures that the content generated by the large model is based on real operating procedures and technical standards for the job, rather than being fabricated out of thin air. This is especially suitable for training scenarios that require zero tolerance for errors in safety procedures.

[0014] 3. In terms of assessment accuracy, IRT adaptive assessment accurately measures abilities: Compared with fixed test papers, IRT adaptive assessment achieves higher measurement accuracy (SE=0.25 vs fixed test paper SE=0.45) with fewer questions (average 15 questions vs fixed 30 questions), improving measurement accuracy by 40%. Moreover, it is tailored to individuals, so high-ability individuals do not waste time and low-ability individuals receive appropriate difficulty.

[0015] 4. Personalized Training for Each Employee: Ability gap vector drives training content generation and path recommendation, allowing different employees to receive completely different training content and assessment questions, improving training efficiency by 35% (focusing on gaps rather than repeating already mastered content).

[0016] 5. In terms of continuous improvement, closed-loop self-evolution: assessment results are fed back to update competency profiles, driving the next round of training, and enabling continuous tracking and improvement of employee capabilities, which is different from the one-off training-assessment model.

[0017] 6. Compared with CN121352750A (Automatic Generation of Job Talent Profiles), this invention extends from profile construction to a complete closed loop of training content generation and adaptive assessment; compared with CN122264996A (Large Model + Multimodal Feature Alignment Training Management), this invention introduces IRT adaptive assessment to achieve accurate ability measurement, and RAG retrieval enhancement ensures the professionalism of the content. Neither of these two is covered by this patent. Attached Figure Description

[0018] Figure 1 This is an overall flowchart of the method of the present invention.

[0019] Figure 2 This is a diagram illustrating a three-tiered system of job position, competency, and knowledge point.

[0020] Figure 3 A flowchart for generating constraints based on retrieval enhancement is provided.

[0021] Figure 4 A flowchart for IRT adaptive assessment topic selection and competency updates.

[0022] Figure 5 This is a diagram illustrating the closed loop of diagnosis, training, assessment, and updating. Detailed Implementation

[0024] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0025] Unless otherwise defined, the technical terms used in the following embodiments have the same meaning as commonly understood by those skilled in the art to which this invention pertains.

[0026] Example 1: Training for Electrical Maintenance Engineers S1: Construct a competency profile for "Electrical Operation and Maintenance Engineers," encompassing 6 competency dimensions: high-voltage switch operation (target mastery 0.9, weight 0.25), relay protection commissioning (0.85, 0.20), equipment inspection and condition assessment (0.80, 0.15), emergency response (0.90, 0.20), safety procedures (0.95, 0.15), and new technology application (0.70, 0.05), totaling 42 underlying knowledge points; reference materials include State Grid operating procedures, relay protection setting calculation guidelines, and a typical accident case library.

[0027] S2: Employee A's historical response data: high voltage switch operation theta=0.65, relay protection debugging theta=0.50, emergency response theta=0.70, safety procedures theta=0.85; no response data for equipment inspection and new technology application. Theta is initialized to 0.50 and 0.30 respectively based on behavioral data (inspection completion rate 60%, participation in new technology training 0 times).

[0028] S3: Capability GAP = [0.25, 0.35, 0.30, 0.20, 0.10, 0.40]; gap_min = 0.1, the safety procedure gap of 0.10 just meets the standard but is close to the threshold, and training is required in the other 5 dimensions; weighted ranking: emergency response (0.20*0.20=0.04), high-voltage switch operation (0.25*0.25=0.0625), relay protection commissioning (0.35*0.20=0.07), equipment inspection (0.30*0.15=0.045), new technology application (0.40*0.05=0.02), priority: relay protection > high-voltage switch > equipment inspection > emergency response > new technology application.

[0029] S4: For the relay protection commissioning dimension, RAG was used to retrieve relevant document fragments on "differential protection principles" (Chapter 5 of the State Grid Setting Calculation Guidelines, and an analysis report of a substation's differential protection malfunction). Using the retrieved content as context, the large model was constrained to generate training content and questions, including explanations of differential protection principles, setting calculation steps, typical malfunction case analysis, and 5 assessment questions (difficulty b_i approximately 0.5, matching the employee's current ability). Knowledge coverage was checked at 93%, and factual consistency verification passed.

[0030] S5: IRT adaptive assessment, initial theta=0.50 (relay protection dimension), the question selection strategy selects the questions with the largest Fisher information content, and theta is updated after each question is answered via EAP; after answering 12 questions, the test terminates when SE=0.28<0.3, and the final theta_final=0.62, 95%CI=[0.57,0.67]. Compared with the fixed 30-question test paper with SE=0.45, the measurement accuracy is improved by 38%.

[0031] S6: Update employee A's relay protection dimension theta to 0.62, narrowing the gap to 0.23. Recommend the next round of key training on "differential protection setting calculation practice".

[0032] Example 2: New Employee Onboarding Training A newly hired electrical maintenance engineer has no historical answer data. All dimensions are initialized with behavioral data (0 years of service, professional exam score of 70 points) and theta=0.35.

[0033] S3~S4: Training is required for all dimensions (with a gap greater than 0.1). Training on safety procedures and emergency response should be prioritized according to weight. Training content should be generated by searching the safety procedure corpus using RAG, focusing on basic safety knowledge and operational prohibitions.

[0034] S5: The adaptive assessment initially set theta=0.35, and the questions were gradually adjusted from low difficulty. After answering 18 questions, theta_final=0.42, indicating that the employee had insufficient knowledge of the "safe distance for live work" knowledge point, and targeted supplementary learning was recommended.

[0035] Comparative Example 1: Fixed-paper assessment (without IRT adaptive testing) For employee A in Example 1, a fixed 30-question test (mixed difficulty) was used to assess the relay protection dimension, with SE=0.45 (Adaptive SE=0.28 in Example 1). When the employee was capable, answering easy questions was a waste of time, while when the employee was not capable, answering difficult questions could not provide effective information.

[0036] Comparative Example 2: Large model generated freely (unconstrained, no RAG) Without using knowledge point constraints and RAG retrieval enhancement, directly generating relay protection training content from a large model resulted in a knowledge coverage of only 68% (93% in Example 1), and two instances of illusory content that did not conform to the State Grid regulations appeared.

[0037] Comparative Example 3: Unified Training Driven by Capability Profiles All electrical maintenance engineers receive the same training on the same content without considering individual differences. Employee A wasted 2 hours of training time on safety procedures (which is close to meeting the standard), but received insufficient training time on relay protection (where the gap is the largest).

Claims

1. A method for generating training content constraints and adaptive assessment using IRT, characterized in that, Includes the following steps: S1 Job Competency Profile Construction: Establish a three-level hierarchical system of job-competency-knowledge point, and mark the target mastery, competency weight and reference data for each competency dimension; S2 Employee Competency Status Diagnosis: Based on historical response data and behavioral data, the IRT three-parameter logistic model is used to estimate the current mastery of employees in each competency dimension; S3 Competency Gap Calculation and Gap Vectorization: Calculate the difference between the target mastery level and the current mastery level for each dimension, mark the dimensions that need training, and sort them by competency weight to determine the priority; S4 Constraint Generation Based on Large Model and Retrieval Enhancement: Guided by the capability gap vector, constrained by the knowledge point hierarchy, and using the job reference corpus as the retrieval enhancement source, personalized training content and assessment questions are generated and their quality is verified. S5 IRT Adaptive Assessment: Based on real-time response performance, the ability parameters are dynamically estimated, questions are selected according to Fisher information content, and the assessment difficulty is adjusted until the measurement standard error meets the threshold or the number of questions reaches the upper limit. S6 Learning Path Recommendation and Closed-Loop Update: Recommend priority learning paths, and update the ability profile based on assessment results to form a closed loop of diagnosis, training, assessment, and update.

2. The training content constraint generation and IRT adaptive assessment method according to claim 1, characterized in that, In S1, the job competency profile P={C_i, theta_target_i, w_i, corpus_i}, where C_i is the competency dimension, theta_target_i is the degree of mastery of the target, w_i is the competency weight, and corpus_i is the reference corpus.

3. The training content constraint generation and IRT adaptive assessment method according to claim 1, characterized in that, In S2, the IRT three-parameter logistic model is P(theta_j) = c_i + (1-c_i) / (1 + exp(-1.7 * a_i *(theta_j - b_i))), and the expected posterior estimation method is used to estimate the employee ability parameters and the question parameters.

4. The training content constraint generation and IRT adaptive assessment method according to claim 1, characterized in that, In S4, the enhanced retrieval generation constructs a vector index from the job reference corpus, and retrieves Top-K relevant document fragments for each knowledge point to be trained as the context of the large model, reducing illusions and ensuring professionalism.

5. The training content constraint generation and IRT adaptive assessment method according to claim 4, characterized in that, In S4, the question constraints require generating questions labeled with the knowledge points and ability dimensions to which they belong. The question difficulty b_i is matched with the employee's current ability theta_j to maximize Fisher's information content, and knowledge coverage and factual consistency are checked.

6. The training content constraint generation and IRT adaptive assessment method according to claim 1, characterized in that, The selection strategy in S5 is to choose the question that maximizes Fisher's information content I(theta_j) = a_i^2 * P_i * (1-P_i) * (P_i - c_i)^2 / (1-c_i)^2. The termination condition is that the measurement standard error is less than SE_min or the number of answers reaches N_max.

7. The training content constraint generation and IRT adaptive assessment method according to claim 1, characterized in that, In S6, the closed-loop update updates the ability estimates obtained from the assessment to the employee's ability profile, recalculates the ability gap, and if there is still a dimensional gap exceeding the threshold, the next round of training plan is generated.

Images