Method and related device for evaluating programming ability of student based on multi-modal data

Abstract

This application provides a method and related apparatus for assessing students' programming abilities based on multimodal data. The method includes: firstly, systematically collecting multimodal data generated by students throughout the entire programming training process, such as interactive behaviors, source code and modification sequences, running results and performance indicators, and extracting them into three types of feature vectors representing process, quality and effectiveness. These features are then input into a pre-trained multidimensional ability assessment model, which maps the above features to professional ability standard items, outputs a fine-grained ability mastery score and identifies ability gaps. Subsequently, based on these gaps, a personalized sequence of subsequent training tasks is dynamically generated. By executing tasks and collecting new data, a closed-loop tracking and continuous updating of the ability status is achieved. Finally, all information is integrated to generate a visualized ability assessment report, which can improve the support effectiveness and accuracy of the assessment results for personalized teaching guidance, and effectively help programming education transform from outcome evaluation to process empowerment.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a method and related apparatus for evaluating students' programming abilities based on multimodal data. Background Technology

[0002] Currently, the assessment methods for trainees' programming abilities mainly rely on the analysis of programming results. Common assessment methods include: automating the testing of submitted code to verify functional correctness, scoring based on the pass rate of pre-set test cases, and checking code style and complexity using static code analysis tools. These methods can reflect the trainees' final output and some code quality to a certain extent, forming the basic means of evaluating programming abilities.

[0003] However, the existing methods mentioned above still have significant limitations. First, the assessment dimensions are relatively singular, usually focusing on the final running result or static quality of the code, lacking in-depth capture and analysis of students' programming process behavior. Second, the assessment system is not strongly correlated with specific professional competency standards or teaching objectives, making it difficult to map one-off programming results onto a systematic competency map, resulting in limited support for personalized learning guidance from the assessment results. Summary of the Invention

[0004] This application provides a method and related apparatus for assessing students' programming abilities based on multimodal data, which can improve the effectiveness and accuracy of assessment results in supporting personalized teaching guidance, and effectively help programming education transform from outcome evaluation to process empowerment.

[0005] A first aspect of this application provides a method for assessing students' programming abilities based on multimodal data, the method comprising: Collect multimodal data generated by trainees during the IoT application development training process; The multimodal data is processed to obtain behavioral feature vectors, code quality feature vectors, and result performance feature vectors; The behavioral feature vector, code quality feature vector, and result effectiveness feature vector are input into a pre-trained multi-dimensional ability assessment model to obtain the trainees' mastery scores for each ability item, and their ability gaps are identified based on the score distribution. Based on the aforementioned capability gaps and the IoT training question bank, a personalized sequence of follow-up training tasks is generated. The personalized follow-up training task sequence is executed, and the trainees' mastery scores and ability gaps are updated based on the new multimodal data collected during the execution process. Based on the mastery score, skill gaps, and their historical trends, a comprehensive visual assessment report on programming skills is generated.

[0006] In one possible implementation, the multimodal data generated by trainees during IoT application development training includes interactive behavior data, source code, modification sequence data, execution results, and performance index data; the methods for collecting the interactive behavior data, source code, modification sequence data, execution results, and performance index data include: By integrating a behavior tracking SDK into the front end of the programming training environment, interactive behavior data is collected, including page access path, code editing dwell time, debugging operation frequency, and code submission number. By integrating a monitoring module into the code editor, source code and modification sequence data are collected. The modification sequence data includes code snapshots of each save or commit, differences in code changes, and code structure information. The monitoring module, integrated into the code execution and testing environment, collects execution results and performance metrics data, including compilation error information, unit test pass rate, code execution time, peak memory usage, and simulated hardware interaction logs.

[0007] In one possible implementation, the processing of multimodal data to extract behavioral feature vectors, code quality feature vectors, and outcome performance feature vectors includes: The interactive behavior data is serialized and statistically processed to extract behavioral feature vectors, wherein the behavioral feature vectors include average debugging depth, effective coding time percentage, and task switching frequency. Static and temporal analysis are performed on the source code and modification sequence data to extract code quality feature vectors, which include code cyclomatic complexity, comment density, module reuse rate, and code refactoring activity. The running results and performance index data are aggregated and calculated to extract the result performance feature vector, which includes the overall pass rate of test cases, the average performance compliance rate, and the hardware interaction command execution success rate.

[0008] In one possible implementation, the construction and scoring process of the pre-trained multi-dimensional capability assessment model includes: Construct a model input layer to receive the behavior feature vector, code quality feature vector, and result performance feature vector; Construct a mapping weight matrix to map the fused features to N predefined competency items in the National Vocational Skill Standard for Internet of Things Engineering Technicians; Calculate the trainee's mastery score on the i-th competency item; Based on the mastery score set of all ability items, ability items with scores below a preset threshold are identified as current ability gaps.

[0009] In one possible implementation, generating a personalized sequence of subsequent training tasks based on the capability gap and the IoT training question bank includes: Based on the identified capability gaps, all candidate training questions associated with the weak capability items are selected from the IoT training question bank; Based on the trainees' historical training records and real-time ability scores, the expected utility value of each candidate topic for filling ability gaps is calculated. Candidate questions are sorted according to expected utility values ​​and combined in a progressive manner to generate a personalized training task sequence containing a specific question order and difficulty gradient.

[0010] In one possible implementation, the execution of the personalized follow-up training task sequence, and the updating of the trainee's mastery score and competency gap based on the newly acquired multimodal data during execution, includes: Trainees perform the questions in the training task sequence, and the system synchronously collects new multimodal data generated during the process; Based on the new multimodal data, update the behavioral feature vector, code quality feature vector, and result performance feature vector; The updated feature vectors are then input back into the multi-dimensional ability assessment model to recalculate and update the mastery scores and ability gaps for each ability item.

[0011] In one possible implementation, the step of generating a visual comprehensive assessment report on programming skills based on the mastery score, skill gaps, and their historical trends includes: Based on the latest mastery score set, a radar chart-style ability profile is generated to intuitively display the strength and weakness distribution of each ability item; Based on historical data on capability gap changes, a capability evolution trend chart is generated to show the improvement of key weak links; Based on the current capability profile and evolution trends, generate structured text recommendations, which should include at least the capability items that should be prioritized for strengthening in the next stage, recommended knowledge points for learning, and typical training tasks.

[0012] This example provides a method for assessing trainees' programming abilities based on multimodal data. First, it systematically collects multimodal data generated by trainees throughout the entire programming training process, including interactive behaviors, source code and modification sequences, running results and performance indicators. This data is then extracted into three types of feature vectors representing process, quality, and effectiveness. These vectors are then input into a pre-trained multidimensional ability assessment model. The aforementioned features are mapped to professional ability standard items, outputting a fine-grained ability mastery score and identifying ability gaps. Subsequently, based on these gaps, a personalized sequence of subsequent training tasks is dynamically generated. By executing these tasks and collecting new data, a closed-loop tracking and continuous updating of the ability status is achieved. Finally, all information is integrated to generate a visualized ability assessment report. By integrating multi-dimensional data on process behavior, code quality, and operational efficiency, and establishing a mapping relationship between these data and systematic competency standards, this approach overcomes the limitations of traditional methods, which suffer from single-dimensional assessments and a disconnect from teaching objectives. It achieves a panoramic and in-depth diagnosis of students' programming abilities. Furthermore, through personalized task recommendations based on competency gaps and a dynamic closed-loop mechanism of "assessment-training-reassessment," it transforms one-time static evaluations into continuous adaptive learning guidance. This significantly improves the effectiveness and accuracy of assessment results in supporting personalized teaching guidance, effectively contributing to the transformation of programming education from outcome evaluation to process empowerment.

[0013] A second aspect of this application provides a learning programming ability assessment device based on multimodal data, the device comprising: The data acquisition module is used to collect multimodal data generated by trainees during the IoT application development training process; The first processing module is used to process the multimodal data to obtain behavioral feature vectors, code quality feature vectors, and result performance feature vectors. The second processing module is used to input the behavior feature vector, code quality feature vector and result effectiveness feature vector into the pre-trained multi-dimensional ability assessment model to obtain the trainee's mastery score for each ability item, and identify their ability gaps based on the score distribution. The third processing module is used to generate a personalized sequence of subsequent training tasks based on the aforementioned capability gap and the IoT training question bank. The fourth processing module executes the personalized subsequent training task sequence and updates the trainees' mastery scores and ability gaps based on the newly collected multimodal data during the execution process. The final assessment module is used to generate a visual comprehensive assessment report on programming skills based on the mastery score, ability gaps, and their historical trends.

[0014] A third aspect of this application provides a terminal including a processor, an input device, an output device, and a memory, wherein the processor, input device, output device, and memory are interconnected, wherein the memory is used to store a computer program, the computer program including program instructions, and the processor is configured to invoke the program instructions to execute the steps of the student programming ability assessment method based on multimodal data as described in the first aspect of this application.

[0015] A fourth aspect of this application provides a computer-readable storage medium storing a computer program for electronic data interchange, wherein the computer program causes a computer to perform some or all of the steps described in the multimodal data-based student programming ability assessment method of the first aspect of this application.

[0016] A fifth aspect of this application provides a computer program product, comprising a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps described in the multimodal data-based student programming ability assessment method of the first aspect of this application. The computer program product may be a software installation package. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This application provides a schematic diagram of the overall process for evaluating students' programming abilities based on multimodal data. Figure 2 This application provides a schematic diagram of the overall structure of a student programming ability assessment device based on multimodal data, as an embodiment of the present application. Figure 3 This application provides a schematic diagram of the structure of a terminal. Figure label: Data acquisition module-1, first processing module-2, second processing module-3, third processing module-4, fourth processing module-5, final evaluation module-6. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0021] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0022] The method for assessing students' programming abilities based on multimodal data is applied to a device for assessing students' programming abilities based on multimodal data. Figure 1 A schematic diagram illustrating the overall process of a method for assessing learners' programming abilities based on multimodal data is shown. Figure 1 As shown, it includes: S1. Collect multimodal data generated by trainees during the IoT application development training process.

[0023] The multimodal data includes interaction behavior data, source code, modification sequence data, execution results, and performance index data; the methods for collecting the interaction behavior data, source code, modification sequence data, execution results, and performance index data include: S101. Collect interactive behavior data through the behavior tracking SDK integrated into the front end of the programming training environment. The interactive behavior data includes page access path, code editing dwell time, debugging operation frequency, and code submission number.

[0024] S102. By integrating the monitoring module into the code editor, source code and modification sequence data are collected, wherein the modification sequence data includes code snapshots of each save or commit, code change differences, and code structure information.

[0025] S103. By integrating the monitoring module into the code execution and testing environment, the running results and performance index data are collected, wherein the performance index data includes compilation error information, unit test pass rate, code execution time, peak memory usage, and simulated hardware interaction logs.

[0026] In this example, through the aforementioned three-pronged collaborative data collection mechanism, the system can comprehensively and objectively record the entire chain of data—from behavior to process to result—of the learner's programming training across three core dimensions: human-computer interaction, code creation, and program execution results. This approach lays a solid data foundation for subsequent competency modeling and analysis, enabling the evaluation model to not only focus on the correctness and performance of the final code but also to gain deeper insights into the learner's problem-solving strategies, coding habits, and debugging efficiency. This makes a comprehensive and in-depth assessment of programming skills possible. S2. Process the multimodal data to obtain behavioral feature vectors, code quality feature vectors, and result performance feature vectors.

[0027] Step S2 includes: S201. The interaction behavior data is serialized and statistically processed to extract behavior feature vectors, wherein the behavior feature vectors include average debugging depth, effective coding time ratio, and task switching frequency.

[0028] The system serializes and statistically processes the collected interactive behavior data to extract behavioral feature vectors. Specifically, the system first sorts all user interactions with the programming environment, such as clicking debug, edit, and switching files, by timestamp to form a structured event sequence. Based on this sequence, the average debugging depth is calculated by statistically analyzing the average number of times in each debugging session are used to view variable values, add breakpoints, and perform other in-depth investigations; the effective coding time percentage is calculated by removing periods of inactivity or frequent task switching from the total editing time; and the task switching frequency is measured by statistically analyzing the number of times a student switches between different code files, task description pages, and the debugger per unit of time.

[0029] In this example, by quantifying the interaction behavior in this fine-grained way, the abstract programming process is transformed into computable features, which can objectively reflect the learner's strategy, focus and work habits when solving problems, and provide key process indicators for assessing the coherence of their logical thinking and debugging efficiency.

[0030] S202. Perform static and temporal analysis on the source code and modification sequence data to extract code quality feature vectors, wherein the code quality feature vectors include code cyclomatic complexity, comment density, module reuse rate, and code refactoring activity.

[0031] The system performs static and temporal analysis on source code and modification sequence data to extract code quality feature vectors. Specifically, code cyclomatic complexity is calculated by using static analysis tools such as Checkstyle and SonarQube to parse the abstract syntax tree of code snapshots and calculate the number of linearly independent paths in the program. Comment density is obtained by calculating the ratio of effective comment lines to total lines of code. Module reusability is measured by identifying functions or classes defined in the code and calculating the proportion of times they are repeatedly called in the current and historical tasks. Code refactoring activity is analyzed by examining consecutive modification sequences and calculating the proportion of changes made between two commits that only optimize code structure, naming, or design rather than modifying functional logic.

[0032] In this example, by integrating the static attributes and dynamic evolution characteristics of the code, its purpose is to go beyond simple syntactic correctness judgment and to quantitatively evaluate the trainee's code engineering ability and professional competence from multiple dimensions such as structural standardization, maintainability, design awareness and continuous improvement habits.

[0033] S203. Aggregate and calculate the running results and performance index data, and extract the result performance feature vector. The result performance feature vector includes the test case comprehensive pass rate, average performance compliance rate, and hardware interaction instruction execution success rate.

[0034] The system aggregates and calculates runtime results and performance metrics to extract performance feature vectors. Specifically, the overall test case pass rate is not simply calculated by counting the number of passed test cases, but rather by assigning different weights to the difficulty and importance of each test case to calculate a weighted pass rate. The average performance compliance rate is calculated by comparing measured performance metrics such as program execution time and memory usage with the pre-set passing and excellent thresholds for the task, thus calculating the average percentage of compliance across all performance dimensions. The hardware interaction command execution success rate, for IoT development scenarios, is calculated by comparing the sequence of hardware operation commands issued by the program with the expected responses in the simulation environment or real devices, thus determining the success rate of correct command recognition and execution.

[0035] In this example, by performing multi-level performance aggregation on the running results, the purpose is not only to evaluate whether the program can do it correctly, but also to evaluate how well it does it and how reliable it is in specific scenarios of interacting with the physical world. This comprehensively measures the functional completeness, efficiency, and practical ability of the student's programming results in a specific field.

[0036] S3. Input the behavioral feature vector, code quality feature vector, and result effectiveness feature vector into the pre-trained multi-dimensional ability assessment model to obtain the trainee's mastery score for each ability item, and identify their ability gaps based on the score distribution.

[0037] The construction and scoring process of the pre-trained multi-dimensional ability assessment model includes: S301. Construct a model input layer to receive the behavior feature vector, code quality feature vector, and result performance feature vector.

[0038] The input layer of the system model receives the behavioral feature vector F_b, code quality feature vector Fc, and result performance feature vector Fp extracted in step S2. Specifically, this input layer is responsible for standardizing and concatenating the three vectors to form a unified fused feature vector, which serves as the starting point for subsequent model processing.

[0039] S302. Construct a mapping weight matrix to map the fused features to N predefined competency items in the National Vocational Skill Standard for Internet of Things Engineering Technicians.

[0040] The system constructs and applies a mapping weight matrix W to map the fused features to N predefined competency items in the National Vocational Skills Standard for IoT Engineering Technicians. Specifically, during the training phase, the model learns the correlation strength between each competency item and various features. Each row vector Wi in the weight matrix W represents the attention weight of the i-th competency item to each feature, thus transforming general programming behavior features into a metric for specific professional abilities.

[0041] S303. Calculate the trainee's mastery score on the i-th competency item.

[0042] The following formula is used for calculation:

[0043] In the formula, This represents the Sigmoid activation function, which normalizes the output to the (0,1) interval, intuitively representing the level of mastery. This is the weight vector corresponding to the i-th ability item in the weight matrix; , , These are feature vectors representing behavior, code quality, and outcome performance, respectively; α, β, and γ are trainable hyperparameters that adjust the relative importance of the three types of features in the comprehensive evaluation, and satisfy α+β+γ=1. is the bias term for the i-th ability item; the operator · represents the dot product operation of vectors. The physical meaning of this formula is that by weighted fusion of three types of features and matching them with preset ability weights, the mastery score of the ability is finally obtained after normalization.

[0044] S304. Based on the mastery score set of all ability items, identify ability items with scores below a preset threshold as the current ability gaps.

[0045] Among them, the system calculates the mastery score set of all ability items { , ,..., The system identifies current skill gaps. Specifically, it presets a skill mastery threshold θ (e.g., θ=0.6) and assigns scores accordingly. Capability entries below the threshold θ are automatically identified and marked as current capability weaknesses, i.e., capability gaps.

[0046] In this example, by constructing a computable model based on standard mapping and feature fusion, the purpose is to establish an objective and fine-grained ability quantification mechanism. It not only systematically links discrete programming activity data to a macro-level professional competency framework, giving the assessment results clear professional orientation and interpretability, but also automatically identifies competency gaps by setting scientific thresholds, providing a direct and reliable basis for subsequent precise and personalized learning interventions.

[0047] S4. Generate a personalized sequence of follow-up training tasks based on the aforementioned capability gaps and the IoT training question bank.

[0048] Step S4 includes the following sub-steps: S401. Based on the identified capability gaps, select all candidate training questions associated with the weak capability items from the IoT training question bank.

[0049] In this process, the system performs targeted searches within the IoT training question bank based on one or more capability gaps identified in step S3. Specifically, each question in the question bank is pre-labeled with tags representing several capability items intended for training and assessment. The system uses the set of tags for all currently weak capability items as query criteria to quickly retrieve all questions matching those tags, forming an initial set of candidate training questions.

[0050] In this example, the precise matching mechanism based on ability tags ensures that the selected training questions are highly aligned with the students' actual weaknesses in terms of training objectives, laying the foundation for building highly targeted training programs and avoiding blind recommendations or a "sea of ​​questions" approach.

[0051] S402. Based on the trainees' historical training records and real-time ability scores, calculate the expected utility value of each candidate topic for filling the ability gap.

[0052] Specifically, to evaluate the training value of each candidate question, an expected utility value U is designed and calculated. In practice, this value is calculated by considering the strength of the correlation between the question and the skill gap, the learner's historical familiarity with the knowledge points related to the question, and the inherent difficulty of the question. The expected utility value U can be calculated using the following formula:

[0053] In the formula, Let G represent the expected utility value of the j-th candidate topic; G represents the set of currently identified capability gaps. This indicates the strength of the association between question j and the ability gap k, and is usually a predefined label weight value; Assess trainees' real-time mastery of ability k; This reflects the urgency of filling this gap; P represents the difficulty level of the question; C represents the appropriate difficulty level dynamically estimated by the system based on the student's historical performance; and C represents the adjustment factor. The exponential part in the formula is used to smoothly map the difficulty fit, ensuring that the difficulty of the recommended questions matches the student's current level and avoiding questions that are too difficult or too easy.

[0054] In this example, the utility calculation model using the aforementioned multi-factor fusion achieves intelligent decision-making from "matching questions" to "optimizing questions." It not only considers the relevance of questions to weaknesses but also incorporates the learner's individual state (current level, historical performance), thereby calculating the questions with the highest training efficiency for a specific learner, transforming the recommendation from "suitable" to "most suitable."

[0055] S403. Sort the candidate questions according to the expected utility value, and combine them in a step-by-step manner to generate a personalized training task sequence containing a specific question order and difficulty gradient.

[0056] The system sorts all candidate questions in descending order based on the calculated expected utility value U_j, initially obtaining a list sorted by individual question value. In practice, the system doesn't simply select the first N questions; instead, it adjusts the list a second time according to a "gradual progression" principle. It ensures that the final generated task sequence presents a reasonable difficulty gradient overall. For example, it places a few highly relevant and moderately difficult questions at the beginning of the sequence to build confidence, then gradually increases the difficulty and comprehensiveness of the questions, and appropriately distributes questions testing the same abilities to avoid fatigue.

[0057] In this example, the task sequence construction method, which combines utility ranking with teaching principles, aims to organize isolated, high-value questions into a learning path that aligns with cognitive patterns and allows for sustainable progress. This not only improves the efficiency of individual training sessions but also promotes the transfer, consolidation, and deepening of knowledge and skills through carefully designed sequences, thereby achieving a systematic improvement in capabilities from individual points to a coherent line.

[0058] S5. Execute the personalized follow-up training task sequence, and update the trainees' mastery scores and ability gaps based on the newly collected multimodal data during the execution process.

[0059] Step S5 includes the following sub-steps: S501. Trainees perform the questions in the training task sequence, and the system synchronously collects new multimodal data generated during the process.

[0060] Specifically, when a trainee begins executing the personalized training task sequence generated in step S4, the system immediately initiates a data collection process that is identical to step S1 but targets this training session. In practice, throughout the entire process of the trainee completing each question in the sequence, the behavior tracking SDK integrated into the front end, the code editor monitoring module, and the background running test monitoring module work together again to collect, in real-time and completely, all interactive behavior data, source code and modification sequence data, as well as running results and performance metrics data generated by the trainee during this round of training.

[0061] In this example, by simultaneously initiating full-dimensional data collection while trainees perform targeted training tasks, the purpose is to ensure that the evaluation model can obtain first-hand data reflecting the trainees' latest status and behavior. This provides a real and timely data source for accurately perceiving changes in their abilities after a round of training intervention, and is a key data input link for achieving dynamic evaluation and closed-loop optimization.

[0062] S502. Based on the new multimodal data, update the behavior feature vector, code quality feature vector, and result performance feature vector.

[0063] The system takes the new multimodal data collected in step S501 as input and feeds it into a feature processing pipeline that is identical to that in step S2. Specifically, the system calls the same serialization statistics, static time-series analysis, and aggregation calculation logic to process the new interactive behavior data, source code and modification sequence data, and running results and performance index data, respectively. It outputs a new set of behavioral feature vectors, code quality feature vectors, and result performance feature vectors. These new vectors reflect the process characteristics, code quality, and result performance exhibited by the trainee in completing the latest training task.

[0064] In this example, a mature feature engineering process is reused to process new data and update the feature vector. This ensures the consistency and comparability of the basic indicators for ability assessment. This allows the system to quantify the differences in trainees' states before and after a new round of training using the same metrics, providing a standardized and comparable feature representation for subsequent accurate calculation of changes in ability.

[0065] S503. Input the updated feature vector back into the multi-dimensional ability assessment model to recalculate and update the mastery score and ability gap for each ability item.

[0066] Specifically, the system uses the latest feature vectors obtained in step S502—namely, the updated behavioral feature vector, code quality feature vector, and result performance feature vector—as input to feed back into the pre-trained multi-dimensional ability assessment model constructed in step S3. In practice, the model, based on the same scoring logic and formula, recalculates the learner's mastery scores across all N ability items using the learner's latest behavioral and achievement data, and then applies a preset mastery threshold to identify new sets of ability gaps.

[0067] In this example, by recalculating the original assessment model with the latest features, the system dynamically tracks and updates the learner's ability status. This marks the formal formation of the "assessment-training-reassessment" closed loop. The system no longer relies on outdated historical data but can provide immediate and accurate ability diagnoses based on the learner's latest learning performance. This allows personalized recommendations to dynamically adjust as the learner progresses, truly achieving adaptive and continuous talent development.

[0068] S6. Based on the mastery score, ability gap and its historical trend, generate a visual comprehensive assessment report on programming ability.

[0069] Step S6 includes: S601. Based on the latest mastery score set, generate a radar chart-style ability profile to intuitively display the strength and weakness distribution of each ability item.

[0070] The system uses all updated competency item labels and their corresponding latest mastery scores from step S5 as data sources to drive the visualization engine for plotting. Specifically, the system uses the N core competency items defined in the National Vocational Skill Standard for IoT Engineering Technicians as the N axes of the radar chart. It maps the mastery score for each competency item to a scale value on the corresponding axis, and then connects all the scale points to form a closed polygon. Different colors or fills can be used to distinguish between historical and current scores, or to identify competency gap dimensions below a threshold.

[0071] In this example, by generating a radar chart-style competency profile, the purpose is to transform abstract, multi-dimensional scoring data into an intuitive and comprehensive visual form. This allows learners and teachers to quickly grasp the overall outline and structural strengths and weaknesses of an individual's abilities, rapidly identifying areas of strength and significant shortcomings, and greatly improving the comprehensibility and communication efficiency of the assessment results.

[0072] S602. Based on historical data on capability gap changes, generate a capability evolution trend chart to show the improvement of key weak links.

[0073] The system extracts all records from the database that indicate a learner's skill gaps across all training and assessment cycles, and organizes them chronologically. Specifically, the system selects several of the most critical weaknesses as tracking indicators, plotting multiple trend curves with time on the horizontal axis and the corresponding skill mastery score on the vertical axis. Each curve clearly shows the trajectory of the score change for that specific skill from its initial identification as a gap to the present, and highlights the time points of each targeted training task through different line types or markers.

[0074] In this example, by drawing a capability evolution trend chart, the purpose is to extend a static snapshot of capabilities into a dynamic growth documentary. It not only shows where the learner is now, but also reveals how they got there and the speed of their progress, providing direct evidence for evaluating the effectiveness of learning interventions. At the same time, it can also motivate learners to gain positive feedback for continuous learning by observing the cumulative results of their own efforts.

[0075] S603. Based on the current capability profile and evolution trend, generate structured text suggestions, which should at least include the capability items that should be prioritized for strengthening in the next stage, the knowledge points recommended for learning, and typical training tasks.

[0076] The system integrates the results of the aforementioned visualization analysis and outputs a structured text summary and recommendations using pre-set rule templates and natural language generation technology. Specifically, the text report first provides a general description of the overall capabilities outlined by the radar chart; then, it uses a trend chart to identify key capabilities that have shown significant progress or stagnated; finally, based on the most pressing capability gaps, historical progress rates, and metadata from the IoT training question bank, it automatically generates specific recommendations. For example, it suggests prioritizing the strengthening of sensor data fusion programming capabilities, recommends reviewing asynchronous programming and message queue communication knowledge, and suggests selecting a specific IoT gateway protocol implementation task from the question bank to begin practice.

[0077] In this example, by generating structured text suggestions that integrate data insights and specific guidance, the final step in transforming data analysis into action guidelines is completed. This transforms the assessment report from a diagnostic report into a personalized learning prescription, directly linking macro-level ability assessments with micro-level knowledge point learning and specific training tasks. This provides a clear and actionable path for students' efficient advancement and teachers' precise guidance.

[0078] This example provides a method for assessing trainees' programming abilities based on multimodal data. First, it systematically collects multimodal data generated by trainees throughout the entire programming training process, including interactive behaviors, source code and modification sequences, running results and performance indicators. This data is then extracted into three types of feature vectors representing process, quality, and effectiveness. These vectors are then input into a pre-trained multidimensional ability assessment model. The aforementioned features are mapped to professional ability standard items, outputting a fine-grained ability mastery score and identifying ability gaps. Subsequently, based on these gaps, a personalized sequence of subsequent training tasks is dynamically generated. By executing these tasks and collecting new data, a closed-loop tracking and continuous updating of the ability status is achieved. Finally, all information is integrated to generate a visualized ability assessment report. By integrating multi-dimensional data on process behavior, code quality, and operational efficiency, and establishing a mapping relationship between these data and systematic competency standards, this approach overcomes the limitations of traditional methods, which suffer from single-dimensional assessments and a disconnect from teaching objectives. It achieves a panoramic and in-depth diagnosis of students' programming abilities. Furthermore, through personalized task recommendations based on competency gaps and a dynamic closed-loop mechanism of "assessment-training-reassessment," it transforms one-time static evaluations into continuous adaptive learning guidance. This significantly improves the effectiveness and accuracy of assessment results in supporting personalized teaching guidance, effectively contributing to the transformation of programming education from outcome evaluation to process empowerment.

[0079] For those consistent with the above, please refer to Figure 2 , Figure 2 This application provides a schematic diagram of the structure of a student programming ability assessment device based on multimodal data, as an embodiment of the present application. Figure 2 As shown, the device includes: Data acquisition module 1 is used to collect multimodal data generated by trainees during the IoT application development training process; The first processing module 2 is used to process the multimodal data to obtain behavioral feature vectors, code quality feature vectors, and result performance feature vectors. The second processing module 3 is used to input the behavior feature vector, code quality feature vector and result effectiveness feature vector into the pre-trained multi-dimensional ability assessment model to obtain the trainee's mastery score for each ability item, and identify their ability gaps based on the score distribution. The third processing module 4 is used to generate a personalized sequence of subsequent training tasks based on the capability gap and the IoT training question bank. The fourth processing module 5 executes the personalized subsequent training task sequence and updates the trainees' mastery scores and ability gaps based on the newly collected multimodal data during the execution process. The final assessment module 6 is used to generate a visual comprehensive assessment report on programming ability based on the mastery score, ability gap and its historical trend.

[0080] For examples consistent with the above embodiments, please refer to... Figure 3 , Figure 3 A schematic diagram of a terminal structure provided in an embodiment of this application is shown in the figure. It includes a processor, an input device, an output device, and a memory. The processor, input device, output device, and memory are interconnected. The memory is used to store a computer program, which includes program instructions. The processor is configured to call the program instructions. The program includes instructions for performing the following steps. Collect multimodal data generated by trainees during the IoT application development training process; The multimodal data is processed to obtain behavioral feature vectors, code quality feature vectors, and result performance feature vectors; The behavioral feature vector, code quality feature vector, and result effectiveness feature vector are input into a pre-trained multi-dimensional ability assessment model to obtain the trainees' mastery scores for each ability item, and their ability gaps are identified based on the score distribution. Based on the aforementioned capability gaps and the IoT training question bank, a personalized sequence of follow-up training tasks is generated. The personalized follow-up training task sequence is executed, and the trainees' mastery scores and ability gaps are updated based on the new multimodal data collected during the execution process. Based on the mastery score, skill gaps, and their historical trends, a comprehensive visual assessment report on programming skills is generated.

[0081] The above mainly describes the solutions of the embodiments of this application from the perspective of the method execution process. It is understood that, in order to achieve the above functions, the terminal includes the corresponding hardware structure and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments provided herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0082] This application embodiment can divide the terminal into functional units according to the above method example. For example, each function can be divided into a separate functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0083] This application also provides a computer storage medium storing a computer program for electronic data interchange, which causes a computer to perform some or all of the steps of any of the multimodal data-based student programming ability assessment methods described in the above method embodiments.

[0084] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program that causes a computer to perform some or all of the steps of any of the student programming ability assessment methods based on multimodal data as described in the above method embodiments.

[0085] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0086] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0087] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0088] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0089] Furthermore, the functional units in the various embodiments of the application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software program module.

[0090] If the integrated unit is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0091] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage device, which may include: a flash drive, a read-only memory, a random access memory, a magnetic disk, or an optical disk, etc.

[0092] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for assessing learners' programming ability based on multimodal data, characterized in that, include: Collect multimodal data generated by trainees during the IoT application development training process; The multimodal data is processed to obtain behavioral feature vectors, code quality feature vectors, and result performance feature vectors; The behavioral feature vector, code quality feature vector, and result effectiveness feature vector are input into a pre-trained multi-dimensional ability assessment model to obtain the trainees' mastery scores for each ability item, and their ability gaps are identified based on the score distribution. Based on the aforementioned capability gaps and the IoT training question bank, a personalized sequence of follow-up training tasks is generated. The personalized follow-up training task sequence is executed, and the trainees' mastery scores and ability gaps are updated based on the new multimodal data collected during the execution process. Based on the mastery score, skill gaps, and their historical trends, a comprehensive visual assessment report on programming skills is generated.

2. The method for evaluating a student's programming ability based on multi-modal data according to claim 1, wherein, The multimodal data generated by trainees during IoT application development training includes interactive behavior data, source code, modification sequence data, execution results, and performance index data. The methods for collecting this interactive behavior data, source code, modification sequence data, execution results, and performance index data include: By integrating a behavior tracking SDK into the front end of the programming training environment, interactive behavior data is collected, including page access path, code editing dwell time, debugging operation frequency, and code submission number. By integrating a monitoring module into the code editor, source code and modification sequence data are collected. The modification sequence data includes code snapshots of each save or commit, differences in code changes, and code structure information. The monitoring module, integrated into the code execution and testing environment, collects execution results and performance metrics data, including compilation error information, unit test pass rate, code execution time, peak memory usage, and simulated hardware interaction logs.

3. The method for evaluating learner programming ability based on multimodal data according to claim 1, characterized in that, The process of processing multimodal data to extract behavioral feature vectors, code quality feature vectors, and result performance feature vectors includes: The interactive behavior data is serialized and statistically processed to extract behavioral feature vectors, wherein the behavioral feature vectors include average debugging depth, effective coding time percentage, and task switching frequency. Static and temporal analysis are performed on the source code and modification sequence data to extract code quality feature vectors, which include code cyclomatic complexity, comment density, module reuse rate, and code refactoring activity. The running results and performance index data are aggregated and calculated to extract the result performance feature vector, which includes the overall pass rate of test cases, the average performance compliance rate, and the hardware interaction command execution success rate.

4. The method for evaluating learner programming ability based on multimodal data according to claim 1, characterized in that, The construction and scoring process of the pre-trained multi-dimensional capability assessment model includes: Construct a model input layer to receive the behavior feature vector, code quality feature vector, and result performance feature vector; Construct a mapping weight matrix to map the fused features to N predefined competency items in the National Vocational Skill Standard for Internet of Things Engineering Technicians; Calculate the trainee's mastery score on the i-th competency item; Based on the mastery score set of all ability items, ability items with scores below a preset threshold are identified as current ability gaps.

5. The method for evaluating learner programming ability based on multimodal data according to claim 1, characterized in that, The step of generating a personalized sequence of subsequent training tasks based on the capability gap and the IoT training question bank includes: Based on the identified capability gaps, all candidate training questions associated with the weak capability items are selected from the IoT training question bank; Based on the trainees' historical training records and real-time ability scores, the expected utility value of each candidate topic for filling ability gaps is calculated. Candidate questions are sorted according to expected utility values ​​and combined in a progressive manner to generate a personalized training task sequence containing a specific question order and difficulty gradient.

6. The method for evaluating learner programming ability based on multimodal data according to claim 1, characterized in that, The process of executing the personalized follow-up training task sequence and updating the trainees' mastery scores and skill gaps based on the newly collected multimodal data during execution includes: Trainees perform the questions in the training task sequence, and the system synchronously collects new multimodal data generated during the process; Based on the new multimodal data, update the behavioral feature vector, code quality feature vector, and result performance feature vector; The updated feature vectors are then input back into the multi-dimensional ability assessment model to recalculate and update the mastery scores and ability gaps for each ability item.

7. The method for evaluating learner programming ability based on multimodal data according to claim 1, characterized in that, Based on the mastery score, ability gaps, and their historical trends, a comprehensive visual assessment report on programming skills is generated, including: Based on the latest mastery score set, a radar chart-style ability profile is generated to intuitively display the strength and weakness distribution of each ability item; Based on historical data on capability gap changes, a capability evolution trend chart is generated to show the improvement of key weak links; Based on the current capability profile and evolution trends, generate structured text recommendations, which should include at least the capability items that should be prioritized for strengthening in the next stage, recommended knowledge points for learning, and typical training tasks.

8. A device for assessing trainees' programming abilities based on multimodal data, characterized in that, include: The data acquisition module is used to collect multimodal data generated by trainees during the IoT application development training process; The first processing module is used to process the multimodal data to obtain behavioral feature vectors, code quality feature vectors, and result performance feature vectors. The second processing module is used to input the behavior feature vector, code quality feature vector and result effectiveness feature vector into the pre-trained multi-dimensional ability assessment model to obtain the trainee's mastery score for each ability item, and identify their ability gaps based on the score distribution. The third processing module is used to generate a personalized sequence of subsequent training tasks based on the aforementioned capability gap and the IoT training question bank. The fourth processing module executes the personalized subsequent training task sequence and updates the trainees' mastery scores and ability gaps based on the newly collected multimodal data during the execution process. The final assessment module is used to generate a visual comprehensive assessment report on programming skills based on the mastery score, ability gaps, and their historical trends.

9. A terminal, characterized in that, The system includes a processor, an input device, an output device, and a memory, which are interconnected. The memory is used to store a computer program, which includes program instructions. The processor is configured to invoke the program instructions to execute the student programming ability assessment method based on multimodal data as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions, which, when executed by a processor, cause the processor to perform the student programming ability assessment method based on multimodal data as described in any one of claims 1-7.

Images