Intelligent practical operation assessment system and method based on multi-modal perception

Abstract

The invention relates to the technical field of intelligent education, in particular to an intelligent practical operation assessment system and method based on multi-modal perception, and the system comprises a question setting and rule configuration module, a local assessment terminal, a hardware connection monitoring module and a remote management service terminal. The question setting and rule configuration module is used for generating a structured test question rule file, the hardware connection monitoring module is used for collecting conduction relation data, and the local assessment terminal comprises a man-machine interaction layer, a sensing and monitoring layer and an application logic and control layer; the remote management service terminal is used for data storage and instruction issuing. The system has the following advantages and effects: through the cooperation of the question setting and rule configuration module, the local assessment terminal, the hardware connection monitoring module and the remote management service terminal, the operation behavior and the tool state are identified; multi-mode perception, automatic monitoring and accurate scoring of electrician practical training type practical operation assessment are realized, and the objectivity, efficiency and traceability of assessment are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent education technology, and in particular to an intelligent practical assessment system and method based on multimodal perception. Background Technology

[0002] With the deepening of China's manufacturing and industrial strategies, the demand for highly skilled personnel is increasing daily. In vocational education and engineering training, practical assessment is a core component for evaluating students' hands-on abilities, process standards, and safety awareness. However, traditional practical assessment models have long faced three major challenges: difficulty in invigilation, difficulty in scoring, and difficulty in tracing the source of knowledge. Specifically: Firstly, the scoring is highly subjective and lacks quantitative standards: traditional assessments mainly rely on the examiner's visual observation and experience. Different examiners often have different standards for judging details such as "whether the operation is standardized" and "whether the wiring is correct," resulting in inconsistent and unobjective scoring results that are prone to disputes.

[0003] Secondly, the assessment is inefficient and resource-intensive: due to the complexity of the practical process, an examiner can usually only focus on the operational details of 1-2 candidates at a time. In large-scale skills assessments or final exams, it often takes several days to complete the assessment of all candidates, and requires a large investment of teaching staff, resulting in extremely high assessment organization costs.

[0004] Thirdly, safety hazards are difficult to prevent in real time, and the risk of accidents is high: Electrical practical operations usually involve high-voltage operations such as 220V or even 380V. In high-risk aspects such as live wiring and power-on debugging, examiners cannot monitor every action of each candidate in real time. Once a candidate engages in violations such as pulling the switch under load or touching exposed conductors while the circuit is live, the examiner often cannot stop them in time, which can easily lead to short circuits and burnout of equipment or even electric shock accidents.

[0005] Fourthly, there is a lack of process data, making it difficult to establish a closed-loop teaching feedback mechanism: Traditional assessments often "emphasize results over process," typically only checking circuit continuity at the end of the assessment. There is a lack of effective recording methods for process-related problems such as "improper tool placement," "incorrect wiring connections," and "non-standard operation." After the assessment, students only receive a score and cannot review their mistakes by watching the recording, making it difficult to form a closed-loop teaching system of "assessment-feedback-improvement."

[0006] Currently, some automated assessment equipment and technologies have attempted to address these issues, such as assessment systems based on virtual simulation or offline testing devices based on the continuity of hardware circuits. However, the former lacks physical feedback and tactile training from real-world operations; the latter can only make static judgments on the final wiring results and cannot perform real-time monitoring and behavioral analysis of the operation process, making it difficult to meet the comprehensive requirements for operational standardization, safety, and process traceability in complex practical scenarios.

[0007] Therefore, there is an urgent need to develop an intelligent practical assessment system that can deeply integrate computer vision perception and underlying hardware status monitoring, and has the functions of real-time monitoring of the entire process, immediate warning of dangerous behaviors, automatic scoring of operating procedures, and complete traceability of evidence chains, in order to solve the technical problems of subjective scoring, low efficiency, lack of safety supervision, and process data gaps in the existing assessment model. Summary of the Invention

[0008] The purpose of this invention is to provide an intelligent practical assessment system and method based on multimodal perception to solve the problems mentioned in the background art.

[0009] The above-mentioned technical objective of the present invention is achieved through the following technical solution: This invention provides an intelligent practical assessment system based on multimodal perception, including a question generation and rule configuration module, a local assessment terminal, and a hardware connection monitoring module; wherein, The question generation and rule configuration module is used to generate and manage structured question rule files containing connection rules and interchangeable terminal pair rules, and to store the structured question rule files in the question rule library; the hardware connection monitoring module is connected to the terminal block being tested, and is used to collect the conduction relationship data between each terminal in the terminal block in real time; The local assessment terminal includes a human-computer interaction layer, a perception and monitoring layer, and an application logic and control layer. The human-computer interaction layer performs human-computer interaction with the examinee. The perception and monitoring layer includes a video access module and a terminal block monitoring module, used to receive and analyze video streams from the examination room cameras in real time to identify examinee operation behaviors and tool status and generate detection result data. Simultaneously, it receives and parses conduction relationship data collected from the hardware connection monitoring module, compares it with the structured test question rule file corresponding to the current test question loaded from the question generation and rule configuration module, and determines the terminal connection status. The application logic and control layer performs automatic scoring based on multimodal data fusion based on the detection result data and the terminal connection status. It also includes a remote management service terminal for receiving and centrally storing data from each local assessment terminal, and for issuing control commands to each local assessment terminal.

[0010] By adopting the above technical solutions, the question-generating and rule-configuration modules enable flexible definition and management of test questions. The local assessment terminal integrates visual and logical analysis to achieve automated on-site monitoring and scoring. The hardware connection monitoring module accurately collects the physical wiring status. Combined with the remote management service terminal, centralized data management and instruction issuance are achieved, forming a complete closed loop from question generation, assessment, scoring to management. This significantly improves the standardization, automation, and organizational management efficiency of the assessment, solving the fundamental problems of traditional assessment models that rely on manual labor, are inefficient, and are difficult to implement on a large scale.

[0011] A further setting is that, in the perception and monitoring layer, the process of identifying the examinee's operational behavior specifically includes: The video stream from the examination room camera is subjected to multi-target parallel detection using a preset visual model, which simultaneously identifies and locates air switch targets and hand targets in the scene; among them, for the identified air switch targets, the visual model is used to further distinguish whether they are in the on or off state. Perform semantic fusion analysis for safe operation: When an air switch target and a hand target that are both identified as being in the open state are present in the same video frame in a preset area, or based on the detection results of multiple consecutive video frames, it is preliminarily determined that there is a risk of illegal operation.

[0012] By adopting the above technical solutions, high-risk behaviors such as "live operation" or "unauthorized contact" can be identified in real time and automatically, realizing proactive and intelligent monitoring of operational safety. This effectively compensates for the negligence that may exist in manual invigilation, greatly reduces the risk of safety accidents in practical assessments, and transforms the abstract requirement of safety regulations into specific indicators that can be quantified and detected, thereby enhancing the rigor of the assessment.

[0013] A further setting includes the following steps after the security operation semantic fusion analysis step: The sliding window algorithm is used for timing filtering and debouncing to continuously verify the risk of the initially determined violation. Only when the risk of the violation persists within the continuous time range set by the sliding window algorithm is it confirmed as a valid violation event and output as a deduction event to the application logic and control layer.

[0014] By adopting the above technical solution and continuously verifying the initial risk assessment, false alarms caused by instantaneous fluctuations in image recognition, accidental gestures, or brief mis-entry into the area are effectively filtered out. This ensures that every confirmed violation event has high reliability and accuracy, thereby avoiding unfairness to candidates' scores due to system misjudgments and improving the fairness of the scoring results and the robustness of the system.

[0015] A further setting is that, in the perception and monitoring layer, the process of identifying the tool's state specifically includes: Multi-target parallel detection is performed on the video stream of the examination room camera using a pre-set visual model, and tool targets and hand targets in the scene are identified and located simultaneously. Dynamic Holding Status Determination: For the identified tool target, obtain its detection box coordinates; simultaneously, obtain the detection box coordinates of all identified hand targets; calculate the spatial overlap relationship between the detection box of the tool target and the detection box of each hand target; if there is an overlapping area between the detection box of at least one hand target and the detection box of the tool target, it is determined that the tool target is currently held or in use; if there is no overlapping area between the detection box of the current tool target and the detection boxes of all hand targets, a standardization check of the tool target's placement is triggered, and then it is determined whether the detection box of the tool target is completely within the detection box of the corresponding preset tool area storage area; if so, it is determined to be a standard placement; otherwise, a detection event indicating that the tool target is improperly placed is generated and output as a deduction event to the application logic and control layer.

[0016] By adopting the above technical solution, the spatial overlap between the tool and the hand is detected by a visual model to determine the tool status, and the tool can be automatically checked for proper return to its original position. This enables refined monitoring of the operator's tool usage habits and incorporates operational requirements such as "returning tools to their original position after use" into the automatic assessment system. This helps cultivate good professional habits in students. At the same time, the automated deduction of points reduces the workload of invigilators who need to repeatedly patrol and check, making the process evaluation more objective and consistent.

[0017] A further setting involves, within the perception and monitoring layer, receiving and parsing connectivity data from the hardware connection monitoring module, and comparing it against the structured question rule file corresponding to the current question loaded from the question generation and rule configuration module to determine the terminal connection status. This process specifically includes: The system receives continuity data from the hardware connection monitoring module. The continuity data is a matrix representing the pairwise continuity relationships between terminals. The continuity data is parsed row by row to obtain a continuity relationship list between each terminal and other terminals. Based on the continuity relationship list, a merging algorithm is used to aggregate all terminals that can be connected to each other into several non-overlapping terminal sets. Each terminal set is defined as a detection connectivity group, representing a group of terminals that are physically located at the same electrical node at the current moment. When an assessment or training session begins, predefined connection rules and interchangeable terminal pair rules are loaded from the structured question rule file corresponding to the current question. The connection rules include a standard connection group list, which is a two-dimensional array. Each element of the standard connection group list represents a standard connection group, corresponding to a complete and correct circuit connection, and is represented by the terminal numbers that the circuit connection should contain. The interchangeable terminal pair rules define two sets of terminals that are logically equivalent and allow interchangeable connections in pairs. Based on the interchangeable terminal pair rules, the terminal numbers in the detection connectivity group or the standard connection group list are verified and mapped for equivalence. For each of the detected connected groups: Iterate through each standard connection group in the list of standard connection groups, and use a set operation algorithm to determine whether the detected connection group is a subset of the standard connection group. If there exists at least one standard connection group such that the detected connection group is a subset of it, then the detected connection group is determined to be a correct connection. If, after iterating through all standard connection groups, the detected connection group is not a subset of any standard connection group, then the detected connection group is determined to be an incorrect connection. The detection connectivity groups of all correct connections generated during the determination process are summarized into a correct result set, and the detection connectivity groups of all incorrect connections are summarized into an incorrect result set. The correct result set and the incorrect result set, along with the terminal numbers involved in the incorrect connections, are sent to the application logic and control layer as the terminal connection status determination results.

[0018] By adopting the above technical solution, accurate, efficient and automated judgment of complex circuit wiring results is achieved. It can not only judge whether the connection is correct or not, but also identify specific incorrect connection groups, providing a direct basis for accurate scoring and error location. It completely changes the inefficient and error-prone method of relying on manual multimeter measurement or visual inspection, ensuring the absolute objectivity and high accuracy of the wiring scoring process.

[0019] A further setting includes a security alarm and visual audit module integrated into the local assessment terminal. The security alarm and visual audit module is controlled by the perception and monitoring layer to trigger a highly visible security warning pop-up. The security alarm and visual audit module is configured to continuously record at least the deduction events during the assessment process in chronological order, and associate and store each deduction event with a set of automatically captured on-site screenshots to form a traceable time-series event log.

[0020] By adopting the above technical solutions, real-time and highly visible warnings of safety risks during the assessment process are achieved, which can effectively remind candidates to stop dangerous behaviors. The deduction events are linked and stored with on-site screenshots to form a time-series log, which constructs a complete, intuitive and tamper-proof evidence chain of the assessment process, greatly enhancing the traceability and credibility of the assessment results, and providing a basis for score review, teaching review or dispute resolution.

[0021] A further setting is that the automatic capture of the on-site screenshot set is achieved through an intelligent screenshot algorithm, which specifically includes: A circular video frame buffer is set up to continuously retain the most recent video frame sequence of a preset duration. When the application logic and control layer triggers a deduction event, the current video frame at the time of the deduction event is saved. At the same time, the video frames at the predetermined time points before and after the deduction event are automatically retrieved from the circular buffer and combined to form a set of on-site screenshots.

[0022] By adopting the above technical solution, key video frames before, during, and after a point deduction event are captured back through a circular buffer. This allows for the dynamic and complete capture of the continuous process of a violation or error, rather than a single instantaneous image. This provides more convincing and contextual visual evidence, further enhancing the credibility of audit logs and the value of teaching feedback.

[0023] A further feature is that the local assessment terminal also includes a remote examination administration communication layer, which forms a communication connection with the remote management service terminal.

[0024] By adopting the above technical solutions, assessment data can be uploaded and aggregated in real time, facilitating big data analysis and teaching quality evaluation. At the same time, administrators can also remotely issue unified instructions (such as unified start and end of exams), which greatly facilitates the organization and supervision of large-scale, multi-exam-room synchronous assessments.

[0025] This invention also provides an intelligent practical assessment method based on multimodal perception, comprising the following steps: S100. Through the question generation and rule configuration module, define circuit connection logic rules including connection rules and interchangeable terminal pair rules. After integrity verification, generate a structured question rule file and store it in the question rule library. S200: The examinee completes identity verification and login at the human-computer interaction layer of the local assessment terminal, and then selects the assessment item and starts the assessment process through the human-computer interaction layer; the local assessment terminal loads the corresponding structured test question rule file according to the selected assessment item, and establishes a communication connection with the examination room camera and the hardware connection monitoring module. S300. After the assessment process is initiated, the perception and monitoring layer of the local assessment terminal simultaneously performs video stream analysis and hardware status acquisition. Among them, the video access module processes the video stream from the examination room camera in real time, identifies air switch targets, hand targets, and tool targets through a visual model, and analyzes the operation behavior and tool placement status based on the target status and spatial position relationship to generate detection result data. At the same time, the terminal block monitoring module periodically receives and parses the conduction relationship data uploaded by the hardware connection monitoring module and determines the terminal connection status. S400: The application logic and control layer of the local assessment terminal receives and integrates the detection result data and terminal connection status from the perception and monitoring layer; and performs automatic scoring based on preset scoring rules through multimodal data fusion. S600. When the assessment ends, the application logic and control layer terminates the video stream analysis and hardware status acquisition, and summarizes and calculates the final assessment score; the local assessment terminal uploads the assessment data to the remote management service terminal through the remote examination communication layer.

[0026] In summary, the present invention has the following beneficial effects: by coordinating the question generation and rule configuration module, the local assessment terminal, the hardware connection monitoring module, and the remote management service terminal, it utilizes a visual model to identify operational behaviors and tool statuses, and combines the conduction relationship data collected by the hardware to achieve multimodal perception, automated monitoring, and accurate scoring of practical assessments for electrical engineering training, thereby improving the objectivity, efficiency, and traceability of the assessment. Attached Figure Description

[0027] Figure 1 This is a system architecture diagram of an embodiment of the example; Figure 2 This is a schematic diagram of the configuration interface of the question generation and rule configuration module in the embodiment; Figure 3 This is the account and password login interface in the human-computer interaction layer of the embodiment; Figure 4 This is the account face login interface in the human-computer interaction layer of the embodiment; Figure 5 This is the registration interface in the human-computer interaction layer of the embodiment; Figure 6 This is the main interface of the local assessment terminal in the embodiment; Figure 7 The example shows a security alert pop-up window issued by the security alarm and visual audit module in the human-computer interaction layer of the embodiment. Figure 8 This is the score interface in the human-computer interaction layer of the embodiment; Figure 9 This is a screenshot taken after a point deduction event is triggered in the human-computer interaction layer of the embodiment. Figure 10 The example shows the performance curve interface in the human-computer interaction layer. Detailed Implementation

[0028] The present invention will be further described in detail below with reference to the accompanying drawings.

[0029] like Figures 1-10 As shown; This embodiment discloses an intelligent practical assessment system based on multimodal perception, targeting electrical engineering training assessment scenarios. It adopts a distributed collaborative architecture and integrates deep learning object detection and embedded hardware acquisition technologies. It mainly consists of four core components: a question generation and rule configuration module, a local assessment terminal, a hardware connection monitoring module, and a remote management service terminal. Example 1

[0030] Deploying the system hardware environment includes: Server-side deployment: Deploy a high-performance host below the console, running a Windows operating system, and configured with a multi-core CPU and a high-performance graphics accelerator card (GPU) to run login management services, database services, and core visual inference services.

[0031] Examination platform terminal configuration: The host connects to a high-definition wide-angle industrial camera (i.e., examination room camera) via a USB 3.0 interface. The camera is installed 70-80 cm directly above the operating platform, with the lens pointing vertically downwards, covering the entire electrical installation base plate.

[0032] Hardware Connection Monitoring Module: The testing platform integrates a self-developed 200-channel IO matrix acquisition board. The acquisition board connects in parallel to the terminals of electrical components such as circuit breakers, contactors, relays, and push-button switches on the training platform via a customized cabling harness. The acquisition board communicates with the industrial control computer via an RS485-to-USB interface, reporting the connection status between terminals in real time. Specifically, the acquisition board does not only collect single-point switch statuses, but rather targets the detection of "conduction relationships between terminals." It performs matrix sampling of the pairwise conduction states between all connected terminals and organizes the sampling results using bitmap compression, allowing for rapid reconstruction of connection relationships and connectivity information between terminals on the host computer. The acquisition board also supports standby / detection / running modes to adapt to the switching needs between acquisition and running states during examinations / training. The data acquisition board is connected in parallel to the terminals of electrical components such as circuit breakers, contactors, relays, and push-button switches on the training platform via a customized wiring harness. Each monitored physical terminal corresponds to one I / O channel of the acquisition board and is assigned a unique "terminal number." The continuity matrix returned by the acquisition board is also encoded according to the terminal number sequence. To facilitate human-machine interaction and consistency with drawings, the system can also maintain a mapping relationship of "terminal number - terminal name / identifier" (e.g., 5-FU1) in the rule configuration. The terminal number is used for set operations and verification during algorithm calculations, and is also used for interface display or error location. The acquisition board communicates with the industrial control computer via an RS485 to USB interface, reporting the connection status between terminals in real time according to a customized communication protocol, providing a data foundation for connection verification and error identification.

[0033] The question generation and rule configuration module allows teachers to pre-define and manage circuit connection logic rules. It generates and manages structured question rule files containing connection rules and interchangeable terminal pair rules, and stores these structured question rule files in the question rule library. Specifically, it includes the following functions: Visual rule editor: Provides a graphical interface for teachers to define circuit connection logic rules. By inputting the terminal numbers on the terminal block, "standard connection groups" and "interchangeable terminal pairs" can be set. At the same time, the position of each terminal can be marked visually on circuit diagrams such as star-delta circuits.

[0034] Dynamic question generation: Supports modification and derivation of existing structured question rule files to create new questions; Rule integrity verification: A built-in logic checker automatically verifies the consistency of the structured question rule file before saving the questions, preventing the same terminal from appearing in multiple connections or undefined terminals from appearing. Rules that pass the verification are then generated as a structured question rule file.

[0035] These structured test rule files constitute the test rule library, which is selected and loaded by the local testing terminal according to the project name during the assessment process, and is used to drive subsequent switchboard monitoring and automatic scoring.

[0036] The local assessment terminal comprises a human-computer interaction layer, a perception and monitoring layer, and an application logic and control layer. Specifically: The human-computer interaction layer provides an intuitive visual interface. The upper right corner of the main interface displays the student's avatar, name, and student ID; the left side displays the real-time camera feed; and the right side is the control panel, providing options for selecting assessment items, setting assessment duration, and selecting "Start Assessment," "Start Training," "Stop," "Clear Log," and "View Results." All user operations and system internal state changes are recorded chronologically in the event log area for easy monitoring and traceability. Within the human-computer interaction layer, invigilators can select specific assessment items from a pre-configured question rule library. The system loads the corresponding structured question rule file based on the selected item and displays key information such as the current question name on the interface. The "View Results" entry on the right side of the main page leads to the assessment results interface, which centrally displays the total score, deduction details, and related screenshots for single and historical assessments / training sessions. A further window displays the assessment / training performance curve, plotting the student's performance changes over multiple assessments / training sessions with time on the horizontal axis and score on the vertical axis, providing graphical support for teaching evaluation and effectiveness analysis.

[0037] The application logic and control layer is responsible for the assessment business logic and status management, including: assessment / training process control, start, end and other status switching; assessment duration and assessment items management, and loading the corresponding structured test question rule file according to the selected items, executing automatic scoring logic according to the rules pre-configured by the teacher, and deducting points from the video access module and junction box monitoring module; storing the scores, deduction details, related screenshots and statistical information of each assessment or training session, and displaying them in the score viewing window.

[0038] The perception and monitoring layer is responsible for real-time perception of examinee behavior and wiring status, including a video access module and a junction box monitoring module. The video access module receives and analyzes the video stream from the examination room camera in real time to identify examinee operation behavior and tool status, generating detection result data. Specifically, it performs multi-target parallel detection on the video stream from the examination room camera using a preset visual model, simultaneously identifying and locating air switch targets and hand targets in the scene. For the identified air switch targets, the visual model further distinguishes whether they are in an on or off state. Semantic fusion analysis of safe operation is performed: when both an air switch target and a hand target are identified as being in an on state in the same video frame within a preset area, or based on the detection results of multiple consecutive video frames, a preliminary judgment is made that there is a risk of illegal operation. The junction box monitoring module receives and parses the continuity relationship data collected from the hardware connection monitoring module, compares it with the structured test question rule file corresponding to the current test question loaded from the question generation and rule configuration module, and determines the terminal connection status. Specifically, it performs multi-target parallel detection on the video stream from the examination room camera using a preset visual model, simultaneously identifying and locating tool targets and hand targets in the scene. Next, dynamic holding status determination is performed: for the identified tool target, its detection box coordinates are obtained; simultaneously, the detection box coordinates of all identified hand targets are obtained; the spatial overlap relationship between the detection box of the tool target and the detection box of each hand target is calculated; if there is an overlapping area between the detection box of at least one hand target and the detection box of the tool target, it is determined that the tool target is currently in a held or used state; if there is no overlapping area between the detection box of the current tool target and the detection boxes of all hand targets, a standardization check of the tool target's placement is triggered, and then it is determined whether the detection box of the tool target is completely within the detection box of the corresponding preset tool area storage area; if so, it is determined to be a standard placement; otherwise, a detection event indicating that the tool target is placed improperly is generated and output as a deduction event to the application logic and control layer. Example 2

[0039] The implementation of the visual analysis logic of the video access module includes: Data Acquisition and Model Training: A dedicated dataset containing thousands of images of real-world training scenes was constructed, covering different lighting conditions, hand angles, and tool types. The images in the dedicated dataset were finely labeled, with categories including: "circuit breaker on," "circuit breaker off," "hand," "screwdriver," "pliers," and "other tools." The visual model uses the lightweight object detection network Yolov8, and its generalization ability and robustness are improved through training and data augmentation techniques.

[0040] Real-time reasoning and logical operations: First, extract all detection boxes for "air switches" and filter out targets with a confidence level greater than the threshold and a category of "air switch on". Next, extract all detection boxes and key points for "hands". When both "air switch on" and "hand" are detected in the designated area, a deduction event is triggered, and the violation is judged as "operation when main switch is on". Next, iterate through all "tool" category targets. For each tool, check whether its detection box overlaps with any "hand" target detection box. If there is overlap, the tool is considered to be in "use" state; if there is no overlap, the tool is considered to be in "out of hand" state. Then, determine whether the detection box of the tool target is completely within the detection box of the corresponding preset tool storage area; if so, it is judged as proper placement; otherwise, it is immediately judged as "tool placed on tabletop" violation.

[0041] Furthermore, after initially determining the existence of a risk of violation, time-series filtering and de-jittering processing are performed based on the sliding window algorithm to continuously verify the initially determined risk of violation. Only when the risk of violation continues to exist within the continuous time range set by the sliding window algorithm is it confirmed as a valid violation event and output as a deduction event to the application logic and control layer.

[0042] The local assessment terminal includes a remote examination administration communication layer and a security alarm and visual audit module. It establishes a communication connection with the remote management service terminal based on the remote examination administration communication layer. The remote management service terminal can receive and centrally store data from each local assessment terminal and issue control commands to each local assessment terminal. The security alarm and visual audit module is controlled by the perception and monitoring layer to trigger highly visible security warning pop-ups. The security alarm and visual audit module is configured to continuously record at least one deduction event during the assessment process in chronological order, and associate each deduction event with an automatically captured set of on-site screenshots to form a traceable time-series event log. The automatic capture of the on-site screenshot set is achieved through an intelligent screenshot algorithm. Specifically, the intelligent screenshot algorithm sets up a circular video frame buffer, continuously retaining a sequence of video frames of a preset duration. When the application logic and control layer triggers a deduction event, it saves the current video frame at the time of the deduction event trigger, and simultaneously automatically retrieves video frames from the circular buffer at predetermined time points before and after the deduction event, forming the on-site screenshot set.

[0043] The hardware connection monitoring module connects to the terminal block being assessed and is used to collect real-time continuity data between the terminals on the terminal block. Specifically, the hardware connection monitoring module acquires the continuity data between the terminals, encodes it according to a preset data frame format, includes a timestamp, packages it, and sends it to the local assessment terminal.

[0044] The sensing and monitoring layer within the local assessment terminal receives conduction relationship data from the hardware connection monitoring module. This conduction relationship data is a matrix representing the pairwise conduction relationships between each terminal. The conduction relationship data is parsed row by row to obtain a list of conduction relationships between each terminal and other terminals. Based on this list, a merging algorithm is used to aggregate all terminals that can conduct to each other into several non-overlapping terminal sets. Each terminal set is defined as a detection connectivity group, representing a group of terminals that are physically located at the same electrical node at the current moment. When an assessment or training session begins, predefined connection rules and interchangeable terminal pair rules are loaded from the structured question rule file corresponding to the current question. The connection rules include a standard connection group list, which is a two-dimensional array. Each element of the standard connection group list represents a standard connection group, corresponding to a complete and correct circuit connection, and is represented by the terminal numbers that the circuit connection should contain. The interchangeable terminal pair rules define two sets of terminals that are logically equivalent and allow interchangeable connections in pairs. Based on the interchangeable terminal pair rules, the terminal numbers in the detection connectivity group or the standard connection group list are verified and mapped for equivalence. For each of the detected connected groups: Iterate through each standard connection group in the list of standard connection groups and use set operation algorithms to determine whether the detected connection group is a subset of the standard connection group. If there exists at least one standard connection group such that the detected connection group is a subset of it, then the detected connection group is determined to be a correct connection. If, after iterating through all standard connection groups, the detected connection group is not a subset of any standard connection group, then the detected connection group is determined to be an incorrect connection. The detection connectivity groups of all correctly connected connections generated during the determination process are summarized into a correct result set, and the detection connectivity groups of all incorrect connections are summarized into an incorrect result set. The correct result set, the incorrect result set, and the terminal numbers involved in the incorrect connection are sent to the application logic and control layer as the terminal connection status judgment result. Example 3

[0045] After the assessment / training begins, the system loads the standard connection group data required for verification from the test rule configuration file rules.json. The connection_rules.connections file in the test rule configuration file describes the list of standard connection groups: its data format is a two-dimensional array. Each element of the outer array corresponds to a complete and valid standard line, and the inner array is the set of terminals contained in that line. For example, [17, 142] indicates that terminals 17 and 142 should be in the same valid connection, and [179, 64, 79, 97, 68] indicates that terminals 179, 64, 79, 97, and 68 together constitute a valid line. The system directly maintains this structure as a List[List[int]] rule table in memory, and during comparison, converts the currently traversed inner terminals into a set to support set operations, thus ensuring that each standard line corresponds to a set of terminal numbers in memory. The system then periodically reads terminal continuity information from the terminal block test board. The continuity status returned by the hardware can be equivalently viewed as a matrix of pairwise continuity relationships between terminals. This matrix is ​​parsed row by row to obtain the continuity relationship between any terminal and other terminals. All terminals that are "mutually conductive" are merged to form several non-overlapping terminal sets, which serve as the "detected connectivity group" for the current moment. Each set represents a group of terminals that are physically located at the same electrical node. After completing the construction of the detected connectivity group, the system performs a subset matching check based on set theory: for each detected connectivity group... Iterate through each standard join group in the standard join group. Determine whether the set containment relationship is satisfied. If any standard connection group exists such that the detected connectivity group is a subset of it, then the detected connectivity group is determined to be within the complete set of terminals of any valid line, is considered a correct connection, and is recorded in the correct result set; if none of the standard groups satisfy the condition after traversing all standard groups, then the connection is considered correct. In such cases, the detected connectivity group is determined not to belong to any legitimate line range, is considered an incorrect connection, and is recorded in the error result set. Example 4

[0046] The implementation of evidence retention and report generation includes: The intelligent screenshot algorithm maintains a circular video frame buffer in memory, retaining the most recent 5 seconds of video frames. When the scoring engine triggers a deduction event, the system not only captures the current real-time frame but also automatically backtracks and extracts key frames from 0.5 seconds before and after the event. This ensures a complete record of the violation, guaranteeing the integrity and persuasiveness of the evidence.

[0047] Automated Report Generation: After the assessment, the report generation module reads the candidate's score data file. The module iterates through all deduction events in chronological order, extracting event descriptions and deduction values. Using a PDF formatting engine, it automatically combines the candidate's basic information, total score, and detailed violation record table into a complete PDF document. Example 5

[0048] The implementation of the assessment results display interface includes: In this embodiment, the system provides a separate "Assessment Result Display" window, which is used to centrally present the results of a single assessment or training session, and supports querying and statistics of historical records.

[0049] Data Loading and Window Launch: After the assessment or training, invigilators can open the score viewing window through the "View Score" entry on their local assessment terminal. This window loads data from the result file output by the scoring module, including but not limited to: candidate basic information, assessment items, mode type (assessment / training), total score, deduction records, related screenshots, and summary statistics.

[0050] Overall Interface Layout: The assessment results window uses a multi-area layout, displaying different information in separate sections. The top basic information area displays the candidate's name, student ID, assessment item, assessment mode, and start and end times for the currently selected record. The assessment statistics area displays the total number of assessments / training sessions and the average score, and also includes "View Assessment Score Curve" and "View Training Score Curve" buttons. Clicking the corresponding button will take you to the score curve interface. The total section displays the total score for the current assessment / training session.

[0051] Intermediate Deduction Details Area: This section displays the deduction items for the current assessment / training session in tabular format, with each row representing a deduction event. It includes fields for deduction time, reason for deduction, deduction score, deduction details, and a screenshot to view. Each row's screenshot column provides a "Click to View" button to open the corresponding violation screenshot.

[0052] The bottom history section displays the candidate's past assessments / training information in a table format, with each row representing one assessment / training event. It includes fields such as start time, total time, mode, assessment items, reason for deduction, number of deducted items, total score, and a screenshot view. Clicking the "Click to View" button in the "View Images" column allows you to see screenshots of all deduction events in the selected assessment / training session, facilitating invigilators' post-examination review, teaching debriefing, and dispute resolution.

[0053] In this embodiment, the assessment result display interface allows the system to present the structured data output by the scoring module to the user in a visual manner, thus achieving a clear display of the results of a single assessment. The system enables centralized management and rapid retrieval of multiple assessment records; and combines textual and graphical analysis of key deduction incidents for evidence playback. This provides a complete means of presenting results and conducting post-assessment reviews for electrical engineering training assessments, enhancing the transparency and traceability of the assessment process. Example 6

[0054] The implementation of the performance curve display window includes: The system provides a separate "Performance Curve Display" window, which graphically presents the performance trend of the same candidate in different assessments or training sessions, assisting teachers in evaluating effectiveness and providing teaching feedback.

[0055] Data Source and Window Launch: After the assessment or training concludes and results are recorded, a score curve window can be opened on the assessment terminal as needed. This window reads historical score records for a specified candidate and mode from the assessment results storage module, obtaining the start time and corresponding total score for each record, which serves as the basis for plotting the curve.

[0056] Interface Layout and Information Area: The top information bar displays the name of the currently selected student, the score type (assessment or training), and the number of records loaded. It also provides statistical information such as the average score, highest score, and lowest score for this batch of records, helping teachers quickly grasp the overall level. The middle section is a graph display area, using time as the horizontal axis and score as the vertical axis. It uses a smooth curve combined with data points to show the student's score trend over time, and overlays an average score reference line and grid on the graph for easy observation of fluctuations. The bottom section contains operation buttons, providing basic operations such as data refresh, chart export, and closing the window.

[0057] Curve Plotting and Trend Analysis: The curve plotting module sorts the retrieved historical records chronologically, maps the start time of each assessment or training session to the horizontal axis, maps the corresponding total score to the vertical axis, and generates a continuous trend curve through interpolation and smoothing algorithms. Data Refresh and Chart Export: To facilitate long-term tracking and archiving, the performance curve window in this embodiment also provides data refresh and chart export functions: When new assessment or training results are generated, users can click the "Refresh Data" button, and the window will reload the latest historical records from the performance storage module and automatically update the curve and statistical information; users can use the "Export Chart" button to export the currently displayed performance curve as an image or document to a local file for insertion into teaching documents, performance reports, or as assessment data for archiving.

[0058] Through the above implementation, the performance curve display window in this embodiment can transform discrete performance records into an intuitive time series trend chart, which not only retains numerical statistical information but also provides a graphical overall perspective, which is conducive to teachers analyzing and judging the dynamic changes in the students' practical training level. Example 7

[0059] The implementation of the question generation and rule configuration module includes: The rule visualization editing tool interface is mainly divided into the "interchangeable terminal pair" configuration area, the "standard connection group" configuration area, and the "circuit diagram point annotation area". In the interchangeable terminal pair configuration area, teachers define an interchangeable terminal pair rule by inputting text such as "61, 70; 64, 73". The two sets of terminals [61, 70] and [64, 73] are considered functionally equivalent during scoring, but cross-wiring (such as [61, 73] or [64, 70]) is prohibited. In the standard connection group configuration area, teachers define a set of terminals that must be connected by inputting a terminal number sequence (such as 5, 9, 15). The system automatically converts this into an internal connection list. In the circuit diagram point labeling area, star-delta and other circuit diagrams are automatically loaded, providing a visual point labeling function on the image. Teachers can select a terminal from the terminal list and freely click or drag and drop it at the corresponding position on the diagram to generate a label with the terminal number. Drag-and-drop fine-tuning, deletion, and re-labeling are supported. The system saves the terminal number corresponding to each label and its relative coordinates on the image to the test question file. When the same test question is reopened, the original point layout can be completely restored, thus achieving integrated management of rule configuration and circuit diagram labeling.

[0060] Intelligent Assistance and Verification: When a teacher enters a terminal number, the system first performs a basic format check on the input, such as whether it is a valid number. When the teacher clicks to save the test questions, the system performs a global consistency check on the entire set of rules. This mainly includes: traversing all standard connection groups and interchangeable terminal pairs to check if the same terminal is assigned to multiple non-connection groups or multiple terminal pairs, preventing the same terminal from belonging to multiple lines simultaneously; at the same time, comparing the input number with the current hardware terminal definition table to indicate undefined terminal numbers and prevent obviously erroneous data from entering the rule set. Once the above problems are detected, the system will mark the relevant rule item on the interface and pop up a prompt message, terminating the current save operation and requiring the teacher to correct the conflicting configuration before generating the test question file, thereby avoiding the formation of self-contradictory or unexplainable assessment rules.

[0061] Question file generation and loading: After verification, the tool saves the interchangeable terminal pairing rules and standard connection group information into a unified JSON question configuration file, stored in a preset rule base directory. Before the assessment or training begins, the local assessment terminal selects and loads the target question file from the rule base. The terminal block monitoring and scoring logic then runs according to the rules in the file, thus flexibly switching between different questions on the same hardware platform and achieving the scalability of "one set of hardware, corresponding to multiple sets of rules".

[0062] This embodiment also discloses an intelligent practical assessment method based on multimodal perception, including the following steps: S100. Through the question generation and rule configuration module, define the circuit connection logic rules, which include connection rules and interchangeable terminal pair rules. After integrity verification, generate a structured question rule file and store it in the question rule library. S200. The examinee completes identity verification and login at the human-computer interaction layer of the local assessment terminal, and then selects the assessment item and starts the assessment process through the human-computer interaction layer; the local assessment terminal loads the corresponding structured test question rule file according to the selected assessment item, and establishes a communication connection with the examination room camera and the hardware connection monitoring module. S300. After the assessment process is initiated, the perception and monitoring layer of the local assessment terminal simultaneously performs video stream analysis and hardware status acquisition. Among them, the video access module processes the video stream from the examination room camera in real time, identifies air switch targets, hand targets, and tool targets through a visual model, and analyzes the operation behavior and tool placement status based on the target status and spatial position relationship to generate detection result data. At the same time, the terminal block monitoring module periodically receives and parses the conduction relationship data uploaded by the hardware connection monitoring module and determines the terminal connection status. S600. When the assessment ends, the application logic and control layer terminates the video stream analysis and hardware status acquisition, and summarizes and calculates the final assessment score; the local assessment terminal uploads the assessment data to the remote management service terminal through the remote examination communication layer.

[0063] This specific embodiment is merely an explanation of the present invention and is not intended to limit the invention. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they are within the scope of the claims of the present invention.

Claims

1. An intelligent practical assessment system based on multimodal perception, characterized in that, It includes a question-generating and rule-configuration module, a local assessment terminal, and a hardware connection monitoring module; among which, The question generation and rule configuration module is used to generate and manage structured question rule files containing connection rules and interchangeable terminal pair rules, and to store the structured question rule files in the question rule library; the hardware connection monitoring module is connected to the terminal block being tested, and is used to collect the conduction relationship data between each terminal in the terminal block in real time; The local assessment terminal includes a human-computer interaction layer, a perception and monitoring layer, and an application logic and control layer. The human-computer interaction layer performs human-computer interaction with the examinee. The perception and monitoring layer includes a video access module and a terminal block monitoring module, used to receive and analyze video streams from the examination room cameras in real time to identify examinee operation behaviors and tool status and generate detection result data. Simultaneously, it receives and parses conduction relationship data collected from the hardware connection monitoring module, compares it with the structured test question rule file corresponding to the current test question loaded from the question generation and rule configuration module, and determines the terminal connection status. The application logic and control layer performs automatic scoring based on multimodal data fusion based on the detection result data and the terminal connection status. It also includes a remote management service terminal for receiving and centrally storing data from each local assessment terminal, and for issuing control commands to each local assessment terminal.

2. The intelligent practical assessment system based on multimodal perception according to claim 1, characterized in that: In the perception and monitoring layer, the process of identifying the examinee's operational behavior specifically includes: The video stream from the examination room camera is subjected to multi-target parallel detection using a preset visual model, which simultaneously identifies and locates air switch targets and hand targets in the scene; among them, for the identified air switch targets, the visual model is used to further distinguish whether they are in the on or off state. Perform semantic fusion analysis for safe operation: When an air switch target and a hand target that are both identified as being in the open state are present in the same video frame in a preset area, or based on the detection results of multiple consecutive video frames, it is preliminarily determined that there is a risk of illegal operation.

3. The intelligent practical assessment system based on multimodal perception according to claim 2, characterized in that: Following the semantic fusion analysis step for safe operations, the following steps are also included: The sliding window algorithm is used for timing filtering and debouncing to continuously verify the risk of the initially determined violation. Only when the risk of the violation persists within the continuous time range set by the sliding window algorithm is it confirmed as a valid violation event and output as a deduction event to the application logic and control layer.

4. The intelligent practical assessment system based on multimodal perception according to claim 1, characterized in that: In the perception and monitoring layer, the process of identifying the tool's state specifically includes: Multi-target parallel detection is performed on the video stream of the examination room camera using a pre-set visual model, and tool targets and hand targets in the scene are identified and located simultaneously. Dynamic Holding Status Determination: For the identified tool target, obtain its detection box coordinates; simultaneously, obtain the detection box coordinates of all identified hand targets; calculate the spatial overlap relationship between the detection box of the tool target and the detection box of each hand target; if there is an overlapping area between the detection box of at least one hand target and the detection box of the tool target, it is determined that the tool target is currently held or in use; if there is no overlapping area between the detection box of the current tool target and the detection boxes of all hand targets, a standardization check of the tool target's placement is triggered, and then it is determined whether the detection box of the tool target is completely within the detection box of the corresponding preset tool area storage area; if so, it is determined to be a standard placement; otherwise, a detection event indicating that the tool target is improperly placed is generated and output as a deduction event to the application logic and control layer.

5. The intelligent practical assessment system based on multimodal perception according to claim 1, characterized in that: In the perception and monitoring layer, the process of receiving and parsing the conduction relationship data from the hardware connection monitoring module, and comparing it with the structured question rule file corresponding to the current question loaded from the question generation and rule configuration module to determine the terminal connection status specifically includes: The system receives continuity data from the hardware connection monitoring module. The continuity data is a matrix representing the pairwise continuity relationships between terminals. The continuity data is parsed row by row to obtain a continuity relationship list between each terminal and other terminals. Based on the continuity relationship list, a merging algorithm is used to aggregate all terminals that can be connected to each other into several non-overlapping terminal sets. Each terminal set is defined as a detection connectivity group, representing a group of terminals that are physically located at the same electrical node at the current moment. When an assessment or training session begins, predefined connection rules and interchangeable terminal pair rules are loaded from the structured question rule file corresponding to the current question. The connection rules include a standard connection group list, which is a two-dimensional array. Each element of the standard connection group list represents a standard connection group, corresponding to a complete and correct circuit connection, and is represented by the terminal numbers that the circuit connection should contain. The interchangeable terminal pair rules define two sets of terminals that are logically equivalent and allow interchangeable connections in pairs. Based on the interchangeable terminal pair rules, the terminal numbers in the detection connectivity group or the standard connection group list are verified and mapped for equivalence. For each of the detected connected groups: Iterate through each standard connection group in the list of standard connection groups, and use a set operation algorithm to determine whether the detected connection group is a subset of the standard connection group. If there exists at least one standard connection group such that the detected connection group is a subset of it, then the detected connection group is determined to be a correct connection. If, after iterating through all standard connection groups, the detected connection group is not a subset of any standard connection group, then the detected connection group is determined to be an incorrect connection. The detection connectivity groups of all correct connections generated during the determination process are summarized into a correct result set, and the detection connectivity groups of all incorrect connections are summarized into an incorrect result set. The correct result set and the incorrect result set, along with the terminal numbers involved in the incorrect connections, are sent to the application logic and control layer as the terminal connection status determination results.

6. The intelligent practical assessment system based on multimodal perception according to claim 1, characterized in that: It also includes a security alarm and visual audit module integrated into the local assessment terminal. The security alarm and visual audit module is controlled by the perception and monitoring layer to trigger a highly visible security warning pop-up. The security alarm and visual audit module is configured to continuously record at least the deduction events during the assessment process in chronological order, and associate and store each deduction event with a set of automatically captured on-site screenshots to form a traceable time-series event log.

7. The intelligent practical assessment system based on multimodal perception according to claim 6, characterized in that: The automatic capture of the on-site screenshot set is achieved through an intelligent screenshot algorithm, which specifically includes: A circular video frame buffer is set up to continuously retain the most recent video frame sequence of a preset duration. When the application logic and control layer triggers a deduction event, the current video frame at the time of the deduction event is saved. At the same time, the video frames at the predetermined time points before and after the deduction event are automatically retrieved from the circular buffer and combined to form a set of on-site screenshots.

8. The intelligent practical assessment system based on multimodal perception according to claim 1, characterized in that: The local assessment terminal also includes a remote examination administration communication layer, which forms a communication connection with the remote management service terminal.

9. A multimodal perception-based intelligent practical assessment method, applied to the multimodal perception-based intelligent practical assessment system as described in any one of claims 1 to 8, characterized in that, Includes the following steps: S100. Through the question generation and rule configuration module, define circuit connection logic rules including connection rules and interchangeable terminal pair rules. After integrity verification, generate a structured question rule file and store it in the question rule library. S200: The examinee completes identity verification and login at the human-computer interaction layer of the local assessment terminal, and then selects the assessment item and starts the assessment process through the human-computer interaction layer; the local assessment terminal loads the corresponding structured test question rule file according to the selected assessment item, and establishes a communication connection with the examination room camera and the hardware connection monitoring module. S300. After the assessment process is initiated, the perception and monitoring layer of the local assessment terminal simultaneously performs video stream analysis and hardware status acquisition. Among them, the video access module processes the video stream from the examination room camera in real time, identifies air switch targets, hand targets, and tool targets through a visual model, and analyzes the operation behavior and tool placement status based on the target status and spatial position relationship to generate detection result data. At the same time, the terminal block monitoring module periodically receives and parses the conduction relationship data uploaded by the hardware connection monitoring module and determines the terminal connection status. S400: The application logic and control layer of the local assessment terminal receives and integrates the detection result data and terminal connection status from the perception and monitoring layer; and performs automatic scoring based on preset scoring rules through multimodal data fusion. S600. When the assessment ends, the application logic and control layer terminates the video stream analysis and hardware status acquisition, and summarizes and calculates the final assessment score; the local assessment terminal uploads the assessment data to the remote management service terminal through the remote examination communication layer.

Images