Skill behavior regulation evaluation system based on multi-modal perception and dynamic rule engine

Abstract

The present application belongs to the technical field of skill behavior specification examination and evaluation of nuclear power plants, and particularly relates to a skill behavior specification examination and evaluation system based on multi-modal perception and a dynamic rule engine. The system comprises a multi-modal perception module and a dynamic rule engine module. The multi-modal perception module collects and pre-processes multi-dimensional data to provide original input for compliance determination of the dynamic rule engine module. The dynamic rule engine module constructs a multi-layer rule base based on skill behavior specification examination and evaluation operations in the nuclear power field, and realizes dynamic updating of rules through scene self-adaptation and data driving. The beneficial effects are as follows: the multi-modal perception fusion technology is adopted to realize full-dimensional accurate monitoring of operation behavior; through fusion of visual and voice multi-modal data, the problem of single mode being easily disturbed by environment and information missing is solved; the coverage rate of operation behavior monitoring is improved to 99.2%, and the missing judgment rate of key steps is reduced to below 0.3%.

Description

Technical Field

[0001] This invention belongs to the technical field of skills and behavior norms assessment and evaluation in nuclear power plants, specifically involving a skills and behavior norms assessment system based on multimodal perception and dynamic rule engine. Background Technology

[0002] The reliance of nuclear power unit operation safety on the standardized behavior of operators is becoming increasingly prominent. Statistics from the International Atomic Energy Agency (IAEA) show that human error accounts for as much as 60% of unsafe incidents at nuclear power plants, with skill-related errors (such as omissions in operating procedures and misjudgments of equipment status) accounting for over 50%. These errors are characterized by their high frequency, high concealment, and significant harm. In recent years, with the rapid development of nuclear power technology, higher requirements have been placed on the skills and behavioral standards of operators to ensure the safety and stability of nuclear power plant operations.

[0003] Existing skills and behavior assessment systems primarily rely on single data sources such as manual observation or paper records, lacking effective integration of multi-dimensional sensory data including visual, auditory, and motor perception. Such systems typically solidify evaluation criteria in code or paper documents, making them ill-suited to the dynamic needs of complex scenarios. Specifically: 1. Data collection is limited to a single dimension, resulting in incomplete information coverage. Limitations of manual observation: Invigilators' visual observation is easily affected by factors such as distraction and fatigue, leading to omissions in key operational steps, and subjective scoring standards are inconsistent.

[0004] Multi-task observation bottleneck: There are multiple operating areas in the examination room (such as integrated circuits), and invigilators cannot cover the entire area (such as observing the control panel and on-site operations at the same time), which easily leads to blind spots in observation.

[0005] No intelligent early warning mechanism: It is impossible to proactively trigger intervention measures (such as suspending the assessment or prompting correction) through real-time data analysis (such as the frequency of continuous misoperations or failure to execute key operation steps), and it relies on invigilators to passively discover problems.

[0006] 2. Rigid rule management with insufficient dynamic adaptability. Insufficient rule adaptability: The rigid rules preset by traditional systems are difficult to cope with the needs of complex scenarios, and when adding new operation modules, the rule base needs to be manually recoded, which is inefficient and prone to human error.

[0007] Rule maintenance relies on manual intervention: operations such as adjusting evaluation criteria and optimizing thresholds depend heavily on manual intervention by domain experts, making it impossible to achieve automated rule evolution through data-driven approaches, resulting in rule base updates lagging behind actual needs.

[0008] Rigid logical combinations: It cannot adapt to the refined evaluation requirements of complex operation scenarios. In the assessment of complex skills such as multi-position collaboration and trap handling, it is easy to have problems such as missed judgments, misjudgments, or incomplete rule coverage. Moreover, rule adjustment requires manual reconstruction of the underlying code logic, resulting in extremely high maintenance costs. Summary of the Invention

[0009] The purpose of this invention is to provide a skills behavior standard assessment system based on multimodal perception and dynamic rule engine. Through multimodal data collection, it can comprehensively capture various information in the operation process. Combined with dynamic rule adaptation technology, it can flexibly respond to the needs of different operation scenarios and make real-time judgments, thereby effectively solving the shortcomings of the existing technology. It not only improves the accuracy of compliance judgment of skills behavior standard operation, but also corrects deviations in operation in a timely manner, and enhances the effectiveness of training and assessment.

[0010] The technical solution of the present invention is as follows: a skills behavior norm assessment system based on multimodal perception and dynamic rule engine, including a multimodal perception module and a dynamic rule engine module. The multimodal perception module provides raw input for the compliance judgment of the dynamic rule engine module by collecting and preprocessing multi-dimensional data. The dynamic rule engine module constructs a multi-layer rule base based on the assessment and evaluation operation of skills behavior norms in the nuclear power field, and realizes dynamic updating of rules through scenario adaptation and data-driven approach.

[0011] The multimodal perception module includes a visual perception module and a voice perception module. The visual perception module includes a hand motion recognition module, a posture tracking recognition module, an equipment nameplate recognition module, an equipment status recognition module, and a carpet grid recognition module, realizing hand motion recognition, posture tracking recognition, equipment nameplate recognition, equipment status recognition, and carpet grid recognition. The voice perception sub-module includes a voice noise reduction and separation module, a voiceprint recognition and role differentiation module, a semantic analysis and text matching module, and a large model structured parsing module, realizing voice noise reduction and separation, voiceprint recognition and role differentiation, semantic analysis and text matching, and large model structured parsing.

[0012] The hand gesture recognition module extracts the coordinates of 21 key points of the hand using the MediaPipe Hands framework, optimizes the gesture classification model by combining it with a self-trained dataset, and identifies hand gestures such as holding tools, pointing at equipment, and confirming gestures in nuclear power operations in real time. The process includes: video frame preprocessing: acquiring video streams from a head-mounted camera, normalizing the size of each frame image, and performing illumination compensation processing; key point extraction using MediaPipe Hands: inputting the preprocessed video frames, using the MediaPipeHands model for inference, outputting the three-dimensional coordinates of 21 hand key points, constructing a 63-dimensional feature vector from the coordinates of the 21 key points, and using a 3-layer fully connected network model to output the probability distribution of gesture categories.

[0013] The posture tracking and recognition module uses the YOLO target detection model to extract key points of the human body's main trunk, and combines it with the ByteTrack algorithm to achieve multi-target tracking. It outputs skeletal data in JSON format with 3D coordinates of 14 key points, including: Human body detection: input video frames, model is a lightweight version of YOLOv11-pose, output human body bounding box and confidence score; Human body key point detection: 14 key points including nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles. Each key point includes coordinates, confidence score, and visibility. The 3D coordinates are obtained through monocular depth estimation.

[0014] The equipment nameplate recognition module performs text recognition on the nameplate area using a CRNN+CTC joint model, including: nameplate area localization: a nameplate detection model based on YOLOv11, with model training data being equipment nameplate images collected at the nuclear power plant site; image preprocessing: bounding box detection based on Hough transform, affine transformation correction, tilt angle tolerance ≤30°, illumination compensation, character sharpening, CNN feature extraction using ResNet50, RNN sequence modeling using bidirectional LSTM, CTC decoding using greedydecoding or beam search; post-processing: text standardization, format validation, and structured output in JSON format.

[0015] The equipment status recognition module uses the YOLO target detection model to detect switch positions and indicator light colors in real time. This includes: the equipment status detection model uses YOLOv8 target detection, with training data consisting of nuclear power equipment images; switch status recognition: switch types are classified as two-position switches, three-position switches, and rotary switches; status is classified as on / off / middle position; the recognition method involves detecting the switch area, extracting the switch handle angle features, and outputting status labels from the classifier; indicator light color recognition: colors are classified as red, yellow, green, and no light; the recognition method involves detecting the indicator light area ROI, HSV color space analysis, and color classification; illumination adaptation uses automatic white balance correction; and temporal status analysis: the temporal window consists of 10 consecutive frames of voting, and if the status stability is determined by ≥7 identical frames, the status is confirmed; status changes are recorded with a timestamp and a status value.

[0016] The carpet grid recognition module, based on the YOLO_shoe lightweight model, detects the carpet grid code under the operator's feet, simulating real-time positioning of the operator's precise location within the nuclear island. This includes: grid code design: the coding rule is a combination of numbers and letters, with the coding position at the four corners or center of the grid; YOLO_shoe lightweight detection model: YOLOv8n, input size 320×320, FPS≥30fps, training data is grid carpet images; multi-scale feature fusion: the backbone network is CSPDarknet+SPP module, feature fusion uses FPN; dynamic lighting adaptation: the scene switches between normal lighting and emergency lighting, the method is adaptive exposure adjustment plus color constancy; indoor positioning calculation: the grid code corresponds to a coordinate mapping table, and trajectory smoothing uses Kalman filtering.

[0017] The speech denoising and separation module employs a deep neural network beamforming algorithm to separate effective speech from environmental noise. This includes: multi-microphone array acquisition: an 8-channel circular array with a sampling rate of 16kHz, sampling precision of 16bit, and an SNR requirement of ≥65dB; frequency domain feature extraction: short-time Fourier transform, frame length 25ms, frame shift 10ms, frequency resolution 512-point FFT; DNN beamforming network: a deep beamforming network structure, with multi-channel spectral features as input, a 3D convolution backbone, and a single-channel enhanced speech spectrum as output; training data consisting of ≥100 hours of nuclear power plant site noise samples; and post-processing: spectral subtraction for residual noise suppression, speech activity detection, and automatic gain control.

[0018] The voiceprint recognition and role differentiation module is based on the ECAPA-TDNN model and identifies the role of the voice source in real time. This includes: Voiceprint feature extraction: inputting denoised speech, with features of MFCC plus first-order difference, frame length 25ms, frame shift 10ms; ECAPA-TDNN voiceprint encoder: the backbone network is an Emulated Attention Pyramid, context modeling is a multi-scale TDNN, statistical pooling is Attentive Statistics Pooling, the embedding dimension is 192, and the loss function is AngularProtoNet+AAM-Softmax; Voiceprint registration and comparison: registration involves recording a voiceprint sample of ≥30 seconds during the student's first use, with features stored in a voiceprint embedding vector database, and comparison using cosine similarity calculation with a judgment threshold of 0.7.

[0019] The semantic analysis and text matching module uses the WhisperX model for speech-to-text conversion, combining TF-IDF and semantic similarity calculations to match standard dialogue for operational tasks. This includes: Speech-to-text conversion: The model is Whisper Large V3 (input is denoised speech), outputting timestamped text; text preprocessing: Chinese word segmentation uses Jieba segmentation, English word segmentation uses BERT Tokenizer, noise filtering removes interjections and repetitive words; TF-IDF text matching: The corpus is a standard dialogue library for nuclear power plant operations, vectorized into TF-IDF vectors, similarity calculation uses cosine similarity, with a threshold of Cosine ≥ 0.92; keyword extraction: Equipment number recognition uses regular expression matching, operation command recognition uses verb dictionary matching, parameter extraction uses numerical plus unit pattern matching; structured output: JSON format, containing timestamps, text, and a keyword list.

[0020] The large-scale model structured parsing module integrates LangChain to manage domain prompts. Combined with the fine-tuned large model deployed on the Dify platform, it performs semantic understanding and structured extraction of speech and text, including: LangChain Prompt Management: prompt word template design includes system prompts, task prompts, and output formats; few-shot examples are 3-5 extraction examples; chain-of-thought prompting is used for chain inference; fine-tuning the large model: the pre-trained model is Qwen3.5-9B, the fine-tuning method is LoRA, the training data is a nuclear power operation instruction dataset, and domain adaptation is the injection of nuclear power professional terms; Dify platform deployment: the deployment method is the Ollam inference engine; structured extraction: the output format is JSON Schema, and the fields include instruction_type, device_id, device_name, action, and parameters; initial screening of nuclear power safety rules: the rule base is a nuclear power safety operation prohibition library; conflict detection is the matching of extraction results with the rule base; and alarm triggering is the marking of high-risk operations.

[0021] The dynamic rule engine module is based on the assessment and evaluation of skills and behaviors in the nuclear power field to build a multi-layer rule base, and realizes dynamic updates of rules through scenario adaptation and data-driven approaches. It includes a rule base construction module and a rule matching and scoring module.

[0022] The rule base is constructed based on the core elements and operational procedures of nuclear power skills and behavior standard assessment. It features a three-tiered tree structure: "Basic Rules - Step Rules - Task Rules," achieving full-dimensional rule coverage from single operational elements to complete task chains. The specific hierarchical structure includes: Basic rules, the smallest judgment unit of the rule base, directly corresponding to the smallest indivisible verifiable element in nuclear power operation standards, used to quantify and constrain the most basic and atomic requirements of operational behavior; Step rules, single-step operation judgment rules built upon basic rules, used to verify whether a single operational action meets the standard requirements, requiring simultaneous satisfaction of multiple basic rule combinations. Each step rule corresponds to a specific operational action, and its judgment integrates the verification results of multiple basic rules, forming an "AND / OR" logical combination; Task rules, complete task judgment rules composed of multiple step rules connected in series / parallel according to the nuclear power operation process logic, used to verify the compliance of the entire operational task. Each task rule needs to define the execution order, logical relationship, and final pass conditions of the steps.

[0023] The rule matching and scoring module adopts a technical solution combining structured rule table storage and a rule engine. The rule table, as the core knowledge carrier, stores business rules in a database table format, supporting visual configuration and dynamic updates. The rule engine parses the condition fields, logical relationships, and weight parameters in the rule table to achieve multi-condition combination matching and compliance score calculation. This includes: the rule table storage uses structured fields to standardize rule definitions; the rule table flexibly supports multi-condition combinations through JSON fields, while using the weight field to differentiate rule importance; rule condition parsing; the rule engine reads the rule table data through a parser module and converts it into executable matching logic; and compliance score calculation involves the rule engine traversing the rule table records in the memory cache, matching each record against the current operation feature, and finally calculating the compliance score.

[0024] The beneficial effects of this invention are as follows: Employing multimodal perception fusion technology, it achieves accurate monitoring of operational behavior across all dimensions. By fusing visual (hand movement recognition, posture tracking, equipment status detection) and voice (voiceprint recognition, semantic analysis) multimodal data, it solves the problems of single-modality susceptibility to environmental interference and information loss. Operational behavior monitoring coverage is increased to 99.2%, and the missed detection rate for key steps is reduced to below 0.3%. The dynamic rule engine architecture enables flexible and adaptive rule system development. Adopting a three-tiered structure of "basic rules - step rules - task rules," it supports visual configuration and automated rule evolution, improving rule update efficiency by 80%. Rule coverage in complex scenarios (such as multi-position collaboration and anomaly handling) increases from 65% to 92%, while the misjudgment rate decreases by 45%. The real-time scoring and feedback mechanism enables immediate correction of operational deviations. Based on a weighted scoring model and dynamic calculation of compliance scores, combined with real-time monitoring of the nuclear power safety rule base, intervention measures are triggered immediately during the assessment process (such as pausing in case of uncertainty, reporting to the main controller), reducing response time to within 500ms and increasing the training and assessment pass rate by 28%. In complex scenarios, reliability under extreme conditions is improved. Through high-precision positioning technology (grid recognition error ≤0.5m), dynamic illumination adaptation algorithms, and device status time-series analysis, the accuracy rate of operation compliance judgment remains ≥95%, an improvement of 37% compared to traditional systems. Model optimization driven by challenging samples achieves continuous performance improvement. "High-risk, low-accuracy" samples mined using LangFuse are augmented and then fed back to the model for fine-tuning. Within 3 months, the overall recall rate of the model increased by 12%, and the false positive rate decreased by 20%, solving the pain point of lagging rule base updates in traditional systems. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the overall architecture of the skill behavior norm assessment system based on multimodal perception and dynamic rule engine provided by the present invention; Figure 2 This is a schematic diagram of visual model training. Figure 3 A hierarchical structure diagram of the dynamic rule engine; Figure 4 This is a flowchart of the rule matching and scoring process. Detailed Implementation

[0026] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] like Figure 1As shown, the skills and behavior norms assessment system based on multimodal perception and dynamic rule engine includes a multimodal perception module and a dynamic rule engine module. The multimodal perception module provides comprehensive and high-precision raw input for the compliance judgment of the dynamic rule engine module through multi-dimensional data collection and preprocessing of vision, voice and other dimensions, solving the problem of single modality being easily affected by environmental interference and information loss. The dynamic rule engine module constructs a multi-layer rule base based on the skills and behavior norms assessment and evaluation operations in the nuclear power field, and realizes dynamic updating of rules through scenario adaptation and data-driven approach, solving the problem that traditional fixed rules cannot cover complex operation variations.

[0028] The multimodal perception module includes a visual perception module and a voice perception module.

[0029] The visual perception module uses a head-mounted motion camera with a resolution of ≥1080P and a frame rate of ≥30fps, integrated into the front of the student's protective helmet to ensure an unobstructed operating view; it is paired with a high-performance GPU server to support local real-time preprocessing and feature extraction, reducing cloud transmission latency.

[0030] The visual perception module includes a hand motion recognition module, a posture tracking recognition module, an equipment nameplate recognition module, an equipment status recognition module, and a carpet grid recognition module, which can realize hand motion recognition, posture tracking recognition, equipment nameplate recognition, equipment status recognition, and carpet grid recognition.

[0031] The hand motion recognition module extracts the coordinates of 21 key points of the hand using the MediaPipe Hands framework, and optimizes the gesture classification model (such as convolutional neural network CNN) by combining it with a self-trained dataset to recognize actions such as holding tools, pointing at equipment, and confirming gestures in nuclear power operations in real time.

[0032] Specifically, it includes: Video frame preprocessing: 1080P@30fps video stream is acquired from the head-mounted camera, and the size of each frame image is normalized (640×640) and illumination compensation is performed (histogram equalization).

[0033] MediaPipe Hands Keypoint Extraction: Input preprocessed video frames, use MediaPipe Hands model for inference, output 21 hand keypoints with 3D coordinates (x, y, z). Keypoints include wrist, thumb base, thumb tip, and the base and tip of the first to fifth fingers. The detection confidence threshold is 0.5.

[0034] Gesture classification model inference: The coordinates of 21 key points are used to construct a 63-dimensional feature vector. The model structure is a 3-layer fully connected network (128→64→N), where N is the number of gesture categories. The output is the probability distribution of gesture categories. Gesture categories in nuclear power scenarios include holding tools, pointing at devices, confirmation gestures, reading gestures, warning gestures, etc.

[0035] Real-time tracking and smoothing: Kalman filtering is used to smooth the trajectory of key points, with a frame rate of ≥30fps and a key point detection error of ≤2mm.

[0036] Technical specifications: Gesture recognition accuracy ≥95%, frame rate ≥30fps, key point detection error ≤2mm. Meets the real-time requirements of dynamic operation scenarios.

[0037] The posture tracking and recognition module uses the YOLO target detection model to extract key points of the human body trunk, and combines it with the ByteTrack algorithm to achieve multi-target tracking, outputting JSON-formatted skeletal data containing the three-dimensional coordinates of 14 joints, including the head, shoulders, elbows, and wrists.

[0038] Specifically, it includes: Human detection: Input video frame (1080P), model is YOLOv11-pose lightweight version, output human bounding box and confidence score, detection threshold 0.5, supports simultaneous detection of ≥5 people.

[0039] Human keypoint detection: 14 keypoints including nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles. Each keypoint includes coordinates (x, y), confidence, and visibility. The 3D coordinates are obtained through monocular depth estimation (based on the MiDaS model).

[0040] Multi-target tracking (ByteTrack algorithm): The tracking process is detection → initial association (matching high-confidence detection boxes with existing trajectories) → secondary association (matching low-confidence detection boxes with unmatched trajectories) → trajectory management. The tracking ID switching success rate is ≥99%, and the human detection recall rate is ≥95%.

[0041] Skeletal data output: JSON format, including personnel ID, list of key point coordinates, and timestamp.

[0042] Technical specifications: Human body detection recall rate ≥95%, tracking ID switching success rate ≥99%, supports simultaneous detection of ≥5 people, and provides accurate posture information for multi-position collaborative operation.

[0043] The equipment nameplate recognition module uses OCR technology (CRNN+CTC joint model) to recognize text in the nameplate area, supporting Chinese and English characters, numbers and special symbols (such as "1SYS114VL", "on", "off").

[0044] Specifically, it includes: Nameplate area localization: Based on the YOLOv11 nameplate detection model, the model training data consists of equipment nameplate images (≥5000 images) collected at the nuclear power plant site, with a localization accuracy of ≥95%.

[0045] Image preprocessing: perspective correction (boundary detection based on Hough transform, affine transformation correction, tilt angle tolerance ≤30°), illumination compensation (adaptive contrast enhancement CLAHE), character sharpening (super-resolution reconstruction ESRGAN).

[0046] CRNN+CTC character recognition: CNN feature extraction uses ResNet50 (with classification layer removed), RNN sequence modeling uses bidirectional LSTM (hidden=256), and CTC decoding uses greedy decoding or beam search. It supports Chinese and English characters, numbers, and special symbols, with a character recognition accuracy of ≥99%.

[0047] Post-processing: Text standardization (removing extra spaces, unifying capitalization), format validation (using regular expressions to match device encoding), and structured output in JSON format.

[0048] Technical specifications: English text and number recognition rate ≥95%, Chinese text recognition rate ≥95%, nameplate positioning accuracy ≥95%.

[0049] The device status recognition module uses the YOLO target detection model to detect the switch position (on / off) and indicator light color (red / yellow / green, color confusion rate ≤3%) in real time.

[0050] Specifically, it includes: Equipment status detection model: YOLOv8 object detection, training data consists of nuclear power equipment images (≥10,000 images, labeled with switch status and indicator light color), supports ≥10 types of nuclear power equipment, and has a detection accuracy of ≥98%.

[0051] Switch status recognition: Switch types are classified as two-position switches, three-position switches, and rotary switches, and status is classified as on / off / middle position. The recognition method is to detect the switch area, extract the angle features of the switch handle, and output the status label from the classifier. The angle recognition error is ≤5°.

[0052] Indicator light color recognition: Colors are classified as red, yellow, green, and no light, with a color confusion rate of ≤3%. The recognition method is to detect the indicator light area ROI, perform HSV color space analysis, and classify the colors. Lighting adaptation is achieved through automatic white balance correction.

[0053] Timing state analysis: The timing window is 10 consecutive frames of voting. If the state stability is determined by ≥7 frames of identical data, the state is determined. The state change is recorded by adding the timestamp and the state value.

[0054] Technical specifications: Equipment status and switch status recognition rate ≥95%, color confusion rate ≤3%, angle recognition error ≤5°.

[0055] The carpet grid recognition module uses the YOLO_shoe lightweight model to detect the carpet grid code under the operator's feet, simulating real-time positioning of the personnel's precise location within the nuclear island (error ≤ 0.5m).

[0056] Specifically, it includes: Grid coding design: The coding rule is a combination of numbers and letters (such as A01, B12). The coding position is at the four corners or center of the grid. The coding size is about 10cm×10cm, covering the main operating area of ​​the core island.

[0057] YOLO_shoe lightweight detection model: YOLOv8n (nano version), input size 320×320, FPS≥30fps, training data is grid carpet images (≥3000 images), recognition accuracy ≥99%.

[0058] Multi-scale feature fusion: The backbone network is CSPDarknet+SPP module, and feature fusion adopts FPN (Feature Pyramid Network). Multi-scale detection is small (13×13), medium (26×26), and large (52×52).

[0059] Dynamic lighting adaptation: The scene switches between normal lighting and emergency lighting. The method is adaptive exposure adjustment plus color constancy. The lighting adaptation time is ≤500ms.

[0060] Indoor positioning calculation: grid coding → coordinate mapping table, positioning accuracy ≤ 0.5m, positioning update frequency ≥ 1Hz, trajectory smoothing uses Kalman filtering.

[0061] Technical specifications: grid recognition accuracy ≥99%, positioning accuracy ≤0.5m, positioning update frequency ≥1Hz.

[0062] The voice perception module ensures clear voice acquisition of students by deploying noise-canceling microphones (beamforming technology, SNR≥65dB) and Bluetooth bone conduction headphones (anti-environmental noise interference); the microphone array supports 360° directional sound pickup and filters background equipment operating noise (such as pump noise).

[0063] The speech perception module includes a speech denoising and separation module, a voiceprint recognition and role differentiation module, a semantic analysis and text matching module, and a large model structured parsing module, which can realize speech denoising and separation, voiceprint recognition and role differentiation, semantic analysis and text matching, and large model structured parsing.

[0064] The speech denoising and separation module uses a deep neural network (DNN) beamforming algorithm to separate effective speech from environmental noise (improving the signal-to-noise ratio by more than 15dB).

[0065] Specifically, it includes: Multi-microphone array acquisition: 8-channel circular array, sampling rate 16kHz, sampling accuracy 16bit, SNR requirement ≥65dB.

[0066] Frequency domain feature extraction: Short-time Fourier Transform (STFT), frame length 25ms (400 sampling points), frame shift 10ms (160 sampling points), frequency resolution 512-point FFT.

[0067] DNN Beamforming Network: The network structure is a deep beamforming network. The input is multi-channel spectral features (8×257×T), the backbone is 3D convolution (channel×frequency×time), and the output is a single-channel enhanced speech spectrum. The training data is nuclear power plant site noise samples (pumps, air conditioners, fans) ≥100 hours.

[0068] Post-processing: Spectral subtraction residual noise suppression, voice activity detection (VAD), automatic gain control (AGC), output signal-to-noise ratio improvement ≥15dB.

[0069] Technical specifications: Speech recognition rate ≥90% (high noise scene), signal-to-noise ratio improvement ≥15dB, sampling rate 16kHz.

[0070] The voiceprint recognition and role differentiation module is based on the ECAPA-TDNN model (pre-trained student voiceprint database) to identify the role of the voice source (operator / guardian) in real time.

[0071] Specifically, it includes: Voiceprint feature extraction: Input is the denoised speech, and the features are MFCC (40-dimensional) plus first-order difference, with a frame length of 25ms and a frame shift of 10ms.

[0072] The ECAPA-TDNN voiceprint encoder uses an Emulated Attention Pyramid (EAP) as its backbone network, multi-scale TDNN (dilated convolutions: 2, 3, 4, 5) for context modeling, AttentiveStatistics Pooling for statistical pooling, 192-dimensional embedding, and Angular ProtoNet+AAM-Softmax as its loss function.

[0073] Voiceprint registration and comparison: Registration is recorded when a student uses the app for the first time (≥30 seconds), the features are stored in a voiceprint embedding vector database, and the comparison is calculated using cosine similarity with a judgment threshold of 0.7.

[0074] Role differentiation: Supports roles such as operator, guardian, and examiner, with an accuracy rate of ≥95% and a real-time performance of ≤100ms.

[0075] Technical specifications: Role recognition accuracy ≥95%, real-time performance ≤100ms, voiceprint judgment threshold 0.7.

[0076] The semantic analysis and text matching module uses the WhisperX model to convert speech to text (WER≤3%), and combines TF-IDF and semantic similarity calculation (Cosine similarity≥0.92) to match the standard speech for the operation task.

[0077] Specifically, it includes: speech-to-text (WhisperX): the model is Whisper Large V3 (fine-tuned version), the input is denoised speech (16kHz), and the output is timestamp-annotated text, with WER (word error rate) ≤3%.

[0078] Text preprocessing: Chinese word segmentation uses Jieba, English word segmentation uses BERT Tokenizer, noise filtering removes modal particles and repeated words, and text normalization is performed from Arabic numerals to Chinese numerals.

[0079] TF-IDF text matching: The corpus is a standard nuclear power plant operation discourse database (≥5000 entries), which is vectorized into TF-IDF vectors (10000 dimensions). The similarity is calculated using cosine similarity, with a threshold of Cosine≥0.92.

[0080] Keyword extraction: Device number is identified by regular expression matching, operation command is identified by verb dictionary matching, and parameter extraction is extracted by numerical value plus unit pattern matching.

[0081] Structured output: JSON format, containing timestamps, text, and a list of keywords.

[0082] Technical specifications: Voice recognition WER≤3%, semantic similarity Cosine≥0.92, supports Mandarin, Pinyin, English letters, as well as industry standard phrases (such as “Confirm opening of No. 1 water pump outlet electric valve”) and keywords (such as equipment number “1SYS114VL”, operation command “open”).

[0083] The large model structured parsing module integrates LangChain to implement domain prompt management, and combines it with the fine-tuned large model (based on Qwen3) deployed on the Dify platform for semantic understanding and structured extraction of speech and text.

[0084] Specifically, it includes: LangChain Prompt Management: The prompt template design includes system prompts (the role is defined as nuclear power operation expert), task prompts (structured extraction of task descriptions), and output format (JSON Schema definition). The few-shot example consists of 3-5 extraction examples, and chain-of-thought prompting is used for chain reasoning.

[0085] Fine-tuning the large model (Qwen3-8B): The pre-trained model is Qwen3.5-9B, the fine-tuning method is LoRA (lora_rank=16), the training data is the nuclear power operation instruction dataset (≥5000), and the domain adaptation is the injection of nuclear power professional terminology.

[0086] Dify platform deployment: The deployment method is the Ollam inference engine, which supports ≥5 concurrent paths, inference latency ≤500ms, and the GPU configuration is four RTX3090 cards (24GB).

[0087] Structured extraction: The output format is JSON Schema, and the fields include instruction_type, device_id, device_name, action, and parameters.

[0088] Initial screening of nuclear power safety rules: The rule base is a database of prohibitions on nuclear power safety operations; conflict detection is the matching of extracted results with the rule base; and alarm triggering is the marking of high-risk operations.

[0089] Technical specifications: Inference latency ≤500ms, concurrent support ≥10 paths, structured extraction accuracy ≥95%.

[0090] The dynamic rule engine module constructs a multi-layered rule base based on the assessment and evaluation operations of skills and behaviors in the nuclear power field. It achieves dynamic rule updates through scenario adaptation and data-driven approaches, solving the problem that traditional fixed rules cannot cover complex operational variations. Specifically, it includes a rule base construction module and a rule matching and scoring module.

[0091] The rule base construction module is based on the core elements and operational procedures of nuclear power skills and behavior standard assessment. It constructs a rule base with a three-level tree structure of "basic rules - step rules - task rules", achieving full-dimensional rule coverage from single operational elements to complete task chains. The specific hierarchical structure includes the following: Basic rules are the smallest decision units in the rule base, directly corresponding to the smallest indivisible verifiable element in nuclear power plant operation specifications. They are used to quantify and constrain the most fundamental and atomized requirements of operational behavior, such as: Equipment status: Electric valve closed status indicator light 065LA is green; Semantic matching class: Whether the operation dialog contains critical device codes (such as 1SYS114VL) or standard instructions (such as 'Confirm Close').

[0092] Step rules are single-step operation judgment rules built upon basic rules. They are used to verify whether a single operation action meets the specification requirements. Typically, they must simultaneously satisfy the combined conditions of multiple basic rules. Each step rule corresponds to a specific operation action (such as "finger-to-verbal confirmation device"). Its judgment requires integrating the verification results of multiple basic rules, forming an "AND / OR" logical combination. Taking the "finger-to-verbal confirmation device" operation as an example, the step rule is defined as follows: Condition 1 (Basic Rule 1): Speech-to-text conversion must include the target device name or code (e.g., “1SYS114VL”). Condition 2 (Basic Rule 2): In the synchronously acquired video frames, the visible area of ​​the target device accounts for ≥70% (verified by YOLO detection).

[0093] Comprehensive judgment: This step is judged to pass only if both condition 1 and condition 2 are met.

[0094] The task rule is a complete task judgment rule composed of multiple step rules connected in series or parallel according to the nuclear power operation process logic, used to verify the compliance of the entire operation task. Each task rule needs to define the execution order of the steps, the logical relationship (such as "sequential execution" or "parallel verification"), and the final pass condition (such as "all steps pass" or "critical steps pass").

[0095] Taking the task "Confirm that the manual-automatic switch 003CC is selected in the 'automatic' position" as an example, the task rule is defined as follows: Step 1 (Sequential Execution): Execute the "Operator reads out operation steps" rule (must pass the basic rule verification containing specified keywords); Step 2 (sequential execution): Perform the "Self-test: Confirm manual-automatic switch 003CC equipment nameplate information" procedure (point to the equipment and verbally describe). Step 3 (sequential execution): Execute the "Self-test: Confirm the status of manual-automatic changeover switch 003CC" step rule (the switch position needs to be verified through the basic rules of the device status class); The task is considered successful only if steps 1, 2, and 3 are all successful.

[0096] The rule matching and scoring module employs a structured rule table storage combined with a rule engine, balancing rule maintainability and matching efficiency. The rule table, serving as the core knowledge carrier, stores business rules in a database table format, supporting visual configuration and dynamic updates. The rule engine parses the condition fields, logical relationships, and weight parameters in the rule table to achieve multi-condition combination matching and compliance score calculation. The specific technical implementation is as follows: Rule table storage mode The standardized definition of rules is achieved through structured fields. The core field design is as follows: The rule table flexibly supports multiple condition combinations through JSON fields (such as "Device status = On AND Voice includes forward rotation"), and differentiates the importance of rules through the weight field. Compared with hard-coded rules, it is easier to maintain and extend (adding a new rule only requires inserting a new record, without modifying the code).

[0097] Rule condition parsing: The rule engine reads the rule table data through the parser module and converts it into executable matching logic. The core process is as follows: Rule table loading and caching: When the system starts, the rule engine loads all rule table records from the database into the memory cache (such as Redis) and builds the rule index (with rule_id as the key and rule details as the value).

[0098] Condition parsing and standardization: The condition_items field (JSON format) of each rule is parsed and converted into a condition expression that the engine can recognize.

[0099] For example, {"field":"device_status","op":"=","value":"On"} is converted to the internal expression: device_status == "On"; and ({"field":"voice","op":"⊇","value":"Forward"}) is converted to voice ⊇ "Forward".

[0100] Logical Relation Combination: Multiple conditions are combined into a compound condition based on the logic_relation field (AND / OR). For example, the rule "Device Status = On AND Voice ⊇ Forward" will be parsed as (device_status == "On") && (voice ⊇ Forward); if it is an AND relationship, it will be parsed as (device_status == "On") && (voice ⊇ Forward).

[0101] The compliance score calculation involves the rule engine traversing the rule table records in the memory cache, matching each record against the current operation characteristics, and finally calculating the compliance score. The specific steps are as follows: ① Single rule matching degree calculation: For each rule, calculate the matching degree of the current operation feature based on the parsed conditional expression and tolerance rule. ② Rule score calculation: Based on the matching degree and rule weight, calculate the contribution score of a single rule: Sj = matching degree j × weightj For example, if a rule has a matching degree of 0.8 and a weight of 1.5, then the score for a single rule is 0.8 × 1.5 = 1.2.

[0102] ③Comprehensive compliance score synthesis: Sum the scores of all rules to get the final compliance score: Total ×100 ④ Scoring and Grading Based on the assessment requirements of skills operation behavior standards, the score can be customized into n levels, for example: This invention achieves comprehensive perception by deeply integrating high-resolution head-mounted cameras, fixed cameras, and handheld PAD devices to collect real-time data on operator actions, equipment status, and environmental information. Combined with noise reduction technology and intelligent voice processing, it enables voice separation and role management for multi-position collaborative operations. Through multi-dimensional data fusion and real-time analysis, this technology effectively improves the monitoring accuracy and response efficiency of nuclear power plant operation scenarios, providing a reliable basis for skills and behavior standard assessments. It adopts a hierarchical rule system design, supporting visual configuration and automated rule evolution. Through a three-tiered architecture of basic rules (atomic judgment units), step rules (single-step operation logic combinations), and task rules (multi-step process sequences), it achieves refined judgment from single operations to complex tasks. The rule engine relies on real-time stream processing technology to dynamically load the rule library and parse multi-condition combination logic, combining it with a weighted scoring model to achieve real-time compliance judgment and scoring classification, meeting dynamic adaptability requirements. Through multi-modal data fusion technology, it constructs a spatiotemporal correlation model of operational behavior, solving the problem of refined judgment in complex scenarios such as multi-position collaboration and anomaly handling. By employing high-precision positioning technology and dynamic illumination adaptation algorithms, combined with equipment status timing analysis, the fault tolerance capability of complex operation processes is improved.

Claims

1. A skill behavior evaluation system based on multimodal perception and dynamic rule engine, characterized in that: It includes a multimodal perception module and a dynamic rule engine module. The multimodal perception module provides raw input for the compliance determination of the dynamic rule engine module by collecting and preprocessing multi-dimensional data. The dynamic rule engine module constructs a multi-layer rule base based on the assessment and evaluation of skills and behaviors in the nuclear power field, and realizes dynamic updates of rules through scenario adaptation and data-driven approaches.

2. The skill behavior evaluation system based on multimodal perception and dynamic rule engine as described in claim 1, characterized in that: The multimodal perception module includes a visual perception module and a voice perception module. The visual perception module includes a hand motion recognition module, a posture tracking recognition module, an equipment nameplate recognition module, an equipment status recognition module, and a carpet grid recognition module, realizing hand motion recognition, posture tracking recognition, equipment nameplate recognition, equipment status recognition, and carpet grid recognition. The voice perception sub-module includes a voice noise reduction and separation module, a voiceprint recognition and role differentiation module, a semantic analysis and text matching module, and a large model structured parsing module, realizing voice noise reduction and separation, voiceprint recognition and role differentiation, semantic analysis and text matching, and large model structured parsing.

3. The skill behavior evaluation system based on multimodal perception and dynamic rule engine as described in claim 2, characterized in that: The hand gesture recognition module extracts the coordinates of 21 key points of the hand using the MediaPipe Hands framework, optimizes the gesture classification model by combining it with a self-trained dataset, and identifies hand gestures such as holding tools, pointing at equipment, and confirming gestures in nuclear power operations in real time. The process includes: video frame preprocessing: acquiring video streams from a head-mounted camera, normalizing the size of each frame image, and performing illumination compensation processing; key point extraction using MediaPipe Hands: inputting the preprocessed video frames, using the MediaPipeHands model for inference, outputting the three-dimensional coordinates of 21 hand key points, constructing a 63-dimensional feature vector from the coordinates of the 21 key points, and using a 3-layer fully connected network model to output the probability distribution of gesture categories.

4. The skill behavior evaluation system based on multimodal perception and dynamic rule engine as described in claim 2, characterized in that: The posture tracking and recognition module uses the YOLO target detection model to extract key points of the human body's main trunk, and combines it with the ByteTrack algorithm to achieve multi-target tracking. It outputs skeletal data in JSON format with 3D coordinates of 14 key points, including: Human body detection: input video frames, model is a lightweight version of YOLOv11-pose, output human body bounding box and confidence score; Human body key point detection: 14 key points including nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles. Each key point includes coordinates, confidence score, and visibility. The 3D coordinates are obtained through monocular depth estimation.

5. The skill behavior evaluation system based on multimodal perception and dynamic rule engine as described in claim 2, characterized in that: The equipment nameplate recognition module performs text recognition on the nameplate area using a CRNN+CTC joint model, including: nameplate area localization: a nameplate detection model based on YOLOv11, with model training data being equipment nameplate images collected at the nuclear power plant site; image preprocessing: bounding box detection based on Hough transform, affine transformation correction, tilt angle tolerance ≤30°, illumination compensation, character sharpening, CNN feature extraction using ResNet50, RNN sequence modeling using bidirectional LSTM, CTC decoding using greedydecoding or beam search; post-processing: text standardization, format validation, and structured output in JSON format.

6. The skill behavior evaluation system based on multimodal perception and dynamic rule engine as described in claim 2, characterized in that: The equipment status recognition module uses the YOLO target detection model to detect switch positions and indicator light colors in real time. This includes: the equipment status detection model uses YOLOv8 target detection, with training data consisting of nuclear power equipment images; switch status recognition: switch types are classified as two-position switches, three-position switches, and rotary switches; status is classified as on / off / middle position; the recognition method involves detecting the switch area, extracting the switch handle angle features, and outputting status labels from the classifier; indicator light color recognition: colors are classified as red, yellow, green, and no light; the recognition method involves detecting the indicator light area ROI, HSV color space analysis, and color classification; illumination adaptation uses automatic white balance correction; and temporal status analysis: the temporal window consists of 10 consecutive frames of voting, and if the status stability is determined by ≥7 identical frames, the status is confirmed; status changes are recorded with a timestamp and a status value.

7. The skill behavior evaluation system based on multimodal perception and dynamic rule engine as described in claim 2, characterized in that: The carpet grid recognition module, based on the YOLO_shoe lightweight model, detects the carpet grid code under the operator's feet, simulating real-time positioning of the operator's precise location within the nuclear island. This includes: grid code design: the coding rule is a combination of numbers and letters, with the coding position at the four corners or center of the grid; YOLO_shoe lightweight detection model: YOLOv8n, input size 320×320, FPS≥30fps, training data is grid carpet images; multi-scale feature fusion: the backbone network is CSPDarknet+SPP module, feature fusion uses FPN; dynamic lighting adaptation: the scene switches between normal lighting and emergency lighting, the method is adaptive exposure adjustment plus color constancy; indoor positioning calculation: the grid code corresponds to a coordinate mapping table, and trajectory smoothing uses Kalman filtering.

8. The skill behavior evaluation system based on multimodal perception and dynamic rule engine as described in claim 2, characterized in that: The speech denoising and separation module employs a deep neural network beamforming algorithm to separate effective speech from environmental noise. This includes: multi-microphone array acquisition: an 8-channel circular array with a sampling rate of 16kHz, sampling precision of 16bit, and an SNR requirement of ≥65dB; frequency domain feature extraction: short-time Fourier transform, frame length 25ms, frame shift 10ms, frequency resolution 512-point FFT; DNN beamforming network: a deep beamforming network structure, with multi-channel spectral features as input, a 3D convolution backbone, and a single-channel enhanced speech spectrum as output; training data consisting of ≥100 hours of nuclear power plant site noise samples; and post-processing: spectral subtraction for residual noise suppression, speech activity detection, and automatic gain control.

9. The skill behavior evaluation system based on multimodal perception and dynamic rule engine as described in claim 2, characterized in that: The voiceprint recognition and role differentiation module is based on the ECAPA-TDNN model and identifies the role of the voice source in real time. This includes: Voiceprint feature extraction: inputting denoised speech, with features of MFCC plus first-order difference, frame length 25ms, frame shift 10ms; ECAPA-TDNN voiceprint encoder: the backbone network is an Emulated Attention Pyramid, context modeling is a multi-scale TDNN, statistical pooling is Attentive Statistics Pooling, the embedding dimension is 192, and the loss function is AngularProtoNet+AAM-Softmax; Voiceprint registration and comparison: registration involves recording a voiceprint sample of ≥30 seconds during the student's first use, with features stored in a voiceprint embedding vector database, and comparison using cosine similarity calculation with a judgment threshold of 0.

7.

10. The skill behavior evaluation system based on multimodal perception and dynamic rule engine as described in claim 2, characterized in that: The semantic analysis and text matching module uses the WhisperX model for speech-to-text conversion, combining TF-IDF and semantic similarity calculations to match standard dialogue for operational tasks. This includes: Speech-to-text conversion: The model is Whisper LargeV3 (input is denoised speech), outputting timestamped text; text preprocessing: Chinese word segmentation uses Jieba segmentation, English word segmentation uses BERT Tokenizer, noise filtering removes interjections and repetitive words; TF-IDF text matching: The corpus is a standard dialogue library for nuclear power plant operations, vectorized into TF-IDF vectors, similarity calculation uses cosine similarity, with a threshold of Cosine ≥ 0.92; keyword extraction: Equipment number recognition uses regular expression matching, operation command recognition uses verb dictionary matching, parameter extraction uses numerical plus unit pattern matching; structured output: JSON format, containing timestamps, text, and a keyword list.

11. The skill behavior evaluation system based on multimodal perception and dynamic rule engine as described in claim 2, characterized in that: The large-scale model structured parsing module integrates LangChain to manage domain prompts. Combined with the fine-tuned large model deployed on the Dify platform, it performs semantic understanding and structured extraction of speech and text, including: LangChain Prompt Management: prompt word template design includes system prompts, task prompts, and output formats; few-shot examples are 3-5 extraction examples; chain-of-thought prompting is used for chain inference; fine-tuning the large model: the pre-trained model is Qwen3.5-9B, the fine-tuning method is LoRA, the training data is a nuclear power operation instruction dataset, and domain adaptation is the injection of nuclear power professional terms; Dify platform deployment: the deployment method is the Ollam inference engine; structured extraction: the output format is JSON Schema, and the fields include instruction_type, device_id, device_name, action, and parameters; initial screening of nuclear power safety rules: the rule base is a nuclear power safety operation prohibition library; conflict detection is the matching of extraction results with the rule base; and alarm triggering is the marking of high-risk operations.

12. The skill behavior evaluation system based on multimodal perception and dynamic rule engine as described in claim 2, characterized in that: The dynamic rule engine module is based on the assessment and evaluation of skills and behaviors in the nuclear power field to build a multi-layer rule base, and realizes dynamic updates of rules through scenario adaptation and data-driven approaches. It includes a rule base construction module and a rule matching and scoring module.

13. The skill behavior evaluation system based on multimodal perception and dynamic rule engine as described in claim 12, characterized in that: The rule base is constructed based on the core elements and operational procedures of nuclear power skills and behavior standard assessment. It features a three-tiered tree structure: "Basic Rules - Step Rules - Task Rules," achieving full-dimensional rule coverage from single operational elements to complete task chains. The specific hierarchical structure includes: Basic rules, the smallest judgment unit of the rule base, directly corresponding to the smallest indivisible verifiable element in nuclear power operation standards, used to quantify and constrain the most basic and atomic requirements of operational behavior; Step rules, single-step operation judgment rules built upon basic rules, used to verify whether a single operational action meets the standard requirements, requiring simultaneous satisfaction of multiple basic rule combinations. Each step rule corresponds to a specific operational action, and its judgment integrates the verification results of multiple basic rules, forming an "AND / OR" logical combination; Task rules, complete task judgment rules composed of multiple step rules logically connected in series / parallel according to the nuclear power operation process, used to verify the compliance of the entire operational task. Each task rule needs to define the execution order, logical relationship, and final pass conditions of the steps.

14. The skill behavior evaluation system based on multimodal perception and dynamic rule engine as described in claim 12, characterized in that: The rule matching and scoring module adopts a technical solution that combines structured rule table storage with a rule engine. The rule table serves as the core knowledge carrier, storing business rules in the form of a database table, and supports visual configuration and dynamic updates. The rule engine parses the condition fields, logical relationships, and weight parameters in the rule table to achieve multi-condition combination matching and compliance score calculation, including: the rule table storage uses structured fields to achieve standardized definition of rules; The rule table flexibly supports multiple condition combinations through JSON fields, and differentiates the importance of rules through the weight field. For rule condition parsing, the rule engine reads the rule table data through the parser module and converts it into executable matching logic. The compliance score calculation involves the rule engine traversing the rule table records in the memory cache, matching the current operation features one by one, and finally calculating the compliance score.

Images