A method and system for real-time intervention of experimental operation violations based on multimodal behavior recognition

Abstract

This invention belongs to the field of laboratory safety monitoring technology, and particularly relates to a method and system for real-time intervention of experimental operation violations based on multimodal behavior recognition. It includes collecting multimodal operation data during the experiment; constructing a database of standard operating procedures (SOPs); identifying the currently executed action; comparing the currently executed action with data in the SOP database in multiple dimensions; calculating a deviation score based on the comparison results; and, combined with preset hazard coefficients, environmental coefficients, and personnel coefficients, using a fuzzy logic algorithm to determine the risk level of the experimental violation, thus achieving real-time intervention for experimental operation violations. This invention is based on the deployment and data acquisition of a multimodal sensor network, performing multidimensional modeling of standard operating procedures, and then using a real-time behavior recognition algorithm for multidimensional comparison and violation determination, thereby initiating corresponding intervention measures to promptly eliminate safety hazards, resulting in significant social and economic benefits.

Description

Technical Field

[0001] This invention belongs to the field of laboratory safety monitoring technology, and in particular relates to a method and system for real-time intervention of experimental operation violations based on multimodal behavior recognition. Background Technology

[0002] Laboratory safety management is a core element in ensuring the smooth operation of scientific research activities, teaching practices, and production and R&D. Its importance is multifaceted, serving as both a fundamental safeguard for personnel safety and a crucial support for research efficiency, compliance requirements, environmental protection, and social stability. Existing laboratory safety monitoring technologies face three major bottlenecks: 1. Video surveillance is limited in biosafety level 2 and above laboratories, as laboratory personnel generally resist facial recognition and full-process video recording; 2. Traditional sensors can only detect environmental parameters and cannot identify the "operational behavior" itself; 3. Violations are mostly detected through retrospective investigation, lacking the ability to intervene in real time at the millisecond level.

[0003] Therefore, how to achieve precise perception of experimental operations while protecting privacy, how to compare sensor data with standard operating procedures in real time, and how to implement effective intervention before violations occur are important issues that current laboratory safety research needs to address. Summary of the Invention

[0004] To overcome the shortcomings of the existing technologies, this invention provides a method and system for real-time intervention of experimental operation violations based on multimodal behavior recognition. Based on the deployment and data acquisition of a multimodal sensor network, multidimensional modeling of standard operating procedures is performed. Then, through real-time behavior recognition algorithms, multidimensional comparison and violation judgment are conducted, thereby initiating corresponding intervention measures to achieve timely elimination of safety hazards, which has significant social benefits.

[0005] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: The first aspect of this invention provides a method for real-time intervention of experimental operation violations based on multimodal behavior recognition.

[0006] A real-time intervention method for experimental operation violations based on multimodal behavior recognition includes the following steps: Deploy a distributed sensor network to collect multimodal operation data during the experiment; A database of standard operating procedures for experiments is constructed, and the standard operations of various experiments are decomposed into time-sequence atomic action sequences, and multi-dimensional constraints are set for each atomic action unit. The multimodal operation data during the experiment is input into the deep learning behavior recognition model to identify the currently executed action, and the currently executed action is compared with the data in the experimental standard operating procedure database in multiple dimensions. Based on the comparison results, the deviation score is calculated, and combined with the preset risk coefficient, environmental coefficient and personnel coefficient, the risk level of experimental violations is determined by fuzzy logic algorithm. Based on the risk level of experimental violations, corresponding intervention measures are initiated within a preset response time to achieve real-time intervention in experimental operational violations.

[0007] A second aspect of the present invention provides a real-time intervention system for experimental operation violations based on multimodal behavior recognition.

[0008] A real-time intervention system for experimental operation violations based on multimodal behavior recognition includes: The sensor deployment and data acquisition module is configured to: deploy a distributed sensor network and acquire multimodal operation data during the experiment; The database configuration module is configured to: build an experimental standard operating procedure database, decompose the standard operations of various experiments into a time-series atomic action sequence, and set multi-dimensional constraints for each atomic action unit. The actual behavior recognition and comparison module is configured to: input multimodal operation data during the experiment into the deep learning behavior recognition model, identify the currently executed action, and perform a multidimensional comparison between the currently executed action and the data in the experimental standard operating procedure database; The risk level assessment module is configured to: calculate the deviation score based on the comparison results, and combine it with preset risk coefficients, environmental coefficients and personnel coefficients, and use fuzzy logic algorithms to determine the risk level of experimental violations; The risk level intervention module is configured to activate corresponding intervention measures within a preset response time based on the risk level of experimental violations, thereby achieving real-time intervention for experimental operation violations.

[0009] The above one or more technical solutions have the following beneficial effects: This invention provides a method and system for real-time intervention of experimental operation violations based on multimodal behavior recognition. It abandons the existing scheme of judging experimental violations by video recognition and face recognition. Based on the deployment and data collection of multimodal sensor networks, it performs multidimensional modeling of standard operating procedures, and then uses real-time behavior recognition algorithms to perform multidimensional comparison and violation judgment, thereby initiating corresponding intervention measures to achieve timely elimination of safety hazards, which has significant social benefits.

[0010] This invention decomposes the standard operations of various experiments into a sequence of time-series atomic actions, and models the sequence of time-series atomic actions using a hierarchical finite state machine. Each layer of the state machine corresponds to an operation process of different granularity. The state transition conditions of the hierarchical finite state machine include sensor feature matching, time event triggering, and external input signals. An experimental standard operating procedure database is constructed, which provides a basis for multi-dimensional comparison of the currently executed action with the data in the experimental standard operating procedure database in the later stage.

[0011] This invention sets multidimensional constraints for each atomic action unit, including spatial constraints, temporal constraints, force constraints, and sequence constraints. In the subsequent multidimensional comparison, based on the above constraints, action sequence comparison, motion trajectory comparison, force curve comparison, and time window comparison are performed to calculate the deviation score and achieve accurate quantification of the deviation.

[0012] This invention combines the calculated deviation score with preset risk coefficients, environmental coefficients, and personnel coefficients, and uses a fuzzy logic algorithm to determine the risk level of experimental violations by considering multiple factors. This enables real-time risk level determination and provides targeted intervention measures, achieving real-time and effective intervention.

[0013] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0014] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0015] Figure 1 This is a flowchart of the method in Example 1. Detailed Implementation

[0016] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0017] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0018] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0019] Example 1 like Figure 1As shown, the real-time intervention method for experimental operation violations based on multimodal behavior recognition includes the following steps: Deploy a distributed sensor network to collect multimodal operation data during the experiment; A database of standard operating procedures for experiments is constructed, and the standard operations of various experiments are decomposed into time-sequence atomic action sequences, and multi-dimensional constraints are set for each atomic action unit. The multimodal operation data during the experiment is input into the deep learning behavior recognition model to identify the currently executed action, and the currently executed action is compared with the data in the experimental standard operating procedure database in multiple dimensions. Based on the comparison results, the deviation score is calculated, and combined with the preset risk coefficient, environmental coefficient and personnel coefficient, the risk level of experimental violations is determined by fuzzy logic algorithm. Based on the risk level of experimental violations, corresponding intervention measures are initiated within a preset response time to achieve real-time intervention in experimental operational violations.

[0020] This embodiment abandons the existing technology of using video recognition and facial recognition to determine experimental violations. Instead, it is based on the deployment and data acquisition of a multimodal sensor network, combined with a hierarchical finite state machine, to perform multidimensional modeling of the standard operating procedure. Then, through a real-time behavior recognition algorithm, it performs multidimensional comparison and violation determination of the currently executed action with the data in the experimental standard operating procedure database, determines the risk level of the currently executed action, and initiates corresponding intervention measures to achieve timely elimination of security risks while protecting privacy.

[0021] The specific implementation steps of this embodiment will be explained in detail below.

[0022] This embodiment includes the following steps: Step 1: Deploy a distributed sensor network on the experimental clothing worn by the experimenters and the experimental equipment used to collect multimodal operation data in real time.

[0023] In this embodiment, the deployment of the multimodal distributed sensor network lays the data foundation for determining the currently executed action and constructing the experimental standard operating procedure database. Overall: (1) The sensor includes: The micro inertial measurement unit includes: a three-axis accelerometer (placed on the wrist and the base of the fingers), a three-axis gyroscope (placed on the wrist and the arm), a three-axis magnetometer (placed on the wrist); a flexible piezoelectric film (placed on the fingertips), a near-field communication reader (placed on the cuff and the experimental table), and a bone conduction microphone (placed on the edge of goggles or a mask and the experimental table), etc.

[0024] (2) In terms of sensor network topology design, this embodiment equips each experimenter with a master control node (integrated in the waist of the experimental clothing), which connects all wearable sensors through the CAN bus; the master control node has a built-in STM32 microcontroller, which is responsible for preliminary data processing and feature extraction; the master control node communicates with the edge computing gateway in real time through Bluetooth Low Energy 5.0; the edge computing gateway is deployed in each laboratory room and adopts a development platform (optional model: NVIDIA Jetson Xavier NX).

[0025] (3) In this embodiment, the data packet format for each sensor is as follows: | Synchronization Header (2B) | Node ID (2B) | Sensor Type (1B) | Timestamp (4B) | Data Length (2B) | Data Payload (NB) | Checksum (2B) |, B represents bytes; the timestamp is synchronized using the IEEE 1588 (Precise Clock Synchronization Protocol for Network Measurement and Control Systems) precision time protocol, with a network-wide time error of less than 1ms (milliseconds).

[0026] More specifically, the distributed sensor network deployed in this embodiment includes: a. A micro inertial measurement unit, deployed on the wrists, fingers, arms, and waist of the experimenter, including a three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer, with a sampling frequency of not less than 100 Hz, is used to collect limb movement trajectory and posture data.

[0027] The triaxial accelerometer, model ADXL345, has a measurement range of ±16g (g is the unit of gravitational acceleration) and a resolution of 2mg / LSB (mg / LSB is milligravimetric acceleration / least significant bit). The three-axis gyroscope selected is ITG3200, with a measurement range of ±2000° / s and a sensitivity of 0.07° / s. The triaxial magnetometer, model HMC5883L, has a measurement range of ±8 Gauss and a resolution of 2 milligauss.

[0028] The micro inertial measurement unit fuses data from the accelerometer, gyroscope, and magnetometer using a Kalman filter algorithm to output precise attitude angles and quaternions.

[0029] b. Flexible piezoelectric thin film sensor, integrated into the fingertip and palm of the glove, with a sampling frequency of not less than 200 Hz, a measurement range of 0-10 N, and an accuracy of not less than 0.01 N, used to collect the gripping force and tactile feedback of the fingers.

[0030] In this embodiment, the flexible piezoelectric film sensor is an LDT0028K type piezoelectric film, which is deployed on the five fingertips and palm of the glove. Each sensor outputs an analog voltage signal, which is sampled by a 24-bit analog-to-digital converter and converted into a force value. The flexible piezoelectric film sensor is also used to detect the surface texture and sliding state of an object, and to identify the material of the object being operated on by analyzing the spectral characteristics of the signal.

[0031] c. Near-field communication tags, deployed on cuffs, experimental equipment, and reagent bottles, with a sampling frequency of 10Hz and a measurement range of 0-10cm, are used to identify the type and size of the object being handled.

[0032] d. Bone conduction microphone, integrated into the edge of goggles or mask, with a sampling frequency of no less than 16kHz, a measurement range of 20kHz, and an accuracy of 42dB, used to collect ambient sound and operation sounds.

[0033] e. The master control node is integrated into the waist of the lab coat. It connects to all sensors via a CAN bus, has a built-in microprocessor for initial data processing, and communicates with the edge computing gateway via Bluetooth Low Energy.

[0034] Step 2: Construct a standard operating procedure database, decompose the standard operations of various experiments into atomic action sequences with timestamps, and set multidimensional constraints for each atomic action.

[0035] The specific process of constructing the experimental standard operating procedure database includes: The standard operations of various experiments are decomposed into atomic action units with timestamps and encoded to obtain a temporal atomic action sequence; The temporal atomic action sequence is modeled using a hierarchical finite state machine, with each layer of the state machine corresponding to an operation process of different granularities. The state transition conditions of a hierarchical finite state machine include sensor feature matching, time event triggering, and external input signals. Constraints are expressed using sequential logic, and multi-dimensional constraints are set for each atomic action unit. The experimental standard operating procedure database also includes exception handling procedures and recovery strategies for operation failures.

[0036] The advantage of using a hierarchical finite state machine model in this embodiment is that: By employing hierarchical finite state machine modeling and combining four-dimensional constraints of space, time, force, and sequence, a qualitative leap from "point-based monitoring" to "process-oriented cognition" has been achieved compared to existing technologies. When performing multi-dimensional comparisons, the system not only compares the current frame of data but also compares the currently collected real-time trajectory with the standard trajectory in the standard state machine. Even if the current operation has not yet triggered a specific threshold (such as the force has not exceeded the limit), if the trend of the hand movement trajectory has deviated from the preset path in the state machine (for example, the angle of the pipette near the bottle opening is incorrect), the system can predict the possible violation in advance, thereby transforming "post-event alarm" into "pre-event intervention," which greatly improves the initiative of laboratory safety response.

[0037] Multidimensional constraints include: a. Spatial constraints: the three-dimensional coordinate range of the hand movement trajectory and the positional relationships of objects; b. Time constraints: minimum and maximum duration of each operation step, and maximum interval between steps; c. Force constraints: Threshold ranges for gripping force, pressing force, and rotational torque; d. Sequence constraints: the order of operation steps and the allowed combinations of parallel operations.

[0038] The advantages of setting multidimensional constraints are: Single-dimensional data often fails to reflect the true intent of an operation, easily leading to false alarms. Multi-dimensional constraints provide "context" for sensor data. In a laboratory environment, many fatal violations are microscopic (such as incorrect force or excessive speed), difficult for the human eye or ordinary cameras to detect. However, these subtle deviations are precisely the direct causes of test tube breakage and liquid splashing. By shifting constraints from the macroscopic "position" to the microscopic "physical quantity," the potential for "excessive force" that is imperceptible to the naked eye can be addressed before accidents occur (such as slipping caps or liquid leaks), achieving proactive risk control.

[0039] This embodiment performs multi-dimensional modeling of standard operating procedures. First, the standard operating procedures are decomposed into atomic action units and encoded, as shown in Table 1.

[0040] Table 1: Encoding table of atomic action units after decomposition of standard operating procedures.

[0041]

[0042] The action of "taking a test tube" is coded as A001, the action of "unscrewing the cap" is coded as A002, the action of "pipetting" is coded as A003, the action of "dropping" is coded as A004, and the action of "shaking" is coded as A005. The action description, typical duration and typical force are set respectively.

[0043] The action sequence was then modeled, and a hierarchical finite state machine was used to describe the complete experimental process, such as: Level 1 divides the entire experimental process into: preparation stage → operation stage → closing stage. Level 2 standardizes the specific operational actions for each stage.

[0044] Constraints are expressed using sequential logic, such as the requirement that the lid of a dish must be closed within 30 seconds after it has been opened, the requirement that the height of the fume hood door never exceed 20cm when handling flammable reagents, and the requirement that the lid must be closed when the centrifuge is running.

[0045] This embodiment decomposes the standard operations of various experiments into a sequence of time-series atomic actions. The sequence of time-series atomic actions is modeled using a hierarchical finite state machine. Each layer of the state machine corresponds to an operation process of different granularity. The state transition conditions of the hierarchical finite state machine include sensor feature matching, time event triggering, and external input signals. This provides a foundation for multidimensional comparison of the currently executed action with the data in the experimental standard operating procedure database.

[0046] Step 3: Input the real-time multimodal data collected in Step 1 into the pre-trained deep learning behavior recognition model to identify the currently executed atomic action and its state, and perform a multidimensional comparison with the standard operating procedure in Step 2.

[0047] The deep learning behavior recognition model is a hybrid architecture of CNN-BiLSTM-Attention (convolutional neural network-bidirectional long short-term memory network-attention mechanism model), including: Input layer: Receives a 128×128 dimensional temporal feature matrix; Two one-dimensional convolutional layers with a kernel size of 3 and a stride of 1, using the ReLU (Rectified Linear Unit) activation function to extract local temporal features; Batch normalization layer and max pooling layer, pooling size 2, are used for feature dimensionality reduction and to prevent overfitting; A bidirectional LSTM (Long Short-Term Memory) layer with 128 hidden units returns the complete sequence and is used to capture long-term dependencies. In the attention mechanism layer, the attention weights at each time step are calculated and summed to obtain the context vector. Fully connected layer, 256 units, activation function ReLU (Rectified Linear Unit), dropout rate 0.3; The Softmax (flexible maximum normalized value) output layer outputs the probability distribution of each atomic action.

[0048] The currently executed action is compared with the data in the experimental standard operating procedure database in multiple dimensions, specifically including: a. Action sequence alignment: The similarity between the actual operation sequence and the standard sequence is calculated using the dynamic time warping algorithm; b. Motion trajectory comparison: Hausdorff distance is used to calculate the spatial deviation between the actual hand trajectory and the standard trajectory; c. Force curve comparison: The correlation between the actual force curve and the standard curve is calculated using the Pearson correlation coefficient; d. Time window comparison: Determine whether the actual duration of each operation step is within the preset range.

[0049] In this step, the relationship between the multidimensional comparison and the multidimensional constraints in step two above is as follows: Step 2 (multidimensional constraints) establishes the "static standard library," while Step 3 (multidimensional comparison) performs real-time calibration between the "dynamic measured data" and the "static standard library." The hierarchical finite state machine acts as the "indexer" and "context manager" connecting the two. Step 2 establishes the hierarchical finite state machine, defining each state node (e.g., S1: disinfection, S2: removing the petri dish, S3: tightening the cap) and state transition conditions (e.g., the condition from S2 to S3 is "hand approaches the cap with force > 1N"). Step 3, during multidimensional comparison, first queries the current state node. For example, the system is currently in state S2: removing the petri dish. The system only calls the spatial constraints (hand coordinate range) and temporal constraints (duration of this step) associated with state S2 for comparison, without comparing the force curve of state S3. When the real-time data in Step 3 satisfies the transition conditions from the previous state to the current state, the system "enters" the current state and immediately activates the comparison task for all constraints related to that state.

[0050] The algorithm employs dynamic time warping to calculate the similarity between the currently executed actual operation sequence and the corresponding temporal atomic action sequence, specifically including: Construct an actual action sequence A and a standard action sequence B, where A = [a1, a2, ..., am], a1 represents the first action in the actual action sequence, a2 represents the second action in the actual action sequence, and am represents the m-th action in the actual action sequence; B = [b1, b2, ..., bn], b1 represents the first atomic action unit in the standard action sequence, b2 represents the second atomic action unit in the standard action sequence, and bn represents the n-th atomic action unit in the standard action sequence; Calculate the distance matrix D, where D(i,j) = ||ai-bj||², ai represents the i-th action in the actual action sequence, bj represents the j-th atomic action unit in the standard action sequence, and D(i,j) represents the distance between the i-th action in the actual action sequence and the j-th atomic action unit in the standard action sequence; ||·|| represents the norm, used to measure the distance between vectors ai and bj; The optimal regular path is solved by dynamic programming, and the length of the regular path is used as a measure of sequence similarity. If the length of the regularized path exceeds a preset threshold, it is determined to be a violation of the action sequence.

[0051] The deviation score is calculated by normalizing the path length and then combining it with path shape features. The optimal normalized path length between the actual operation sequence A and the standard action sequence B is calculated using a dynamic time warping algorithm. This process then calculates the normalized basic deviation score, the path shape penalty term, the key point weight penalty term, and finally, the overall deviation score.

[0052] In this embodiment, multi-dimensional constraints are set for each atomic action unit, including spatial constraints, temporal constraints, force constraints, and sequence constraints. In the subsequent multi-dimensional comparison, based on the above constraints, action sequence comparison, motion trajectory comparison, force curve comparison, and time window comparison are performed to calculate the deviation score and achieve accurate quantification of the deviation.

[0053] More specifically: (1) In this step, firstly, this embodiment constructs a real-time behavior recognition algorithm, uses Kalman filtering to remove high-frequency noise, transforms the sensor coordinate system to the world coordinate system, and extracts time-domain features and frequency-domain features.

[0054] The time-domain features include: mean, variance, peak-to-peak value, root mean square, kurtosis, and skewness; the frequency-domain features include: spectral centroid, spectral entropy, and Mel frequency cepstral coefficients; and the attitude features include: Euler angles and quaternions.

[0055] (2) Next, perform data processing for the deep learning model architecture: Input layer: 128×128 dimensional feature matrix (time step × feature dimension); Convolutional layer 1: One-dimensional convolutional layer (64, kernel size 3, activation function is linear rectified unit); Batch normalization layer, max pooling layer: pooling size is 2; Convolutional layer 2: One-dimensional convolutional layer (128, kernel size 3, activation function is linear rectified unit); Batch normalization layer, max pooling layer: pooling size is 2; Bidirectional Long Short-Term Memory (LSTM) network layer: 128 hidden units, returns a complete sequence; Attention mechanism layer: Calculates the attention weights at each time step; Fully connected layer: 256 hidden units, linear rectified unit activation function, random deactivation rate of 0.3; Random deactivation layer: The random deactivation rate is 0.3; Flexible maximum normalized output layer: Output probability distribution (number of action categories).

[0056] (3) Perform model training: A total of 100,000 operation samples were collected from 50 experimenters, 20 common experiments, and each frame of data was independently labeled by 3 experts. The consistency test Kappa coefficient was >0.85. The optimizer was Adam. The initial learning rate was 0.001. The loss function was classification cross-entropy + temporal consistency loss. The number of training rounds was 200. The early stopping patience value was 20. The final accuracy was 97.3% for action recognition and <0.2 seconds for action start point error.

[0057] (4) Optimize real-time inference for the model: Model quantization: FP32 (32-bit floating-point) weights are quantized to INT8 (8-bit integers), improving inference speed by 3 times; Model pruning: Remove connections with weights less than 0.01, and compress the model size by 60%. Edge deployment: Optimized with inference accelerator, single inference time <15ms.

[0058] Step 4: Calculate the deviation score based on the comparison results, and combine it with the preset risk coefficient, environmental coefficient and personnel coefficient to determine the risk level of the violation using a fuzzy logic algorithm.

[0059] Fuzzy logic algorithms include: Input variables: deviation score x1, risk factor x2, environmental factor x3, personnel factor x4; Fuzzification: The triangular membership function is used to map the input variables to three fuzzy sets: low, medium, and high. Fuzzy rule base: Contains 81 fuzzy inference rules, such as "IF high deviation score and high risk coefficient THEN high risk level"; Inference Engine: The Mandani inference method is used to calculate the activation strength of each rule; Defuzzification: The centroid method is used to convert the fuzzy output into an accurate risk value.

[0060] Risk levels include: a. Level 1 Risk: Minor deviation from standard procedures but no direct danger, deviation score 0.2-0.4; b. Level 2 Risk: Operational violations may lead to minor accidents, with a deviation score of 0.4-0.7; c. Level 3 risk: Operational violations directly threaten personnel safety, deviation score 0.7-1.0.

[0061] This embodiment combines the calculated deviation score with preset risk coefficients, environmental coefficients, and personnel coefficients, and uses a fuzzy logic algorithm to determine the risk level of experimental violations by considering multiple factors. This enables real-time risk level determination and provides targeted intervention measures, achieving real-time and effective intervention.

[0062] This embodiment performs multi-dimensional comparison and violation determination, comparing real-time data with standard operating procedures, including action sequence comparison, spatial trajectory comparison, force curve comparison, and time window comparison, and returns deviation score and violation type based on the comparison results.

[0063] Fuzzy logic is used to determine the risk level, where risk level = f(deviation score, hazard coefficient, environmental coefficient, personnel coefficient), and: f is a mapping function based on fuzzy logic, which realizes multi-factor risk level decision-making through three steps: fuzzification, fuzzy reasoning (based on expert rule base) and defuzzification. Deviation score: [0,1] calculated by the alignment algorithm; Risk factor: Preset according to the type of experiment, such as cell culture = 0.3, strong acid operation = 0.8; Environmental factor: Calculated based on environmental sensor data, such as harmful gas concentration = 0.6; Personnel coefficient: Based on personnel qualifications, novice = 0.9, skilled worker = 0.3.

[0064] Defuzzification: The centroid method is used to calculate the final risk value.

[0065] A deviation score range of 0.2-0.4 is classified as Level 1 risk, a deviation score range of 0.4-0.7 is classified as Level 2 risk, and a deviation score range of 0.7-1.0 is classified as Level 3 risk.

[0066] Step 5: Activate the corresponding intervention measures within the preset response time according to the risk level.

[0067] The equipment used to implement intervention measures includes: Bone conduction headphones: Bluetooth 5.0 communication method, response time ≤50ms, used for voice prompts; Smartwatch: Communication method is Bluetooth 5.0, customized based on STM32, integrated linear vibration motor, response time ≤30ms, used for vibration warning and display; Solid-state relay: Communication method: 485 bus, operating voltage: 220V, current: 10A, response time: ≤10ms, used to cut off the power supply of equipment; PID controller: Analog communication mode, receives 0-10V analog signals, response time ≤500ms, used to regulate exhaust volume.

[0068] Intervention measures include: a. Level 1 Risk: Sending voice prompts via bone conduction headphones or smart glasses; b. Level 2 risk: Send vibration warnings via smartwatches or wristbands, and simultaneously suspend the relevant equipment via electromagnetic locks or relays integrated into the experimental equipment; c. Level 3 risk: Immediately cut off the power supply to the hazardous source, activate the emergency ventilation system, and notify the nearest safety officer to rush to the scene via the indoor positioning system.

[0069] This embodiment employs a multi-level real-time intervention mechanism, constructing an intervention strategy decision tree. Different alert methods are used according to different risk levels. Level 1 risks are indicated by voice prompts, Level 2 risks are first warned, and if no correction is made, the equipment is suspended. In the event of Level 3 risks, the hazard source is immediately cut off, the power supply to the relevant equipment is cut off, emergency ventilation is activated to maximum power, a message is pushed to the three nearest safety officers through the indoor positioning system, emergency lighting and evacuation indicator lights are turned on, and an emergency voice broadcast is made: "Emergency situation, please stop operation immediately and wait for the safety officer."

[0070] After each intervention, the system automatically records the intervention effect: Intervention effectiveness score = (risk reduction after intervention / risk value before intervention) × 100%.

[0071] If the score is less than 50%, the intervention strategy optimization process is triggered, the intervention intensity is adjusted (e.g., upgrading from voice prompts to device locking), the intervention threshold is adjusted (lowering the threshold for determining violations), and the report is submitted for manual review.

[0072] Step 6: Store the operational data, intervention measures, and handling results of the violations into the database, use the elastic weight solidification algorithm for incremental learning, and regularly optimize the behavior recognition model and risk classification threshold.

[0073] This embodiment employs online model learning and optimization. A data feedback mechanism is established, uploading violation data collected from the edge to the cloud server daily. This data includes: raw sensor data, identification results, intervention measures, and manual review results. Incremental learning is performed using the EWC (Elastic Weights Fixed) algorithm, with incremental training conducted weekly. When the accuracy on the new dataset improves by more than 1%, the edge model is updated. Model version management retains the five most recent versions for rollback purposes.

[0074] Incremental learning employs an elastic weight fixation algorithm, with the loss function L(θ) as follows:

[0075] in: Let be the cross-entropy loss function for the current task; The diagonal elements of the Fisher information matrix represent parameters. The importance of; λ represents the parameters of the old model; λ is the regularization coefficient, which controls the degree to which old knowledge is retained. This represents the current i-th parameter.

[0076] The implementation steps of this embodiment will be described in detail below, taking into account the specific application scenario of monitoring cell culture operations inside a biosafety cabinet.

[0077] Experimental scenario: In a biosafety level 3 laboratory, the operator is performing passage culture of Vero cells (African green monkey kidney cells), which involves the potential risk of zoonotic viruses, requiring strict aseptic operation and biosafety standards.

[0078] (a) Detailed hardware deployment configuration: 1. Smart lab coat with integrated sensors: Waist main control unit: STM32F407 microcontroller + ESP32WROOM32 wireless communication module with WiFi and Bluetooth functions; Battery: 7.4V 5000mAh lithium battery, providing up to 24 hours of battery life; Arm: 2 MPU6050 six-axis sensors (sampling rate 100Hz); Back: Flexible temperature sensor array (monitoring body temperature distribution); Chest: Miniature camera (only used to record the position of the glove, not the face, resolution 320×240).

[0079] 2. Smart gloves with integrated sensors: Each fingertip: LDT0028K piezoelectric thin film sensor (range 010N, accuracy 0.01N). Second joint of each finger: ADXL345 accelerometer (sampling rate 200Hz); Palm: Flexible pressure sensor matrix (8×8 array, for detecting gripping posture); Wrist: ITG3200 gyroscope + HMC5883L magnetometer; Thumb base: PN532 near-field communication reader (reading distance 5cm).

[0080] 3. Smart goggles with integrated sensors: Left temple of the glasses: Bone conduction microphone; Right temple of the glasses: bone conduction headphones + miniature vibration motor; Inner side of the lens: Infrared proximity sensor (to detect whether it is being worn).

[0081] 4. Intelligent laboratory equipment, intelligent pipettes: Integrated miniature electromagnetic lock, Bluetooth communication; Intelligent incubator: Hall effect switches are installed on the door, and RFID readers are installed inside; Smart petri dish: RFID tag and temperature sensor integrated at the bottom.

[0082] (II) Detailed modeling of standard operating procedures and detailed constraints for cell passaging operations: Step 1: Disinfect gloves with 75% alcohol.

[0083] Action: Move your hands toward the alcohol spray bottle → press the nozzle → rub your hands together; Space: Both hands are positioned in the center of the control panel, at a height of 1020cm; Time: Kneading time ≥ 30 seconds; Force: Press the nozzle with a force of 35N; Sensor characteristics: The accelerometer detected a kneading motion frequency of 23Hz.

[0084] Operation step 2: Remove the petri dish from the incubator.

[0085] Actions: Open the door → Reach out → Pinch the petri dish → Remove it → Close the door; Space: Hands should be inserted into the incubator to a depth of ≤30cm; Time: From opening to closing the door ≤ 10 seconds (to prevent temperature fluctuations); Constraints: The petri dish must be removed horizontally with a tilt angle of less than 5 degrees.

[0086] Step 3: Open the lid of the dish.

[0087] Action: Pinch the edge of the lid with your thumb and forefinger → lift it vertically upwards by 2cm → flip it over → place it on the table; Force: Squeeze the cap with 0.5-1.5N; Constraints: The opening of the petri dish lid should face upwards, and the placement position should be ≤5cm away from the petri dish; Time: From picking up the lid to placing the dish in place, ≤ 3 seconds; Critical monitoring: If the dish is detected to be placed with the lid facing down (accelerometer detects a tilt angle >90 degrees), an immediate warning will be issued.

[0088] …Intermediate steps omitted… Operation step 12: Place in an incubator.

[0089] Actions: Open the door → Place the petri dish in the petri dish → Close the door; Constraint: After closing the door, it must be checked that the door is closed tightly (Hall switch signal); Data recording: Record the location of the petri dish (which shelf and which layer).

[0090] (III) Real-time identification and intervention examples (detailed timeline): Operation begins at 00:00:00; system initialization completes; all sensors self-test normal. At 00:00:05, the hand moves toward the alcohol spray bottle, the near-field communication system detects the spray bottle, and the system enters the "disinfection preparation" state. At 00:00:07, the nozzle was pressed, and the force sensor detected 4.2N (normal). The system recorded "Alcohol spraying completed". The hands were rubbed together starting at 00:00:08, and the accelerometer detected a rubbing frequency of 2.5Hz. At 00:00:38, the kneading continued for 30 seconds, and the system voice prompt said, "Disinfection complete, you can begin the operation." At 00:00:40, the hand moved towards the incubator, and near-field communication detected the incubator door handle; The door opens at 00:00:42, the Hall switch signal changes, and the system starts timing; At 00:00:43, a hand was placed into the incubator, and infrared ranging detected a depth of 25cm (normal). At 00:00:45, the petri dish was pinched, and the fingertip force sensor detected 1.2N (normal). At 00:00:46, the culture dish was removed, and the accelerometer detected horizontal movement (tilt angle 2 degrees, normal). The door closes at 00:00:47, the Hall switch signal is restored, and the door remains open for 5 seconds (less than 10 seconds, which is normal). At 00:00:50, the petri dish was placed on the operating table, and the RFID reader identified the petri dish ID. At 00:00:52, begin opening the lid of the dish, applying 1.1N of force with your thumb and forefinger; At 00:00:53, the lid of the dish was lifted upwards, and the accelerometer detected vertical motion. At 00:00:54, when the lid of the dish was flipped, the gyroscope detected a rotational angular velocity of 150 degrees per second. Time 00:00:55 Anomaly detection: Gyroscope data and magnetometer data show that the lid flipped 195 degrees (it should have flipped 180 degrees ± 5 degrees); At 00:00:55, the system calculates: the flip angle is 195 degrees, the deviation is 15 degrees, and based on the current action, it is determined that "the lid of the dish may be placed face down". Time 00:00:55 Risk level calculation: Deviation score 0.35, risk factor 0.4, personnel factor 0.5 (this operator is a novice), environmental factor 0.2 → Fuzzy logic calculation yields a risk value of 0.42, which is judged as a level 2 risk; Intervention execution at 00:00:56: 1. Bone conduction earphone voice message: "Dish lid may be facing down, please check." 2. The smartwatch vibrates and displays a red warning icon; 3. Simultaneously start a 0.5-second countdown to wait for correction; At 00:00:57, after hearing the prompt, the operator checked the lid of the dish and found that it was indeed tilted, and manually corrected it. At 00:00:58, the sensor detected that the lid had been readjusted and the angle had been restored to 180 degrees. At 00:00:59, the system confirmed the correction was complete, the warning was lifted, and monitoring resumed. (iv) Data recording and reporting: After the intervention is completed, the system automatically generates a violation report.

[0091] Example 2 This embodiment discloses a real-time intervention system for experimental operation violations based on multimodal behavior recognition.

[0092] A real-time intervention system for experimental operation violations based on multimodal behavior recognition includes: The sensor deployment and data acquisition module is configured to: deploy a distributed sensor network and acquire multimodal operation data during the experiment; The database configuration module is configured to: build an experimental standard operating procedure database, decompose the standard operations of various experiments into a time-series atomic action sequence, and set multi-dimensional constraints for each atomic action unit. The actual behavior recognition and comparison module is configured to: input multimodal operation data during the experiment into the deep learning behavior recognition model, identify the currently executed action, and perform a multidimensional comparison between the currently executed action and the data in the experimental standard operating procedure database; The risk level assessment module is configured to: calculate the deviation score based on the comparison results, and combine it with preset risk coefficients, environmental coefficients and personnel coefficients, and use fuzzy logic algorithms to determine the risk level of experimental violations; The risk level intervention module is configured to activate corresponding intervention measures within a preset response time based on the risk level of experimental violations, thereby achieving real-time intervention for experimental operation violations.

[0093] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0094] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for real-time intervention of experimental operation violations based on multimodal behavior recognition, characterized in that, Includes the following steps: Deploy a distributed sensor network to collect multimodal operation data during the experiment; A database of standard operating procedures for experiments is constructed, and the standard operations of various experiments are decomposed into time-sequence atomic action sequences, and multi-dimensional constraints are set for each atomic action unit. The multimodal operation data during the experiment is input into the deep learning behavior recognition model to identify the currently executed action, and the currently executed action is compared with the data in the experimental standard operating procedure database in multiple dimensions. Based on the comparison results, the deviation score is calculated, and combined with the preset risk coefficient, environmental coefficient and personnel coefficient, the risk level of experimental violations is determined by fuzzy logic algorithm. Based on the risk level of experimental violations, corresponding intervention measures are initiated within a preset response time to achieve real-time intervention in experimental operational violations.

2. The method for real-time intervention of experimental operation violations based on multimodal behavior recognition as described in claim 1, characterized in that, The distributed sensor network specifically includes: Micro inertial measurement unit, used to collect limb movement trajectory and posture data; Flexible piezoelectric thin film sensors are used to collect the gripping force and tactile feedback of fingers; Near-field communication tags are used to identify the type and size of the object being operated on. Bone conduction microphones are used to collect ambient sound and operation sounds; The master node is used to connect all sensors via the CAN bus and communicate with the edge computing gateway. The deployment method for distributed sensor networks is as follows: The micro inertial measurement unit is deployed on the wrist, base of the fingers, arm, and waist of the experimenter; the micro inertial measurement unit includes a three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer, wherein the three-axis accelerometer is deployed on the wrist and base of the fingers of the experimenter, the three-axis gyroscope is deployed on the wrist and arm of the experimenter, and the three-axis magnetometer is deployed on the wrist. Flexible piezoelectric film sensors are deployed at the fingertips and palm of the glove; Near-field communication tags are deployed on the cuffs of lab coats, lab equipment, and reagent bottles; Bone conduction microphones are placed along the edges of goggles or masks; The main control node is integrated into the waist of the lab coat.

3. The method for real-time intervention of experimental operation violations based on multimodal behavior recognition as described in claim 1, characterized in that, The specific process of constructing the experimental standard operating procedure database includes: The standard operations of various experiments are decomposed into atomic action units with timestamps and encoded to obtain a temporal atomic action sequence; The temporal atomic action sequence is modeled using a hierarchical finite state machine, with each layer of the state machine corresponding to an operation process of different granularities. The state transition conditions of a hierarchical finite state machine include sensor feature matching, time event triggering, and external input signals. Constraints are expressed using sequential logic, and multi-dimensional constraints are set for each atomic action unit. The experimental standard operating procedure database also includes exception handling procedures and recovery strategies for operation failures.

4. The method for real-time intervention of experimental operation violations based on multimodal behavior recognition as described in claim 1, characterized in that, The multidimensional constraints specifically include: Spatial constraints, specifically the three-dimensional coordinate range of the hand movement trajectory and the positional relationships of objects; The time constraints are specifically the minimum and maximum duration of each operation step and the maximum interval between steps. The force constraint conditions are specifically the threshold ranges for gripping force, pressing force, and rotational torque; Sequence constraints specifically refer to the order of operation steps and the allowed combinations of parallel operations.

5. The method for real-time intervention of experimental operation violations based on multimodal behavior recognition as described in claim 1, characterized in that, The deep learning behavior recognition model is a CNN-BiLSTM-Attention hybrid architecture, including: Input layer: used to receive the temporal feature matrix; Two one-dimensional convolutional layers: used to extract local temporal features; Batch normalization layer and max pooling layer: used for feature dimensionality reduction and to prevent overfitting; Bidirectional LSTM layer: used to capture long-term dependencies; Attention mechanism layer: used to calculate the attention weights at each time step, and then sum them up to obtain the context vector; Output layer: Used to output the probability distribution of each atomic action.

6. The method for real-time intervention of experimental operation violations based on multimodal behavior recognition as described in claim 1, characterized in that, The currently executed action is compared with the data in the experimental standard operating procedure database in multiple dimensions, specifically including: Action sequence alignment: The dynamic time warping algorithm is used to calculate the similarity between the currently executed actual operation sequence and the corresponding temporal atomic action sequence; Motion trajectory comparison: Hausdorff distance is used to calculate the spatial deviation between the actual hand trajectory and the standard trajectory; Force curve comparison: The correlation between the actual force curve and the standard curve is calculated using the Pearson correlation coefficient; Time window comparison: Determine whether the actual duration of each operation step is within the preset range; The algorithm employs dynamic time warping to calculate the similarity between the currently executed actual operation sequence and the corresponding temporal atomic action sequence, specifically including: Construct an actual action sequence A and a standard action sequence B, where A = [a1, a2, ..., am], a1 represents the first action in the actual action sequence, a2 represents the second action in the actual action sequence, and am represents the m-th action in the actual action sequence; B = [b1, b2, ..., bn], b1 represents the first atomic action unit in the standard action sequence, b2 represents the second atomic action unit in the standard action sequence, and bn represents the n-th atomic action unit in the standard action sequence; Calculate the distance matrix D, where D(i,j) = ||ai-bj||², ai represents the i-th action in the actual action sequence, bj represents the j-th atomic action unit in the standard action sequence, and D(i,j) represents the distance between the i-th action in the actual action sequence and the j-th atomic action unit in the standard action sequence; ||·|| represents the norm, used to measure the distance between vectors ai and bj; The optimal regular path is solved by dynamic programming, and the length of the regular path is used as a measure of sequence similarity. If the length of the regularized path exceeds a preset threshold, it is determined to be a violation of the action sequence.

7. The method for real-time intervention of experimental operation violations based on multimodal behavior recognition as described in claim 1, characterized in that, The deviation score is calculated through a comparison algorithm, the risk coefficient is preset according to the experiment type, the environmental coefficient is calculated based on environmental sensor data, and the personnel coefficient is obtained based on personnel qualifications. Fuzzy logic algorithms are used to determine the risk level of experimental violations, specifically including: The input variables are deviation score, risk factor, environmental factor, and personnel factor. The triangular membership function is used to map the input variables to three fuzzy sets: low, medium, and high, for fuzzification processing. The activation strength of each rule in the fuzzy rule base is calculated using inference methods, and the fuzzy output is obtained based on the activation strength of each rule. The centroid method is used to convert fuzzy outputs into precise risk values; The risk level is determined based on the risk value and the preset multi-level risk assessment thresholds.

8. The method for real-time intervention of experimental operation violations based on multimodal behavior recognition as described in claim 1, characterized in that, The intervention measures specifically include: When the risk level is Level 1: Send a voice prompt; When the risk level is level 2: send a vibration warning and simultaneously suspend the relevant equipment via an electromagnetic lock or relay integrated into the experimental equipment; When the risk level is level three: immediately cut off the power supply to the hazardous source and start the emergency ventilation system.

9. The method for real-time intervention of experimental operation violations based on multimodal behavior recognition as described in claim 1, characterized in that, This also includes storing operational data, intervention measures, and handling results of experimental violations into the experimental standard operating procedure database, using an elastic weight solidification algorithm for incremental learning, and periodically conducting deep learning of behavior recognition models and risk classification thresholds; Incremental learning employs an elastic weight fixation algorithm, with a loss function... for: in, Let be the cross-entropy loss function for the current task; The diagonal elements of the Fisher information matrix represent parameters. The importance of; λ represents the parameters of the old model; λ is the regularization coefficient, which controls the degree to which old knowledge is retained. This represents the current i-th parameter.

10. A real-time intervention system for experimental operation violations based on multimodal behavior recognition, characterized in that, include: The sensor deployment and data acquisition module is configured to: deploy a distributed sensor network and acquire multimodal operation data during the experiment; The database configuration module is configured to: build an experimental standard operating procedure database, decompose the standard operations of various experiments into a time-series atomic action sequence, and set multi-dimensional constraints for each atomic action unit. The actual behavior recognition and comparison module is configured to: input multimodal operation data during the experiment into the deep learning behavior recognition model, identify the currently executed action, and perform a multidimensional comparison between the currently executed action and the data in the experimental standard operating procedure database; The risk level assessment module is configured to: calculate the deviation score based on the comparison results, and combine it with preset risk coefficients, environmental coefficients and personnel coefficients, and use fuzzy logic algorithms to determine the risk level of experimental violations; The risk level intervention module is configured to activate corresponding intervention measures within a preset response time based on the risk level of experimental violations, thereby achieving real-time intervention for experimental operation violations.

Images