Computer vision based laboratory behavior monitoring and warning method and related devices

Abstract

The application discloses a laboratory behavior monitoring and early warning method based on computer vision and related equipment. The application comprises: collecting experimental scene image data and reagent use related monitoring data associated with a target object; performing visual detection on the experimental scene image data and the reagent use related monitoring data to determine experimental behavior information and reagent use whole-process tracking information; based on a preset judgment rule, performing fusion analysis on the experimental behavior information and the reagent use whole-process tracking information to identify a laboratory irregular behavior type of the target object; and according to the laboratory irregular behavior type, starting a corresponding target early warning strategy. The application can realize efficient and comprehensive monitoring of a laboratory, accurately identify various irregular behaviors and timely give early warnings, effectively reduce safety hazards, solve the problems of low efficiency of laboratory monitoring and insufficient safety guarantee, improve the laboratory safety management level, and can be widely applied to the technical field of laboratory safety management.

Description

Technical Field

[0001] This invention relates to the field of laboratory safety management technology, and in particular to a laboratory behavior monitoring and early warning method and related equipment based on computer vision. Background Technology

[0002] Laboratories are important venues for various scientific research, teaching, testing and R&D activities. They are widely used in schools, hospitals, chemical plants and various production plants, especially in the fields of chemistry, biology, medical testing, and materials research and development. The experimental process generally involves hazardous chemicals, precision experimental equipment and various reagents. The standardization of operation is directly related to the personal safety of experimental personnel, the safety of laboratory property and the normal operation of related units.

[0003] In existing technologies, laboratories typically use methods such as on-site manual supervision, post-event review of ordinary videos, manual registration of hazardous chemicals and various reagents, traditional smoke detectors, and regular training to meet the laboratory's behavioral monitoring and early warning needs.

[0004] Therefore, existing laboratories suffer from insufficient regulatory capacity and chaotic management of hazardous chemicals and various reagents, which directly leads to low monitoring efficiency and frequent safety accidents. This not only threatens the personal safety of laboratory personnel but also affects the normal operation of related units, causing property damage and adverse social impact. Summary of the Invention

[0005] This invention provides a computer vision-based method and equipment for monitoring and early warning of laboratory behavior, which solves the technical problems of inefficient laboratory monitoring and prominent safety hazards.

[0006] In a first aspect, the present invention provides a laboratory behavior monitoring and early warning method based on computer vision, comprising:

[0007] Collect experimental scene image data and reagent usage-related monitoring data associated with the target object;

[0008] Visual inspection is performed on the experimental scene image data and reagents using relevant monitoring data to determine experimental behavior information and reagent usage tracking information throughout the entire process.

[0009] Based on preset judgment rules, the experimental behavior information and the reagent usage full-process tracking information are fused and analyzed to identify the types of laboratory violations by the target object.

[0010] Based on the type of laboratory violation, the corresponding target early warning strategy will be activated.

[0011] In some embodiments, the step of performing visual inspection on the experimental scene image data and reagent usage-related monitoring data to determine experimental behavior information and reagent usage end-to-end tracking information includes:

[0012] Visual recognition processing is performed on the experimental scene image data to determine the experimental behavior information of the target object;

[0013] Visual analysis is performed on the relevant monitoring data of the reagents to determine the full-process tracking information of the reagents used by the target object.

[0014] In some embodiments, the step of performing visual recognition processing on the experimental scene image data to determine the experimental behavior information of the target object includes:

[0015] Skeletal point detection and pose estimation are performed on the experimental scene image data to determine the experimental operation action information of the target object;

[0016] The experimental scene image data is used to perform target localization and regional spatial analysis to determine the experimental area activity information of the target object;

[0017] The experimental scene image data is processed to identify human body and protective equipment, and the protective equipment wearing status information of the target object is determined.

[0018] Based on the experimental operation action information, the experimental area activity information, and the protective equipment wearing status information, the experimental behavior information of the target object is determined.

[0019] In some embodiments, the reagent usage-related monitoring data includes reagent cabinet usage monitoring data, RFID data, weighing sensor monitoring data, and camera visual tracking data. The step of performing visual analysis processing on the reagent usage-related monitoring data to determine the full-process tracking information of the target object's reagent usage includes:

[0020] The reagent cabinet usage monitoring data is analyzed to extract reagent usage and return information;

[0021] The radio frequency identification data and the camera visual tracking data are analyzed in a coordinated manner to determine the movement trajectory information of the reagent from its collection to its return;

[0022] The monitoring data from the weighing sensor and the visual tracking data from the camera are analyzed together to determine the reagent usage process information.

[0023] Based on the reagent retrieval information, the reagent return information, the movement trajectory information, and the usage process information, the full-process tracking information of the reagent usage of the target object is determined.

[0024] In some embodiments, the preset judgment rules include experimental behavior violation judgment rules and reagent use violation judgment rules. The step of fusing and analyzing the experimental behavior information and the reagent use full-process tracking information based on the preset judgment rules to identify the types of laboratory violations by the target object includes:

[0025] Based on the aforementioned experimental behavior violation judgment rules, the experimental behavior information is analyzed to identify each experimental behavior violation data and the experimental behavior violation type corresponding to each experimental behavior violation data.

[0026] Based on the reagent use violation judgment rules, the entire process tracking information of reagent use is analyzed to identify each reagent use violation data and the reagent use violation type of each reagent use violation data;

[0027] Based on the violation data of each experimental behavior and the violation data of each reagent use, time synchronization and behavior correlation analysis are performed to determine each violation combination data and the violation combination type associated with each violation combination data;

[0028] Based on the types of experimental behavior violations, reagent use violations, and combinations of violations, the types of laboratory violations by the target object are determined.

[0029] In some embodiments, the step of performing time synchronization and behavioral correlation analysis based on each of the experimental behavior violation data and each of the reagent use violation data to determine each violation combination data and the violation combination type associated with each violation combination data includes:

[0030] Analyze the violation data of each experimental behavior to determine the start time, end time and related information of each violation data.

[0031] Analyze the data on violations of reagent use for each of the aforementioned reagents to determine the start time, end time, and associated information of reagent use corresponding to each data point.

[0032] When the behavior association information of one experimental behavior violation data is consistent with the reagent association information of one reagent use violation data, a time consistency analysis is performed on the one experimental behavior violation data and the one reagent use violation data.

[0033] When the time consistency analysis shows that both the first time difference and the second time difference are within a preset time threshold, the violation data of one of the experimental behaviors and the violation data of one of the reagent uses are associated and bound to obtain violation combination data; the first time difference is the difference between the start time of the behavior and the start time of the reagent use; the second time difference is the difference between the end time of the behavior and the end time of the reagent use.

[0034] Based on the types of experimental behavior violations and reagent use violations associated with the violation combination data, the violation combination type associated with the violation combination data is determined.

[0035] In some embodiments, activating a corresponding target early warning strategy based on the type of laboratory violation includes:

[0036] Obtain the preset mapping relationship between violation types and early warning strategies;

[0037] The target early warning strategy for handling the laboratory violation type is matched from the preset mapping relationship and activated; the target early warning strategy includes at least the target early warning level, early warning prompt method, early warning notification target, and violation handling guidance information.

[0038] Secondly, the present invention provides a laboratory behavior monitoring and early warning device based on computer vision, comprising:

[0039] The visual monitoring module is used to collect experimental scene image data associated with the target object and related monitoring data on reagent usage;

[0040] The visual inspection module is used to perform visual inspection on the experimental scene image data and reagent usage-related monitoring data to determine experimental behavior information and reagent usage full-process tracking information.

[0041] The violation analysis module is used to perform fusion analysis on the experimental behavior information and the reagent usage full-process tracking information based on preset judgment rules, and to identify the types of laboratory violations of the target object;

[0042] The violation warning module is used to activate the corresponding target warning strategy based on the type of violation in the laboratory.

[0043] Thirdly, the present invention provides an electronic device including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the above-described computer vision-based laboratory behavior monitoring and early warning method.

[0044] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed, implements the above-mentioned computer vision-based laboratory behavior monitoring and early warning method.

[0045] As can be seen from the above technical solutions, the present invention has the following advantages: The present invention provides a laboratory behavior monitoring and early warning method and related equipment based on computer vision. It collects experimental scene image data and reagent usage-related monitoring data associated with the target object, performs visual detection on the experimental scene image data and reagent usage-related monitoring data, determines experimental behavior information and reagent usage full-process tracking information, and performs fusion analysis based on preset judgment rules to identify laboratory violation types and activate corresponding target early warning strategies. The present invention can achieve efficient and comprehensive monitoring of laboratories, accurately identify various violations and provide timely warnings, effectively reduce safety hazards, solve the problems of inefficient laboratory monitoring and insufficient safety assurance, standardize experimental operations, and improve the level of laboratory safety management. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 A flowchart illustrating a computer vision-based laboratory behavior monitoring and early warning method provided in an embodiment of the present invention;

[0048] Figure 2 A schematic diagram of a computer vision-based laboratory behavior monitoring and early warning device provided in an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0050] This invention provides a computer vision-based laboratory behavior monitoring and early warning method and related equipment to address the technical problems of inefficient laboratory monitoring and prominent safety hazards.

[0051] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0052] Please see Figure 1 , Figure 1 An optional flowchart for a computer vision-based laboratory behavior monitoring and early warning system provided in an embodiment of the present invention includes steps 101 to 103.

[0053] Step 101: Collect experimental scene image data and reagent usage-related monitoring data associated with the target object;

[0054] Experimental scene image data refers to real-time video / frame images captured by multiple cameras, including those for laboratory panoramas, lab benches, reagent cabinets, and access control, encompassing all elements such as personnel, equipment, reagents, and environment.

[0055] Reagent usage-related monitoring data refers to data collected jointly by reagent cabinet smart locks, RFID readers, weighing sensors, visual tracking devices, etc., including reagent storage, weight, location, and movement path.

[0056] Optionally, the relevant monitoring data and experimental scene image data of the target object's reagents are associated and bound with the target object's current experiment identifier. The current experiment identifier refers to the identifier combination used to bind this experiment, which includes at least the target object's identity identifier and the current experiment operation identifier, in order to achieve accurate association between behavior and reagent data.

[0057] For example, taking the collection and binding of data from the entire process of student experiments in a chemistry laboratory as an example, the collection and binding of relevant monitoring data on reagent use and experimental scene image data are explained in detail below:

[0058] Step A1: Obtain the current experiment identifier of the target object: Student name Li Si, Experiment table: Experiment table 5, Identity identifier: ID–20240308, Current experiment operation identifier: Table 5–202603231440;

[0059] Step A2, Acquisition of Experimental Scene Image Data: A panoramic camera captures the entire laboratory environment and locates student positions. Camera No. 5 captures data on student hand movements, test tube information, reagent bottle information, and alcohol lamp information during the experiment. Test tube information includes tube posture, nozzle orientation, liquid level, heating status, grip method, damage status, and internal reaction status. Reagent bottle information includes label content, placement posture, cap status, pouring speed, pouring posture, reagent mixing behavior, and transfer status. Alcohol lamp information includes flame status, flame height, tilt angle, liquid addition operation, extinguishing method, and surrounding safety distance. A reagent cabinet camera captures reagent storage and retrieval actions. An access control camera captures the status of students wearing protective equipment when entering the laboratory.

[0060] Step A3: Reagent usage monitoring data collection: Real-time upload of reagent weight via reagent cabinet weighing sensor; reading of reagent bottle labels via RFID reader; recording of cabinet door opening / closing status via reagent cabinet smart lock.

[0061] Step 102: Visually inspect the experimental scene image data and reagent usage-related monitoring data to determine experimental behavior information and reagent usage full-process tracking information;

[0062] The reagent usage full-process tracking information refers to the identity, time, location, weight, and trajectory data of the reagent throughout its entire lifecycle, from retrieval to movement, use, and return / disposal, which are all bound to the target object.

[0063] Experimental behavioral information refers to complete, structured behavioral data of a target object in the laboratory, formed by integrating operational actions, area activities, and protective clothing, which can be used to determine violations.

[0064] In some embodiments, step 102 may include the following sub-steps:

[0065] Visual recognition processing is performed on experimental scene image data to determine the experimental behavior information of the target object;

[0066] Visual analysis and processing of reagent usage-related monitoring data are used to determine the full-process tracking information of reagent usage for the target object.

[0067] For example, taking the monitoring of the entire experimental process of students in a chemistry laboratory as an example, the visual detection process for laboratory behavioral data (including experimental scene image data and reagent usage-related monitoring data) is explained in detail. The specific implementation steps are as follows:

[0068] Step B1: Obtain experimental scene image data and reagent usage monitoring data of student Li Si from steps A1 to A3 above.

[0069] Step B2 involves performing visual recognition processing on the experimental scene image data to obtain experimental behavior information, including:

[0070] Experimental operation information:

[0071] Action: The gas was smelled using the fan-smelling method, not directly through the nose;

[0072] Test tube heating: The mouth of the test tube should face the uninhabited area at a 45° angle to the horizontal.

[0073] Equipment usage: Use test tube clamps to hold the test tubes, and follow the operating procedures.

[0074] Experimental area activity information:

[0075] The entire process took place within the No. 5 laboratory area and did not enter the hazardous chemicals restricted area.

[0076] Did not linger in front of the reagent cabinet in violation of regulations;

[0077] Keep a safe distance from open flames;

[0078] Protective equipment wearing status information:

[0079] Goggles: Worn correctly throughout the entire process and not removed;

[0080] Lab coat: fully worn and buttoned;

[0081] Gloves: Wear rubber gloves as required;

[0082] Step B3 involves visually analyzing and processing the monitoring data related to reagent usage to obtain full-process tracking information on reagent usage, including:

[0083] Reagent dispensing information:

[0084] Time: 14:41:05; Operator: Li Si;

[0085] Reagent: dilute sulfuric acid (0.5 mol / L); Weight to be used: reduce by 30 g (approximately 25 ml);

[0086] Movement trajectory information:

[0087] Path: Reagent cabinet → Corridor → Lab bench #5;

[0088] Bottle holder: Li Si, no act of passing the bottle;

[0089] Usage information:

[0090] Location of use: Experimental table No. 5;

[0091] Procedure: Pour slowly, label facing palm, no improper mixing;

[0092] Actual dosage: Approximately 20ml;

[0093] Reagent return information:

[0094] Return time: 14:55:30;

[0095] Re-weighing: The remaining weight matches the usage amount, and there is no abnormal loss;

[0096] Return status: Returned to the original cabinet and shelf, RFID tag matches;

[0097] Step B4: Perform data association and binding using the current experiment identifier as the index. The binding result is as follows:

[0098] Identification: ID–20240308 (Li Si) + Experimental Operation Identification: Station No. 5–202603231440 + Experimental Behavior Information + Reagent Usage Full Process Tracking Information.

[0099] In some embodiments, the above-described visual recognition processing of experimental scene image data to determine the experimental behavior information of the target object includes:

[0100] Skeletal point detection and pose estimation are performed on experimental scene image data to determine the experimental operation action information of the target object;

[0101] Target localization and regional spatial analysis are performed on experimental scene image data to determine the activity information of the target object in the experimental area;

[0102] Human body and protective equipment recognition processing is performed on experimental scene image data to determine the protective equipment wearing status information of the target object;

[0103] Based on experimental operation information, experimental area activity information, and protective equipment wearing status information, the experimental behavior information of the target object is determined.

[0104] Skeletal point detection refers to the location and tracking of key points such as the head, neck, shoulders, elbows, wrists, waist, and knees of a target object.

[0105] Pose estimation refers to the calculation of human / hand pose, orientation, angle, and movement trajectory based on the three-dimensional spatial coordinates of skeletal points.

[0106] Human body and protective equipment identification refers to locating human body areas and determining whether goggles, lab coats, masks, and gloves are worn and whether they are worn correctly through instance segmentation and target detection.

[0107] Experimental operation action information refers to the action characteristics and compliance results of test tube operation, reagent pouring, open flame operation, and smelling gas, which are estimated by skeletal points and posture.

[0108] Experimental area activity information refers to the location, stay, and movement information of the target object within the experimental platform, restricted area, and safe distance.

[0109] The protective equipment wearing status information refers to outputting three statuses for goggles, lab coats, gloves, and masks: not worn, worn incorrectly, and worn correctly.

[0110] Optionally, by performing visual recognition processing on the experimental scene image data, the experimental operation action information of the target object can be determined, specifically including:

[0111] Human skeleton point detection was performed on the experimental scene images to locate key points of the head, hands, and torso;

[0112] The distance between the head and the test tube, hand posture, test tube angle, tilting action, and heating action are calculated using 3D pose estimation.

[0113] Based on the test results of the equipment (including test tubes, reagent bottles, alcohol lamps, etc.), determine whether the actions are standardized and output experimental operation information.

[0114] Target localization and regional spatial analysis are performed on the experimental scene image data to determine activity information in the experimental area, specifically including:

[0115] A semantic segmentation model is used to divide the experimental scene image into multiple preset experimental regions, and the spatial boundaries of each experimental region are defined. The preset experimental regions include an experimental operation area, a reagent storage area, an instrument and equipment area, a safety passage area, and a restricted area.

[0116] The whole-body bounding box of the target object is identified by the target detection model, and the target object is tracked continuously frame by frame using a multi-target tracking algorithm to obtain the real-time position coordinates of the target object in the image coordinate system. Combined with the pre-calibrated camera intrinsic and extrinsic parameters, the image coordinates are mapped to the laboratory physical coordinate system to obtain the physical position information of the target object.

[0117] The physical location information is compared with the spatial boundaries of each preset experimental area to determine the current experimental area of ​​the target object. The duration of stay and movement sequence of the target object in each area in a continuous time series are recorded to generate the experimental area activity information of the target object.

[0118] By identifying human bodies and protective equipment from experimental scene image data, the wearing status information of protective equipment is determined, specifically including:

[0119] Human instance segmentation is used to separate the target object from the image background;

[0120] Each of the preset detection items of the target object is tested one by one, including goggles for testing the eye area, lab coats for testing the torso area, and gloves for testing the hand area.

[0121] Output the protective equipment wearing status of each preset test item (such as goggles test, lab coat test, and gloves test). The protective equipment wearing status is one of the following: whether it is worn, whether it is worn correctly, or whether it has fallen off midway. Based on the protective equipment wearing status of each test part, generate protective equipment wearing status information.

[0122] Based on experimental operation information, experimental area activity information, and protective equipment wearing status information, experimental behavior information is determined, specifically including:

[0123] All experimental operation information, experimental area activity information, and protective equipment wearing status information belonging to the same time slice and the same experimental area are associated and bound together to obtain intermediate data;

[0124] Extract the corresponding compliance test results of experimental operation actions, experimental area activities, and protective equipment wearing from the intermediate data. Generate experimental behavior information based on the compliance test results of experimental operation actions, experimental area activities, and protective equipment wearing.

[0125] In some embodiments, taking a student conducting an experiment in a laboratory as an example, the relevant process of determining experimental behavior information based on experimental scene image data is explained in detail below, with the specific steps as follows:

[0126] Example scenario: A middle school chemistry lab; Target: Student Wang Wu, ID: 20240508, Lab table: Lab table number 5;

[0127] Step C1: Acquire experimental scene image data: The overhead camera on stage 5 captures real-time images, including: the student's upper body, hands, test tubes, dilute hydrochloric acid reagent bottle, alcohol lamp, and tabletop area;

[0128] Step C2 involves performing visual inspection on the experimental scene image data to determine the experimental operation action information, including:

[0129] Perform skeletal point detection: locate key points on the head, left hand, right hand, and elbow;

[0130] Perform 3D posture estimation: The distance between the head and the mouth of the test tube is >15cm, and the reagent is not smelled directly; the test tube is held with the test tube clamp in the right hand, not held directly in the hand; the test tube is tilted at about 45°, with the mouth of the tube facing the direction where no one is present; there are no violations such as blowing out the alcohol lamp or quickly tilting the tube.

[0131] Output the corresponding experimental operation information: standard, no violations;

[0132] Step C3 involves performing target localization and regional spatial analysis on the experimental scene image data to determine activity information in the experimental area, including:

[0133] Scene segmentation: Based on the semantic segmentation model, the experimental scene image is divided into multiple preset experimental areas, including the No. 5 experimental table operation area (the table surface and its edge extending 30cm outward), the reagent storage area (the designated shelf area on the left side of the table surface), the safety passage area (the passage between experimental tables), and the restricted area (the vertical projection area 20cm above the flame of the alcohol lamp).

[0134] Target detection and tracking: The target detection model identifies the full-body bounding box of student Wang Wu, and a multi-target tracking algorithm is used to track him continuously frame by frame to obtain his real-time position coordinates in the image coordinate system; combined with the pre-calibrated camera intrinsic and extrinsic parameters, the image coordinates are mapped to the laboratory physical coordinate system to obtain the student's physical position coordinates (x, y).

[0135] Perform spatial location comparison: compare the student's physical location coordinates with the spatial boundaries of each preset experimental area to determine the experimental area where the student is currently located; at the same time, record the duration of the student's stay in each area and the order of movement in the continuous time series.

[0136] Output experimental area activity information: The student remained in the operation area of ​​Experiment 5, did not enter the restricted area, did not enter the safety passage area of ​​the adjacent experiment table, and briefly stopped in the reagent storage area for 5 seconds to retrieve reagent bottles. The activity trajectory in the area complied with the experimental procedure specifications.

[0137] Step C4 involves identifying the human body and protective equipment in the experimental scene image data to determine the protective equipment wearing status information, including:

[0138] Human body instance segmentation and positioning of student regions;

[0139] Eye area detection: Features of goggles are present, completely covering the eyes, and correctly worn;

[0140] Torso region examination: Wearing a white lab coat, with the correct length and buttons fastened, in accordance with proper attire;

[0141] Hand area inspection: Wear rubber gloves, and wear them correctly;

[0142] Output: Protective equipment wearing status information: All protective equipment was worn correctly, and none was removed midway;

[0143] Step C5: Integrate and determine experimental behavior information

[0144] Experimental operation information: Compliant;

[0145] Experimental area activity information: In the area of ​​experimental table No. 5, no restricted areas were entered, and a safe distance was maintained;

[0146] Protective equipment wearing status information: Compliant;

[0147] The system integrates and outputs experimental behavior information of the target object: experimental operation specifications, compliance of regional activities, correct wearing of protective equipment, and overall compliance of behavior.

[0148] In some embodiments, reagent usage-related monitoring data includes reagent cabinet usage monitoring data, RFID data, weighing sensor monitoring data, and camera visual tracking data. The aforementioned visual analysis processing of reagent usage-related monitoring data to determine the target object's full-process reagent usage tracking information includes:

[0149] Analyze the reagent cabinet usage monitoring data to extract reagent usage and return information;

[0150] By collaboratively analyzing RFID data and camera visual tracking data, the movement trajectory information of the reagent from collection to return can be determined;

[0151] By linking and analyzing the monitoring data from the tachymeter sensor and the visual tracking data from the camera, information about the reagent usage process can be determined.

[0152] Based on reagent collection information, reagent return information, movement trajectory information, and usage process information, the entire process tracking information of reagent use for the target object is determined.

[0153] The reagent cabinet usage monitoring data refers to data collected by the reagent cabinet smart lock and cabinet door status sensors, such as cabinet door opening / closing time, operator identity, and cabinet door opening duration.

[0154] Radio frequency identification (RFID) data refers to the reagent bottle label information read by the reagent cabinet RFID reader, including reagent name, concentration, specifications, batch number, and unique identifier ID.

[0155] The weighing sensor monitoring data refers to the real-time weight data of the reagent bottles collected by the weighing sensor under the reagent cabinet shelf, which is used to calculate the amount taken, the amount used, and the amount remaining.

[0156] Camera visual tracking data refers to the data collected by panoramic cameras, laboratory bench cameras, and reagent cabinet cameras, including reagent location, bottle holder, movement path, and usage operation screen data.

[0157] The reagent dispensing information includes the time of dispensing, the amount dispensed, the operator's identity, and the reagent name.

[0158] The reagent return information includes the return time, returned quantity, return matching status, and remaining quantity.

[0159] The movement trajectory information includes the spatial movement path of the reagent from the reagent cabinet to the laboratory table for use and back to the reagent cabinet, the person holding the bottle, and the areas traversed.

[0160] Information on the usage process includes reagent usage time, actual dosage, usage actions, and usage location.

[0161] Optionally, reagent usage tracking information can be determined through relevant monitoring data, specifically including:

[0162] Analyze reagent cabinet usage monitoring data to extract reagent usage and return information, including:

[0163] Based on the cabinet door opening / closing signal and the operator's facial recognition results, the retrieval time and return time are determined;

[0164] By combining the weight difference before and after weighing the sensor, calculate the amount of reagent taken and returned;

[0165] Collaborative analysis of RFID data and camera visual tracking data determines reagent movement trajectory information, including:

[0166] The target reagent is located using a unique RFID ID;

[0167] The spatial location of the person holding the target reagent is determined by using a panoramic camera for target tracking.

[0168] The complete movement trajectory of reagent cabinet → experimental area → reagent cabinet is formed by splicing together the timestamps;

[0169] By linking and analyzing weighing data with visual tracking data, information on reagent usage processes can be determined, including:

[0170] Weighing difference calculation for actual usage;

[0171] Time difference calculation usage duration;

[0172] The experimental platform's camera motion recognition system identifies usage actions such as tilting, mixing, and placing.

[0173] Data integration yields end-to-end reagent usage tracking information, including:

[0174] Based on the same reagent ID, the same operator, and the same time period, the reagent retrieval information, reagent movement trajectory information, reagent usage process information, and reagent return information are integrated to form a structured full-process information.

[0175] In some embodiments, taking the end-to-end tracking of dilute hydrochloric acid reagent as an example, the relevant process for determining the end-to-end tracking information of reagent use based on relevant monitoring data is described in detail below, with the specific steps as follows:

[0176] Scene: A middle school chemistry lab, student Zhao Liu (identity number: 20240606), lab table number 6, experiment: the properties of acids;

[0177] Step D1: Obtain relevant monitoring data on the use of multi-source reagents, including:

[0178] Reagent cabinet usage monitoring data: Cabinet door opening time 14:28:15, closing time 14:28:28, operator Zhao Liu;

[0179] RFID data: Reagent bottle ID: RHCL006, Name: Concentrated hydrochloric acid, Concentration: 36%;

[0180] Weighing sensor data: Weight before use 620g, weight after use 580g;

[0181] Camera visual tracking data: footage of the person holding the bottle moving, footage of the experimental table being used, and footage of the bottle being returned.

[0182] Step D2: Analyze the reagent cabinet usage monitoring data, extract reagent usage information and reagent return information, including:

[0183] Reagent dispensing information: Dispensing time: 14:28:15; Amount dispensed: 620g - 580g = 40g (approximately 33mL); Operator: Zhao Liu; Reagent name: Concentrated hydrochloric acid;

[0184] Reagent return information: Return time: 14:45:30; Return quantity: Re-weighed weight 570g; Return matching: RFID ID matches, return to original cabinet and location;

[0185] Step D3: Collaboratively analyze RFID data and camera visual tracking data to determine movement trajectory information, including:

[0186] Location for retrieval: 3rd shelf of reagent cabinet No. 2;

[0187] Movement path: Reagent cabinet No. 2 → Central aisle → Experiment table No. 6;

[0188] Person in charge of the bottle: Zhao Liu; no passing, handing over, or removal from the laboratory was performed.

[0189] Return route: Lab bench #6 → Reagent cabinet #2;

[0190] Track status: Complete and compliant;

[0191] Step D4: Analyze the weighing data and visual tracking data together to determine usage process information, including:

[0192] Usage time: 14:28–14:45, a total of 17 minutes;

[0193] Dosage: Take 40g - return 50g = actual usage 10g;

[0194] Use the following actions:

[0195] Label facing palm, pour slowly, do not pour quickly, and do not mix with contraindicated reagents;

[0196] Location of use: Experimental table No. 6;

[0197] Step D5: Data integration to obtain full-process tracking information for reagent use, outputting structured tracking information as follows:

[0198] Reagent: Concentrated hydrochloric acid (ID: RHCL006); Operator: Zhao Liu (20240606); Picking: 14:28:15, volume picked up: 33mL; Track: Cabinet No. 2 → Station No. 6, no transfer or removal; Usage: Duration: 17 minutes, volume used: 10mL, operation in accordance with regulations; Return: 14:45:30, returned volume in compliance with regulations, returned completely.

[0199] Step 103: Based on preset judgment rules, perform integrated analysis on experimental behavior information and reagent usage full-process tracking information to identify the types of laboratory violations by the target object;

[0200] In some embodiments, the preset judgment rules include experimental behavior violation judgment rules and reagent use violation judgment rules, and step 103 includes the following sub-steps:

[0201] Based on the rules for judging experimental behavior violations, experimental behavior information is analyzed to identify each experimental behavior violation data and the corresponding experimental behavior violation type.

[0202] Based on the rules for determining reagent usage violations, the tracking information of the entire reagent usage process is analyzed to identify each reagent usage violation data and the type of reagent usage violation for each data point.

[0203] Based on the data on violations of experimental behaviors and reagent usage, we conducted analysis on time synchronization and behavioral correlation to determine the types of violation combinations and their associated data.

[0204] Based on the types of violations in each experimental behavior, the types of violations in each reagent use, and the types of combinations of violations, the types of laboratory violations by the target object are determined.

[0205] The rules for judging violations of experimental behavior refer to the set of rules used to determine whether an operation, protective clothing, or activity in a designated area is in violation of regulations. Examples include: not wearing goggles, pointing a test tube at a person, smelling reagents directly through the nose, and improper handling with open flames.

[0206] The rules for determining violations of reagent use refer to a set of rules used to determine whether the storage, movement, use, and return of reagents are in violation of regulations. Examples include: unauthorized use of reagents, unauthorized transfer, failure to return reagents on time, abnormal dosage, and removal of reagents from the laboratory.

[0207] Experimental behavior violation data refers to structured data that has been identified and characterizes the experimental behavior violations of the target object, including the location, time, action, and status of the violation.

[0208] The types of experimental behavior violations are categorized as follows: violations of protective clothing, violations of operational procedures, and violations of area activities.

[0209] Reagent use violation data refers to structured data that has been identified and characterizes violations in the reagent use process, including reagent ID, information on the use / return / dosage / trajectory anomalies.

[0210] The types of reagent usage violations are classifications of such violations, such as: unauthorized access, unauthorized transfer, failure to return reagents on time, abnormal dosage, and cabinet opening after the expiration date.

[0211] Time synchronization refers to determining whether violations of experimental behavior and violations of reagent use occur within the same time period and on the same experimental platform.

[0212] Behavioral correlation refers to determining whether there is a causal or accompanying relationship between experimental behavioral violations and reagent use violations, such as: not wearing goggles while simultaneously handling strong acid reagents.

[0213] Violation combination data refers to the combined data formed by binding experimental behavior violation data that is time-synchronized and behavior-related with reagent use violation data.

[0214] The violation combination type refers to the violation category classified according to the relationship, including: violation of a single experimental behavior, violation of a single reagent use, and violation of behavior-reagent linkage.

[0215] Optionally, based on the rules for judging violations of experimental behavior, the three pieces of information—operational actions, protective clothing, and area activities—are matched one by one to output data on violations of experimental behavior and their corresponding types.

[0216] Based on the rules for determining reagent usage violations, the four items of reagent collection information, movement trajectory information, usage process information, and reagent return information are matched one by one to output reagent usage violation data and corresponding types.

[0217] Based on the analysis of time synchronization and behavioral correlation, the illegal combination data and combination types are identified, including:

[0218] Time synchronization analysis: Based on timestamps, determine whether two types of violations occurred within the same time slice;

[0219] Behavioral association analysis: Determine whether there is a comorbid, causal, or dependent relationship between two types of violations, such as: handling hazardous reagents without protection, or mixing incompatible reagents during violation.

[0220] Time-synchronized and behavior-related violation data are bound together to form violation combination data, and this violation combination data is classified into one of the following violation combination types: single experimental behavior violation, single reagent use violation, and linkage violation of experimental behavior and reagent use.

[0221] In some embodiments, the above-described analysis of temporal synchronization and behavioral correlation based on the violation data of each experimental behavior and the violation data of each reagent use, to determine each violation combination data and the violation combination type associated with each violation combination data, includes:

[0222] Analyze the data on violations of each experimental behavior to determine the start time, end time, and associated information of each violation; the associated information includes the target experimental workstation and the identifier of the target associated reagent.

[0223] Analyze the data on reagent usage violations to determine the start time, end time, and associated information of each violation. The associated information includes the reagent identifier and the associated laboratory workstation.

[0224] When the behavior association information of one experimental behavior violation data is consistent with the reagent association information of one reagent use violation data, perform a time consistency analysis on one experimental behavior violation data and one reagent use violation data.

[0225] When the time consistency analysis shows that both the first time difference and the second time difference are within the preset time threshold, one of the experimental behavior violation data and one of the reagent use violation data are associated and bound to obtain the violation combination data; the first time difference is the difference between the behavior start time and the reagent use start time; the second time difference is the difference between the behavior end time and the reagent use end time.

[0226] Based on the types of experimental behavior violations and reagent use violations associated with the violation combination data, determine the types of violation combinations associated with the violation combination data.

[0227] Specifically, the following are extracted from experimental behavior violation data: behavior start time, behavior end time, target experimental operation station, and associated reagent ID; and the following are extracted from reagent usage violation data: reagent usage start time, reagent usage end time, reagent ID, and reagent-associated experimental operation station.

[0228] When the behavior association information of one of the experimental behavior violation data is consistent with the reagent association information of one of the reagent use violation data (including the target experimental operation station is consistent with the reagent-related experimental operation station, and the associated reagent ID is consistent with the reagent ID), it enters the time verification.

[0229] Then, the calculation of the execution time consistency check:

[0230] First time difference = |Beginning time of behavior − Beginning time of reagent use|

[0231] Second time difference = |Beginning time - End time of reagent use|

[0232] When both the first time difference and the first time difference do not exceed a preset threshold (such as 30 seconds), they are judged as the same violation event;

[0233] Finally, the experimental behavior violation data and reagent use violation data identified as the same violation event are bound together as violation combination data, and the corresponding experimental behavior violation type and reagent use violation type are superimposed to determine the violation combination type.

[0234] In some embodiments, an example of determining a combination of experimental conduct violations and reagent use violations is provided, with the specific steps as follows:

[0235] Basic Information Target: Student Zhou Ba Identifier: 20240808 Lab Station: Lab Table No. 8 Illegal Reagent: Dilute Nitric Acid (Reagent ID: HNO3_008, High-Risk Reagent) Preset Time Threshold: 30 seconds;

[0236] Step E1: Extract the spatiotemporal and correlation information of the experimental behavior violation data, specifically including:

[0237] Experimental violations: not wearing safety goggles, and directly heating test tubes by hand without using test tube clamps;

[0238] Beginning time of the behavior: 14:35:10;

[0239] End time of the behavior: 14:36:05;

[0240] Target experimental workstation: Experimental table No. 8;

[0241] Target-associated reagent identifier: HNO3_008;

[0242] Step E2 involves extracting the spatiotemporal and correlation information of the reagent usage violation data, specifically including:

[0243] Reagent use violations: unauthorized access to high-risk reagents, failure to return reagents promptly after use;

[0244] Reagent usage start time: 14:35:15;

[0245] Reagent usage ended at 14:36:10;

[0246] Reagent label: HNO3_008;

[0247] Reagent-related experimental operation station: Experimental table No. 8;

[0248] Step E3, consistency judgment of related information, includes:

[0249] Behavior-related workstation: Experimental station No. 8;

[0250] Reagent associated workstation: Lab bench #8, workstation consistency is determined.

[0251] Behavior-related reagent ID: HNO3_008;

[0252] Reagent ID: HNO3_008, reagent consistency confirmed;

[0253] If the associated information is consistent, proceed to time consistency analysis;

[0254] Step E4, time consistency analysis, specifically includes:

[0255] Calculate the time difference:

[0256] First time difference = |14:35:10−14:35:15| = 5 seconds ≤ 30 seconds

[0257] Second time difference = |14:36:05−14:36:10| = 5 seconds ≤ 30 seconds

[0258] If both differences are less than the preset time threshold, the judgment time is consistent.

[0259] Step E5: Obtain the data of the illegal combination by associating and binding, including:

[0260] Group the following two violations together:

[0261] Experimental violations: failure to wear safety goggles, failure to use test tube clamps;

[0262] Reagent usage violation: Unauthorized use of dilute nitric acid and failure to return it in a timely manner;

[0263] The following combinations of non-compliant data were generated:

[0264] At experiment table #8, student Zhou Ba, from 14:35:10 to 14:36:10, did not wear goggles and did not use test tube clamps. He also took dilute nitric acid without authorization and did not return it in time. His behavior was highly related to the use of reagents.

[0265] Step E6, determine the types of violations, including:

[0266] Types of violations in experimental conduct: violations of protective clothing requirements + violations of operational procedures;

[0267] Types of reagent usage violations: Unauthorized access violation + Overdue return violation;

[0268] The final determination is as follows: Type of violation combination = linkage between experimental behavior violation and reagent use violation (high-risk combination violation) = violation of protective clothing + violation of operation action + violation of unauthorized access + violation of return time limit.

[0269] Step 104: Activate the corresponding target early warning strategy based on the type of laboratory violation.

[0270] In some embodiments, step 104 includes the following sub-steps:

[0271] Obtain the preset mapping relationship between violation types and early warning strategies;

[0272] Match the target early warning strategy for handling laboratory violations from the preset mapping relationship and activate the target early warning strategy; the target early warning strategy shall include at least the target early warning level, early warning prompt method, early warning notification recipients and violation handling guidance information.

[0273] The preset mapping relationship is a pre-configured rule base that corresponds one-to-one between violation types and warning strategies, used to automatically match warning levels and strategies.

[0274] The target warning level is a warning level divided according to the safety risk, for example: Level 1 (minor), Level 2 (relatively serious), and Level 3 (serious / urgent).

[0275] Warning notification methods refer to the ways in which warnings are issued, such as: local voice reminders, laboratory bench light prompts, laboratory large screen prompts, App push notifications, sound and light alarms, and access control linkage, etc.

[0276] Optionally, the preset mapping relationship of "violation type - early warning strategy" can be retrieved from the system rule base;

[0277] The identified violation types are compared with preset mapping relationships to locate the matching target warning strategy, and the target warning level, warning prompt method, warning notification recipient and violation handling guidance information are obtained from the target warning strategy.

[0278] Finally, the target early warning strategy is initiated and executed, automatically triggering the corresponding notification channels for the early warning notification method, and simultaneously pushing early warning notifications and handling instructions to the corresponding early warning notification recipients, thus completing the closed-loop early warning.

[0279] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0280] The following describes the computer vision-based laboratory behavior monitoring and early warning device provided in the embodiments of this application. The computer vision-based laboratory behavior monitoring and early warning device described below can be referred to in correspondence with the computer vision-based laboratory behavior monitoring and early warning method described above.

[0281] Reference Figure 2 , Figure 2 This is an optional structural diagram of a computer vision-based laboratory behavior monitoring and early warning device provided in an embodiment of the present invention. This device is used to implement the aforementioned computer vision-based laboratory behavior monitoring and early warning method, and may include:

[0282] The visual monitoring module 200 is used to collect experimental scene image data associated with the target object and reagent usage-related monitoring data;

[0283] The visual inspection module 300 is used to perform visual inspection on experimental scene image data and reagent usage-related monitoring data to determine experimental behavior information and reagent usage full-process tracking information.

[0284] The violation analysis module 400 is used to perform fusion analysis on experimental behavior information and reagent usage full-process tracking information based on preset judgment rules to identify the types of laboratory violations of the target object.

[0285] The violation warning module 500 is used to activate corresponding target warning strategies based on the type of violation in the laboratory.

[0286] In some embodiments, the visual detection module 300 specifically includes:

[0287] The experimental behavior detection unit is used to perform visual recognition processing on experimental scene image data to determine the experimental behavior information of the target object.

[0288] The reagent usage detection unit is used to perform visual analysis and processing of monitoring data related to reagent usage, and to determine the full-process tracking information of reagent usage for the target object.

[0289] In some embodiments, the experimental behavior detection unit specifically performs the following functions:

[0290] Skeletal point detection and pose estimation are performed on experimental scene image data to determine the experimental operation action information of the target object;

[0291] Target localization and regional spatial analysis are performed on the experimental scene image data to determine the activity information of the experimental area;

[0292] Human body and protective equipment recognition processing is performed on experimental scene image data to determine the protective equipment wearing status information of the target object;

[0293] Based on experimental operation information, experimental area activity information, and protective equipment wearing status information, the experimental behavior information of the target object is determined.

[0294] In some embodiments, the reagent uses the detection unit to specifically perform the following functions:

[0295] Analyze the reagent cabinet usage monitoring data to extract reagent usage and return information;

[0296] By collaboratively analyzing RFID data and camera visual tracking data, the movement trajectory information of the reagent from collection to return can be determined;

[0297] By linking and analyzing the monitoring data from the tachymeter sensor and the visual tracking data from the camera, information about the reagent usage process can be determined.

[0298] Based on reagent collection information, reagent return information, movement trajectory information, and usage process information, the entire process tracking information of reagent use for the target object is determined.

[0299] In some embodiments, the violation analysis module 400 specifically includes:

[0300] The experimental behavior violation analysis unit is used to analyze experimental behavior information based on experimental behavior violation judgment rules, identify each experimental behavior violation data and the corresponding experimental behavior violation type;

[0301] The reagent use violation analysis unit is used to analyze the reagent use full-process tracking information based on the reagent use violation judgment rules, and to identify each reagent use violation data and the type of reagent use violation for each data.

[0302] The violation combination analysis unit is used to analyze the time synchronization and behavior correlation of violation data of each experimental behavior and each reagent use violation data, and to determine the violation combination data and the violation combination type associated with each violation combination data.

[0303] The violation type integration unit is used to determine the type of laboratory violation of the target object based on the violation type of each experimental behavior, the violation type of each reagent use, and the violation combination type.

[0304] In some embodiments, the violation combination analysis unit specifically performs the following functions:

[0305] Analyze the data on violations of each experimental behavior to determine the start time, end time, and associated information of each violation; the associated information includes the target experimental workstation and the identifier of the target associated reagent.

[0306] Analyze the data on reagent usage violations to determine the start time, end time, and associated information of each violation. The associated information includes the reagent identifier and the associated laboratory workstation.

[0307] When the behavior association information of one experimental behavior violation data is consistent with the reagent association information of one reagent use violation data, perform a time consistency analysis on one experimental behavior violation data and one reagent use violation data.

[0308] When the time consistency analysis shows that both the first time difference and the second time difference are within the preset time threshold, one of the experimental behavior violation data and one of the reagent use violation data are associated and bound to obtain the violation combination data; the first time difference is the difference between the behavior start time and the reagent use start time; the second time difference is the difference between the behavior end time and the reagent use end time.

[0309] Based on the types of experimental behavior violations and reagent use violations associated with the violation combination data, determine the types of violation combinations associated with the violation combination data.

[0310] In some embodiments, the violation warning module 500 specifically performs the following functions:

[0311] Obtain the preset mapping relationship between violation types and early warning strategies;

[0312] Match the target early warning strategy for handling laboratory violations from the preset mapping relationship and activate the target early warning strategy; the target early warning strategy shall include at least the target early warning level, early warning prompt method, early warning notification recipients and violation handling guidance information.

[0313] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0314] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0315] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0316] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0317] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0318] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned computer vision-based laboratory behavior monitoring and early warning method. This electronic device can be any smart terminal, including a tablet computer.

[0319] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0320] Please see Figure 3 , Figure 3 The hardware structure of an electronic device according to another embodiment is illustrated, including:

[0321] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0322] The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 to implement the computer vision-based laboratory behavior monitoring and early warning method of the embodiments of this invention.

[0323] The input / output interface 903 is used to implement information input and output;

[0324] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0325] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0326] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0327] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned computer vision-based laboratory behavior monitoring and early warning method.

[0328] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0329] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0330] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A computer vision-based laboratory behavior monitoring and early warning method, characterized in that, include: Collect experimental scene image data and reagent usage-related monitoring data associated with the target object; Visual inspection is performed on the experimental scene image data and reagents using relevant monitoring data to determine experimental behavior information and reagent usage tracking information throughout the entire process. Based on preset judgment rules, the experimental behavior information and the reagent usage full-process tracking information are fused and analyzed to identify the types of laboratory violations by the target object. Based on the type of laboratory violation, the corresponding target early warning strategy will be activated.

2. The computer vision based laboratory behavior monitoring and alerting method of claim 1, wherein, The visual inspection of the experimental scene image data and reagent usage monitoring data to determine experimental behavior information and reagent usage full-process tracking information includes: Visual recognition processing is performed on the experimental scene image data to determine the experimental behavior information of the target object; Visual analysis is performed on the relevant monitoring data of the reagents to determine the full-process tracking information of the reagents used by the target object.

3. The computer vision based laboratory behavior monitoring and alerting method of claim 2, wherein, The step of performing visual recognition processing on the experimental scene image data to determine the experimental behavior information of the target object includes: Skeletal point detection and pose estimation are performed on the experimental scene image data to determine the experimental operation action information of the target object; The experimental scene image data is used to perform target localization and regional spatial analysis to determine the experimental area activity information of the target object; The experimental scene image data is processed to identify human body and protective equipment, and the protective equipment wearing status information of the target object is determined. Based on the experimental operation action information, the experimental area activity information, and the protective equipment wearing status information, the experimental behavior information of the target object is determined.

4. The computer vision based laboratory behavior monitoring and alerting method of claim 2, wherein, The reagent usage-related monitoring data includes reagent cabinet usage monitoring data, RFID data, weighing sensor monitoring data, and camera visual tracking data. The visual analysis and processing of the reagent usage-related monitoring data to determine the full-process tracking information of the target object's reagent usage includes: The reagent cabinet usage monitoring data is analyzed to extract reagent usage and return information; The radio frequency identification data and the camera visual tracking data are analyzed in a coordinated manner to determine the movement trajectory information of the reagent from its collection to its return; The monitoring data from the weighing sensor and the visual tracking data from the camera are analyzed together to determine the reagent usage process information. Based on the reagent retrieval information, the reagent return information, the movement trajectory information, and the usage process information, the full-process tracking information of the reagent usage of the target object is determined.

5. The computer vision based laboratory behavior monitoring and alerting method as claimed in claim 1, wherein, The preset judgment rules include experimental behavior violation judgment rules and reagent use violation judgment rules. Based on the preset judgment rules, the experimental behavior information and the reagent use full-process tracking information are fused and analyzed to identify the types of laboratory violations of the target object, including: Based on the aforementioned experimental behavior violation judgment rules, the experimental behavior information is analyzed to identify each experimental behavior violation data and the experimental behavior violation type corresponding to each experimental behavior violation data. Based on the reagent use violation judgment rules, the entire process tracking information of reagent use is analyzed to identify each reagent use violation data and the reagent use violation type of each reagent use violation data; Based on the violation data of each experimental behavior and the violation data of each reagent use, time synchronization and behavior correlation analysis are performed to determine each violation combination data and the violation combination type associated with each violation combination data; Based on the types of experimental behavior violations, reagent use violations, and combinations of violations, the types of laboratory violations by the target object are determined.

6. The computer vision based laboratory behavior monitoring and alerting method of claim 5, wherein, The step involves analyzing the temporal synchronization and behavioral correlation of the data on violations of experimental behaviors and reagent usage, determining each combination of violations and the associated violation combination types, including: Analyze the violation data of each experimental behavior to determine the start time, end time and related information of each violation data. Analyze the data on violations of reagent use for each of the aforementioned reagents to determine the start time, end time, and associated information of reagent use corresponding to each data point. When the behavior association information of one experimental behavior violation data is consistent with the reagent association information of one reagent use violation data, a time consistency analysis is performed on the one experimental behavior violation data and the one reagent use violation data. When the time consistency analysis shows that both the first time difference and the second time difference are within a preset time threshold, the violation data of one of the experimental behaviors and the violation data of one of the reagent uses are associated and bound to obtain violation combination data; the first time difference is the difference between the start time of the behavior and the start time of the reagent use; the second time difference is the difference between the end time of the behavior and the end time of the reagent use. Based on the types of experimental behavior violations and reagent use violations associated with the violation combination data, the violation combination type associated with the violation combination data is determined.

7. The computer vision based laboratory behavior monitoring and alerting method of claim 1, wherein, The step of activating a corresponding early warning strategy based on the type of laboratory violation includes: Obtain the preset mapping relationship between violation types and early warning strategies; The target early warning strategy for handling the laboratory violation type is matched from the preset mapping relationship and activated; the target early warning strategy includes at least the target early warning level, early warning prompt method, early warning notification target, and violation handling guidance information.

8. A computer vision based laboratory behavior monitoring and alerting device, characterized in that, include: The visual monitoring module is used to collect experimental scene image data associated with the target object and related monitoring data on reagent usage; The visual inspection module is used to perform visual inspection on the experimental scene image data and reagent usage-related monitoring data to determine experimental behavior information and reagent usage full-process tracking information. The violation analysis module is used to perform fusion analysis on the experimental behavior information and the reagent usage full-process tracking information based on preset judgment rules, and to identify the types of laboratory violations of the target object; A violation early warning module is configured to start a corresponding target early warning strategy according to the laboratory violation behavior type.

9. An electronic device, comprising: A computer program product comprises a memory and a processor, the memory storing a computer program, and the computer program being executed by the processor to make the processor execute the steps of the computer vision-based laboratory behavior monitoring and early warning method according to any one of claims 1-7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed to implement the computer vision-based laboratory behavior monitoring and early warning method according to any one of claims 1-7.

Images