A laboratory intelligent monitoring method and system based on multi-model fusion

Abstract

This invention provides a laboratory intelligent monitoring method and system based on multi-model fusion, relating to the field of intelligent monitoring technology combining computer vision and deep learning. The method acquires real-time video streams from multiple cameras in the laboratory. After video frame preprocessing, the video frames are input into a customized model built on a LAB-YOLO network to detect targets such as personnel clothing, behaviors, and equipment door opening / closing states. For different monitoring object characteristics, LAB-YOLO achieves anomaly detection with limited target sample states, while LAB-YOLO combined with an improved residual network achieves anomaly detection with infinite target sample states. Anomalies are verified through multi-frame verification to eliminate false detections. A new real-time video stream processing module aggregates the results of multiple algorithms into a single frame and pushes the output via a real-time streaming protocol, ultimately generating abnormal video clips, images, and information, and pushing alarms.

Description

Technical Field

[0001] This invention relates to the field of intelligent monitoring technology combining computer vision and deep learning, and in particular to a laboratory intelligent monitoring method and system based on multi-model fusion. Background Technology

[0002] Existing laboratory monitoring systems mostly rely on manual inspections or simple image recognition technology, which has the following shortcomings:

[0003] Limited detection accuracy: The accuracy of identifying personnel wearing proper attire, complex behaviors, and equipment status is low, and false alarms or missed alarms are prone to occur.

[0004] Insufficient real-time performance: When multiple cameras are collecting data concurrently, it is difficult to process video streams quickly and issue timely alarms, and it lacks the ability to synchronously aggregate the results of multiple algorithms.

[0005] Poor adaptability: The detection and recognition performance drops significantly in complex scenarios such as changes in laboratory lighting and target occlusion.

[0006] Low level of intelligence: It lacks the ability to comprehensively analyze and reason about various types of information, cannot cope with complex and abnormal situations, and does not support real-time video streaming output. Summary of the Invention

[0007] Purpose of the invention: Based on the problems mentioned in the background technology, this invention proposes a laboratory intelligent monitoring method and system based on multi-model fusion. The aim is to monitor and detect anomalies in laboratory personnel's attire, behavior, and equipment door opening and closing status in real time and accurately. By adding a real-time video stream processing module, the invention achieves synchronous aggregation and streaming output of multi-algorithm results, and promptly issues alarm information and records abnormal data, thereby improving the automation and intelligence level of laboratory safety management.

[0008] In a first aspect, this invention proposes a laboratory intelligent monitoring method based on multi-model fusion, comprising the following steps:

[0009] Deploy multiple cameras to collect real-time video streams of the target space, assign a unique frame identifier to each frame image in the real-time video stream, and store the frame images containing the frame identifiers into a shared frame queue.

[0010] A sample judgment model is constructed, in which the LAB-YOLO network structure is used as the finite sample judgment model and the ResLab-Net network structure is used as the infinite sample judgment model.

[0011] The frame image containing the frame identifier is obtained from the frame queue and imported into the sample judgment model. First, the LAB-YOLO network structure is used to determine whether there is an anomaly in the category of the finite sample. Then, the ResLab-Net network structure is used to detect the features of the infinite sample and determine whether the feature meets the preset conditions. If it does not meet the preset conditions, the anomaly result is stored in the result queue along with the frame identifier.

[0012] When an anomaly is detected, it is verified from several consecutive frames. If the number of times the anomaly occurs exceeds a preset value, it is confirmed as a real anomaly. A video clip containing the result of the real anomaly is generated, and the alarm annotation image and alarm original image are saved. The time node, location of occurrence and alarm content are recorded.

[0013] In a further embodiment of the first aspect, the LAB-YOLO network structure is used as a finite sample judgment model, and the finite samples include gloves, two-person mutual inspection behavior, fall behavior, and device door opening status;

[0014] The LAB-YOLO network structure includes a Backbone network, in which the C3k2 module is replaced by a multi-scale adaptive feature extraction module (MSAFE), as shown in the following expression:

[0015]

[0016] In the formula, β and γ are adaptive weight parameters that are dynamically adjusted through an attention weight allocation mechanism;

[0017] Lab-SPPF, a spatial pyramid pooling module, is constructed for laboratory scenarios. Its expression is as follows:

[0018] Lab-SPPF(F)

[0019] =Conv 1×1 ([F, MaxPool) 5×5 (F), MaxPool 9×9 (F),MaxPool 13×13 (F), AvgPool 5×5 (F)])

[0020] In the formula, [*] represents channel dimension concatenation; Conv 1×1 Used to adjust the number of channels.

[0021] In a further embodiment of the first aspect, the LAB-YOLO network structure further includes a Neck network and a Head network;

[0022] In the Neck network, the C3k2 module uses 2×2 convolutional kernels combined with depthwise separable convolutions to reduce computational complexity; a bidirectional feature fusion enhancement module (BFEM) is designed, employing a cross-attention mechanism, as shown in the following formula:

[0023] BFEM(F high ,F low ) = Attention(F high ,F low )⊙F high +Attention(F low ,F high )⊙F low

[0024] In the formula, ⊙ represents element-wise multiplication; Attention(Q,K) is the cross-attention function;

[0025] The Head network employs a decoupled detection head designed for laboratory scenarios, introducing behavioral semantic understanding and confidence branches on top of the traditional classification and regression branches. The structure of the decoupled detection head, Lab-DetHead, is as follows:

[0026] Lab-DetHead=[ClassBranch,BBoxBranch,BehaviorBranch,ConfidenceBranch]

[0027] In the formula, ClassBranch is the traditional classification branch; BBoxBranch is the regression branch; BehaviorBranch is the behavior semantic understanding branch; and ConfidenceBranch is the confidence branch.

[0028] In a further embodiment of the first aspect, the ResLab-Net network structure is used as an infinite sample judgment model, wherein the infinite samples include shoe colors;

[0029] In the ResLab-Net network structure, the 3×3 convolutions in ResNet-34 are replaced with multi-scale residual fusion blocks (MSRBs) composed of multiple convolutions of different sizes.

[0030]

[0031] In the formula, x represents the residual block input feature. β and γ are the fusion weights; the output features are fused by adding them element by element.

[0032] The ResLab-Net network structure uses ResNet-34 as the backbone network, removes the original 1000-dimensional classification layer, and introduces a 128-dimensional fully connected layer as a bottleneck feature layer after global average pooling (GAP) for anomaly identification by fusing with the detection results; the bottleneck feature formula is as follows:

[0033] F bottleneck =FC 128 (GAP(x))

[0034] In the formula, GAP represents global average pooling; FC 128 It is a 128-dimensional fully connected layer that maps pooling results to low-dimensional bottleneck features.

[0035] In a further embodiment of the first aspect, a laboratory scene feature enhancement module LSFEM is added after each multi-scale residual fusion block (MSRB) to model the relationship between channels and space based on a coordinate attention mechanism. Its structure is as follows:

[0036] LSFEM(x)=x⊙σ(Conv 1×1 (GAP(x)))+x⊙σ(Conv 1×1 (GMP(x)))

[0037] In the formula, GAP represents global average pooling; GMP represents global max pooling; and σ represents the sigmoid function.

[0038] The coordinate attention generation process is as follows:

[0039] Coordinate information embedding:

[0040]

[0041] In the formula, x c The input feature is the Cth channel; h and w are the height and width of the feature map; and These are aggregation features along the height and width directions, respectively;

[0042] Channel attention generation:

[0043] s=σ(FC r (ReLU(FC r ([z h ;z W ]))))

[0044] In the formula, [z h ;z W [ ] represents concatenation along the channel dimension; FC r For dimension reduction fully connected layers; σ is the sigmoid function;

[0045] Attention weighting:

[0046] x′ C =x C ·s C

[0047] In the formula, s C x′ represents the attention weight for the c-th channel. C The weighted output features.

[0048] In a further embodiment of the first aspect, the frame queue and the result queue are implemented based on Python's thread module and queue module. The frame queue manages frame images according to the first-in-first-out principle and discards the earliest frame when the capacity is exceeded to prevent memory overflow.

[0049] In a further embodiment of the first aspect, the real-time video stream uses a shared frame queue to achieve frame synchronization during streaming. The processed frames are pushed to the RTMP server in FLV format using FFmpeg. The detection results are aggregated using the Multimodal Feature Fusion (MMFA) algorithm, as shown in the following formula:

[0050]

[0051] In the formula, w i The dynamic weights of the i-th algorithm are calculated using a confidence-weighted strategy; R i This represents the detection result of the i-th algorithm;

[0052] At the same time, adaptive bitrate adjustment is used to ensure streaming quality:

[0053] BitRate adaptive =BitRate base ×(1+β×ConplexityFactor)

[0054] In the formula, BitRate base β is the base bitrate; β is the complexity adjustment coefficient; and ComplexityFactor is the video content complexity factor.

[0055] In a further embodiment of the first aspect, when an anomaly is detected, a timing consistency verification is performed from a series of consecutive frames. If the number of anomalies exceeds a threshold θ, ... threshold If so, it is confirmed as a genuine anomaly;

[0056] The formula for verifying timing consistency is as follows:

[0057]

[0058] In the formula, A t The result of anomaly detection in frame t; M is the verification window size; w i For time series weights; I(A)i ) is the exception indicator function; when TCV(A t )>θ threshold When confirmed as a genuine anomaly, θ threshold For dynamic thresholds;

[0059] Take K frames and subsequent K frames from queue A to generate video clips, save alarm annotation images and original alarm images, and record time nodes, occurrence locations and alarm content;

[0060] Queue A is used to store consecutive frame images in the video stream. When no anomaly is detected, queue A is updated in real time; when an anomaly is detected, historical frames are extracted from queue A to generate video clips.

[0061] In a second aspect, the present invention proposes a laboratory intelligent monitoring system based on multi-model fusion, which includes a data acquisition module, an algorithm processing module, a queue management module, a real-time streaming module, and an alarm and storage module.

[0062] The data acquisition module acquires real-time video streams of the target space through multiple cameras, assigns a unique frame identifier to each frame in the real-time video stream, and stores the frame images containing the frame identifiers into a shared frame queue.

[0063] The algorithm processing module is used to construct a sample judgment model, in which the LAB-YOLO network structure is used as a finite sample judgment model and the ResLab-Net network structure is used as an infinite sample judgment model. Frame images containing the frame identifier are obtained from the frame queue and imported into the sample judgment model. First, the LAB-YOLO network structure is used to determine whether there are any anomalies in the category of the finite samples. Then, the ResLab-Net network structure is used to detect the features of the infinite samples and determine whether the features meet the preset conditions. If they do not meet the conditions, the anomaly result is stored in the result queue along with the frame identifier.

[0064] The queue management module coordinates data flow between the two queues; the frame queue stores pre-processed images; the result queue stores algorithm detection results; and a unique frame identifier ensures a one-to-one correspondence between images and results.

[0065] The real-time streaming module is used to draw the detection results on the original screen, and push the processed screen to the RTMP server in the form of a video stream via FFmpeg. It supports real-time viewing on the laboratory management platform. The detection results include anomaly boxes, labels, and confidence scores.

[0066] The alarm and storage module is used to perform multi-frame verification through the timing consistency verification module after an anomaly is detected, save the abnormal video clips, alarm original images and alarm annotation images, record the time node, occurrence location and alarm content, and push the alarm to the laboratory management platform.

[0067] In a third aspect, the present invention provides an electronic device comprising at least one processor and a memory communicatively connected to at least one of the processors; wherein the memory stores instructions executable by the processor to implement the multi-model fusion-based intelligent laboratory monitoring method described in the first aspect.

[0068] Beneficial Effects: This invention proposes a multi-model fusion-based intelligent laboratory monitoring method and system. Employing a custom LAB-YOLO model combined with a ResLab-Net classification model for anomaly detection, it can accurately identify personnel attire, behavior, and equipment door opening / closing. Through optimized frame processing mechanisms, it achieves real-time detection and alarm for multiple monitoring channels, supporting real-time video output. It performs stably in complex scenarios such as changing lighting and target occlusion, eliminating false detections and recording and pushing anomaly information, ensuring consistent results, improving the reliability and intelligence of laboratory monitoring, and contributing to efficient and safe management. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein:

[0070] Figure 1 This is a schematic diagram of the overall process of the laboratory intelligent monitoring method based on multi-model fusion in the embodiment.

[0071] Figure 2 The diagram shows the LAB-YOLO network structure in the example.

[0072] Figure 3 The ResLab-Net network structure is shown in the example.

[0073] Figure 4 This is a diagram of the anomaly detection structure in the embodiment.

[0074] Figure 5 This is a diagram illustrating video frame preprocessing and queue data flow in the embodiment.

[0075] Figure 6 This is a diagram illustrating the real-time video stream processing and alarm structure in the embodiment.

[0076] Figure 7 This is an architecture diagram of a laboratory intelligent monitoring system based on multi-model fusion, as shown in the embodiment. Detailed Implementation

[0077] In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without one or more of these details. In other instances, certain technical features well-known in the art have not been described in order to avoid obscuring the invention.

[0078] Example 1:

[0079] This embodiment discloses a laboratory intelligent monitoring method based on multi-model fusion, the overall structure of which is shown below. Figure 1 The specific steps are as follows:

[0080] Step 1: Data Acquisition and Preprocessing

[0081] a. Multi-channel video acquisition: Deploy multiple high-definition cameras in key areas of the laboratory to acquire real-time video streams. Start an independent read_frame thread, assign a unique frame_id to each frame, and store the frame image containing the frame_id into a shared frame_queue.

[0082] b. Video frame preprocessing:

[0083] Improved Limiting Contrast Adaptive Histogram Equalization: The base contrast limit threshold C is set to 2.5, which enhances local image contrast while avoiding noise amplification.

[0084] Multi-layer gamma correction: Through adaptive calculation of regional brightness distribution, γ is set to 0.4 when the light is too dim, γ is set to 1.8 when the light is too bright, and γ is set to 1.0 when the light is normal.

[0085] Noise reduction filtering: An improved multi-scale nonlocal mean filtering is adopted, combined with an edge protection mechanism, to ensure the stability of subsequent detection.

[0086] Step 2: Target Detection

[0087] a. Model Loading and Configuration: Load the pre-trained LAB-YOLO custom model, which has optimized the network structure and adjusted the training parameters for different laboratory scenarios, and accelerated the process with TensorRT quantization.

[0088] b. Detection parameter settings: Set the detection box confidence threshold to 0.7 or higher, and the non-maximum suppression (NMS) IoU threshold to 0.5 to filter detection results with low confidence and excessive overlap.

[0089] c. Target detection range: Detect category objects and output the detection bounding box with frame_id and category information, and store it in result_queue.

[0090] Step 3: Anomaly Detection

[0091] a. Limited sample state anomaly identification: Directly identify anomalies (such as gloves, falls, equipment door opening and closing, etc.) using a model trained with LAB-YOLO.

[0092] b. Anomaly identification in an infinite sample state: For categories such as shoes, the detection results of the LAB-YOLO-trained model are input into the classification model trained by ResLab-Net. The detection-classification process distinguishes between laboratory-specified color / model slippers and other color / model slippers. Shoes that are not laboratory-specified are judged as abnormal.

[0093] c. Result storage: All anomaly detection results are associated with frame_id and stored in result_queue.

[0094] Step 4: Result Aggregation and Real-time Video Stream Processing

[0095] a. Multi-algorithm result aggregation: Call the draw_results function to extract all detection results of the same frame from the result_queue by frame_id and draw them on the frame image. The multi-algorithm result aggregation mechanism aggregates detection results through the MMFA algorithm.

[0096] b. Real-time streaming output: The processed frames are pushed to the RTMP server in FLV format via FFmpeg, and RTSP protocol output is supported. The laboratory management platform can view the video stream with anomaly annotations in real time through the RTSP address.

[0097] Step 5: Anomaly Detection and Alarm

[0098] a. Multi-frame verification: When an anomaly is detected, M frames are continuously verified. If the anomaly occurs ≥ N times (N is a configurable size), it is confirmed as a real anomaly.

[0099] b. Anomaly Recording and Push: Extract the K frames before the anomaly (e.g., K=5) and the last 5 frames from queue A (e.g., storing the most recent 300 frames of images), generate an H.264 encoded video clip, and save the original alarm image, labeled image, and anomaly information (time, location, alarm category). Push the clip to the laboratory management platform via HTTP protocol to trigger an alarm.

[0100] Example 2:

[0101] Based on Example 1, this example discloses the details of the execution steps of a laboratory intelligent monitoring method based on multi-model fusion.

[0102] Data Acquisition and Preprocessing: Real-time video streams from the biological laboratory are acquired using multiple deployed cameras. An independent read_frame thread is initiated to assign a unique frame identifier (frame_id) to each image frame, and the frames containing the frame_id are stored in a shared frame queue. Adaptive environmental awareness preprocessing is performed on the video frames, including adaptive histogram equalization, gamma correction, and noise reduction filtering. The intelligent histogram equalization employs an improved contrast-limited adaptive histogram equalization method, incorporating an adaptive adjustment factor for ambient brightness. The contrast limit threshold is dynamically adjusted to:

[0103]

[0104] In the formula, c base =2.5 is the basic contrast limit threshold; L Current The current ambient light level; L standard Standard ambient light; This is an adaptive adjustment factor.

[0105] Multi-layer gamma correction employs a region-based adaptive gamma correction algorithm, dividing the image into nine regions, with each region independently adjusting its gamma value based on local brightness characteristics.

[0106]

[0107] In the formula, γ region(x,y) This is the dynamic gamma value of the region where the pixel (x,y) is located; it is adaptively calculated based on the regional brightness distribution (γ=0.4 when the light is too dim, γ=1.8 when the light is too bright, and γ=1.0 when the light is normal).

[0108] The intelligent noise reduction filter utilizes an improved multi-scale nonlocal mean filter, combined with an edge protection mechanism. The filter strength formula is as follows:

[0109]

[0110] In the formula, h base The basic filter strength; σ is the gradient magnitude; σ is the edge sensitivity parameter, used to remove random particle noise in video frames and ensure the stability of subsequent detection.

[0111] Object detection: All algorithm threads retrieve frame images containing frame_id from the frame_queue. A custom model built with LAB-YOLO (Laboratory Adaptive Behavior-YOLO) and a classification model trained with ResLab-Net (Residual Laboratory Network) are used to detect anomalies in personnel clothing (shoes, gloves, etc.), behavior (mutual inspection, falls, etc.), and the opening / closing status of equipment doors. Detection boxes containing frame_id and category information are output, and the results are stored in the result queue (result_queue). The confidence threshold for detection boxes is set to 0.7 or higher. The confidence score reflects the likelihood that the box contains the target and that the predicted category is true. The formula is as follows:

[0112] Conf=p (object) ×IoU

[0113] In the formula, p (object) The probability of an object being detected within the bounding box is given; the Intersection over Union (IoU) is the ratio of the predicted bounding box to the ground truth bounding box.

[0114] Non-maximum suppression (NMS) is used to filter out detection boxes with excessive overlap. For detection boxes of the same category, their intersection-over-union (IoU) is calculated, with an IoU threshold set to 0.5. The IoU formula is as follows:

[0115]

[0116] In the formula, A and B are two detection boxes; Area(A∩B) is the area of ​​the intersection of the two boxes; Area(A∪B) is the area of ​​the union of the two boxes.

[0117] Anomaly detection: For categories such as gloves, mutual inspection between two people, falls, and equipment door opening / closing status, LAB-YOLO is used to directly determine whether the category status is abnormal; for categories such as shoes, the shoes detected by LAB-YOLO are input into the ResLab-Net network, and the classification model trained by the improved ResLab-Net network is used to determine whether they are slippers of the specified color and model in the laboratory. Slippers of different colors and models are judged as abnormal; the anomaly detection results of the above two different methods include frame_id and are stored in result_queue;

[0118] Results aggregation and real-time video stream processing: A new result drawing function (draw_results) is added, which aggregates all algorithm results from the result_queue by frame_id. After all results for the same frame_id are collected, all detection boxes, labels and anomaly markers are drawn on the corresponding frame image. The processed frames are pushed to the Real-Time Messaging Protocol (RTMP) server in FLV format via FFmpeg, and real-time output of RTSP protocol is supported.

[0119] Anomaly detection and alarm: When an anomaly is detected, it is verified from M consecutive frames. If the anomaly occurs more than N times (N is configurable), it is confirmed as a real anomaly. K frames and subsequent K frames are taken from queue A to generate a video clip. The video clip is encoded in H.264 format. The alarm annotation image and alarm original image are saved. The time node, occurrence location and alarm content are recorded and sent to the platform for display and alarm.

[0120] Example 3:

[0121] Based on Example 2, this example further discloses the LAB-YOLO network structure, which includes:

[0122] Backbone Network:

[0123] The traditional C3k2 module is replaced by a multi-scale adaptive feature extraction module (MSAFE), as shown in the following formula:

[0124]

[0125] In the formula, β and γ are adaptive weight parameters that are dynamically adjusted through an attention weight allocation mechanism.

[0126] The SPPF module is optimized for laboratory scenarios using a spatial pyramid pooling module (Lab-SPPF), as shown in the following formula:

[0127] Lab-SPPF(F)

[0128] =Conv 1×1 ([F, MaxPool) 5×5 (F), MaxPool 9×9 (F),MaxPool 13×13 (F), AvgPool 5×5 (F)])

[0129] In the formula, [.] represents channel dimension concatenation; Conv 1×1 Used to adjust the number of channels.

[0130] The Neck network employs the C3k2 module, using 2×2 convolutional kernels combined with depthwise separable convolutions to reduce computational complexity and enhance multi-scale feature fusion capabilities. A bidirectional feature fusion enhancement module (BFEM) is designed, employing a cross-attention mechanism, as shown in the following formula:

[0131] BFEM(F high ,F low ) = Attention(F high ,F low )⊙F high +Attention(F low ,F high )⊙F low

[0132] In the formula, ⊙ represents element-wise multiplication; Attention(Q,K) is the cross-attention function.

[0133] The Head network employs a decoupled detection head (Lab-DetHead) designed for laboratory scenarios. Building upon traditional classification (ClassBranch) and regression (BBoxBranch) branches, it introduces a behavior semantic understanding branch (BehaviorBranch) and a confidence branch (ConfidenceBranch) to achieve decoupled modeling across multiple tasks, improving representation and generalization capabilities. Simultaneously, the classification and regression branches utilize dilated convolutions with varying dilation rates to enhance multi-scale perception, improving the loss function and training stability and detection accuracy. The behavior semantic understanding branch identifies complex behavioral patterns through temporal feature modeling. The decoupled detection head structure is as follows:

[0134] Lab-DetHead=[ClassBranch,BBoxBranch,BehaviorBranch,ConfidenceBranch]

[0135] In addition to optimizing the network structure and adjusting parameters, the LAB-YOLO model also adopts differentiated configurations for different detection targets: shoes, mutual inspection between two people, and the opening and closing status of equipment doors are trained and optimized using the Nano model, while gloves, falls, etc. are trained and optimized using the Small model, all of which are quantized and optimized using TensorRT.

[0136] Example 4:

[0137] Based on Example 2, this example further discloses the ResLab-Net network structure, including:

[0138] Basic Residual Block Optimization: The 3×3 convolutions in ResNet-34 are replaced with multi-scale residual fusion blocks (MSRBs) composed of 1×1, 3×3, and 5×5 convolutions. This multi-scale convolution enhances feature extraction capabilities and is suitable for classification tasks involving fine-grained objects such as footwear. The formula for the hybrid convolutional block is as follows:

[0139]

[0140] In the formula, x represents the residual block input feature. β and γ are the fusion weights; the output features are fused by adding them element by element.

[0141] Depth and width adaptation: ResNet-34 is used as the backbone network. The original 1000-dimensional classification layer is removed, and a 128-dimensional fully connected layer is introduced as the bottleneck feature layer after global average pooling (GAP). This reduces the feature dimension while retaining discriminative information, which is then fused with the detection results for anomaly identification. The bottleneck feature formula is as follows:

[0142] F bottleneck =FC 128 (GAP(x))

[0143] In the formula, GAP represents global average pooling; FC 128 It is a 128-dimensional fully connected layer that maps pooling results to low-dimensional bottleneck features.

[0144] Attention Mechanism Embedding: A Lab Scene Feature Enhancement Module (LSFEM) is added after each residual block. Based on the Coordinate Attention (CA) mechanism, it models the relationship between channels and space, improving the ability to perceive key features in complex backgrounds. Its structure is as follows:

[0145] LSFEM(x)=x⊙σ(Conv 1×1 (GAP(x)))+x⊙σ(Conv 1×1 (GMP(x)))

[0146] In the formula, GAP represents global average pooling; GMP represents global max pooling; and σ represents the sigmoid function.

[0147] The coordinate attention generation process described above is as follows:

[0148] a. Coordinate information embedding:

[0149]

[0150] In the formula, x c The input feature is the Cth channel; h and w are the height and width of the feature map; and These are aggregation features along the height and width directions, respectively.

[0151] b. Channel attention generation:

[0152] s=σ(FC r (ReLU(FC r ([z h ;z W ]))))

[0153] In the formula, [z h ;z W [ ] represents concatenation along the channel dimension; FC r σ is a dimension-reduced fully connected layer; σ is the sigmoid function.

[0154] c. Attention-weighted

[0155] x′ C =x C ·s C

[0156] In the formula, s C x′ represents the attention weight for the c-th channel. C The weighted output features.

[0157] As a preferred embodiment, frame_queue and result_queue are implemented based on Python's threading and queue modules. frame_queue manages frame images according to the first-in-first-out principle, and discards the oldest frame when the capacity is exceeded to prevent memory overflow.

[0158] As a preferred embodiment, a shared frame queue is used for frame synchronization in real-time video streaming. Processed frames are pushed to the RTMP server in FLV format via FFmpeg, and the frame queue can be configured with a maximum capacity. An innovative multi-algorithm result aggregation mechanism is used, employing a multimodal feature fusion algorithm (MMFA) to aggregate detection results using the following formula:

[0159]

[0160] In the formula, w i The dynamic weights of the i-th algorithm are calculated using a confidence-weighted strategy; R i This represents the detection result of the i-th algorithm.

[0161] At the same time, adaptive bitrate adjustment is used to ensure streaming quality:

[0162] BitRate adaptive =BitRate base ×(1+β×ConplexityFactor)

[0163] In the formula, BitRate base β is the base bitrate; β is the complexity adjustment coefficient; and ComplexityFactor is the video content complexity factor.

[0164] In a preferred embodiment, queue A is used to store consecutive frame images in the video stream. When no anomaly is detected, queue A is updated in real time; when an anomaly is detected, historical frames are extracted from queue A to generate video segments.

[0165] Example 5:

[0166] This embodiment discloses a laboratory intelligent monitoring system based on multi-model fusion, the system comprising:

[0167] The data acquisition module is used to acquire laboratory video in real time from multiple cameras, assign a unique frame_id to each frame and store it in the frame_queue, and perform preprocessing such as brightness adjustment and noise reduction on the image to ensure clear recognition later.

[0168] The algorithm processing module is used by the LAB-YOLO custom model to detect personnel's attire (such as gloves, cleanroom suits), behavior (such as mutual inspection between two people, falling), and equipment status (such as opening and closing doors), directly judging finite sample anomalies such as "present / absent" and "open / closed"; for infinite sample anomalies such as shoes, the LAB-YOLO custom model combines with a classification model trained by ResLab-Net to distinguish compliance (e.g., non-standard color slippers are considered anomalies).

[0169] The queue management module coordinates data flow between the two queues. The frame_queue stores the pre-processed frames, and the result_queue stores the algorithm detection results. A unique frame_id ensures a one-to-one correspondence between frames and results, avoiding confusion.

[0170] The real-time streaming module is used to draw the detection results (such as anomaly boxes, labels, and confidence scores) onto the original screen, and push the processed screen to the RTMP server as a video stream using FFmpeg. It supports real-time viewing on the laboratory management platform.

[0171] The alarm and storage module is used to perform multi-frame verification (e.g., confirming a true anomaly only after N anomalies occur within M frames) through the timing consistency verification module (TCV-Module) after detecting an anomaly, thus avoiding false alarms. It then saves the video clip of the anomaly, the original alarm image, and the alarm annotation image, recording the time point, location, and alarm content, and pushes the alarm to the laboratory management platform. The timing consistency verification formula is as follows:

[0172]

[0173] In the formula, A t The result of anomaly detection in frame t; M is the verification window size; w i For time series weights; I(A) i ) is the exception indicator function; when TCV(A t )>θ threshold When confirmed as a genuine anomaly, θ threshold This is a dynamic threshold.

[0174] Example 6:

[0175] Based on Example 5, this example discloses the details of the intelligent monitoring system for the laboratory, see [link to example]. Figure 1 Specifically, it includes the following modules:

[0176] I. Data Acquisition Module, see Figure 5 :

[0177] 1) Real-time acquisition of video stream and start an independent read_frame thread to assign a unique frame_id to each frame image and store it in frame_queue (e.g., the capacity is set to 500 frames, and the earliest frame is discarded when the capacity is exceeded).

[0178] 2) Call the preprocessing interface to complete adaptive histogram equalization, gamma correction and noise reduction filtering, and output the preprocessed frame image.

[0179] II. Algorithm processing module, see Figure 2 , Figure 3 , Figure 4 :

[0180] 1) Model deployment: Deploy the LAB-YOLO model and the ResLab-Net classification model on the server, and process multiple video streams in parallel using Python multithreading.

[0181] 2) Detection logic:

[0182] a. For samples with limited state targets (such as gloves, equipment doors), the LAB-YOLO model is used to directly detect and output anomaly judgment results.

[0183] b. For infinite state targets (such as shoes), the LAB-YOLO model first detects the target region, and then inputs it into the ResLab-Net classification model to determine the compliance of infinite samples such as shoes.

[0184] III. Queue Management Module, see Figure 5 :

[0185] 1) Queue design: Implemented using Python's threading and queue modules.

[0186] a.frame_queue: Stores preprocessed frame images, using a first-in-first-out strategy to ensure data sequence.

[0187] b.result_queue: Stores the detection results. The frame_id corresponds one-to-one with the frame image in the frame_queue to avoid data corruption.

[0188] 2) Synchronization mechanism: When the algorithm thread reads frame_queue, it will block and wait if the queue is empty to prevent empty read errors.

[0189] IV. Real-time streaming module, see Figure 6 :

[0190] 1) Result rendering: The detection box and label are superimposed on the frame image using the OpenCV library. The font size and line thickness can be freely set.

[0191] 2) Push Stream Configuration: Use FFmpeg commands to push the processed frames to the RTMP server. The push stream address format is rtmp: / / server_ip / live / lab_stream.

[0192] 3) Platform integration: The laboratory management platform displays data in real time via the RTSP protocol.

[0193] V. Alarm and Storage Module, see Figure 6 :

[0194] 1) False alarm filtering: False alarms caused by changes in lighting or temporary occlusion of the target are eliminated by multi-frame verification of the TCV-Module (such as abnormal N frames of M consecutive frames).

[0195] 2) Data storage: Abnormal video clips, original images, and labeled images are stored on the server; abnormal information (frame_id, time, alarm category content, type) is stored in a MySQL database, associated with camera ID and location information.

[0196] 3) Alarm methods: Alarms are provided through pop-up windows on the laboratory management platform to ensure timely response to abnormalities.

[0197] VI. System Architecture, see Figure 7 :

[0198] The system consists of a data layer, a model layer, an inference layer, a service layer, and an application layer, realizing a closed-loop process from video acquisition and anomaly detection to alarm push.

[0199] 1) Data layer: Video streams are captured by multiple cameras and stored in a frame queue after video frame preprocessing; labeled data is augmented and divided into training / validation / test sets to support model training.

[0200] 2) Model layer: The improved model is trained and validated, and its performance is improved through continuous iteration and optimization to achieve anomaly detection of personnel's clothing, behavior and equipment door opening and closing status; the model is deployed to GPU server through TensorRT quantization to ensure efficient inference.

[0201] 3) Inference Layer: The LAB-YOLO model outputs detection boxes with a confidence level ≥ 0.7, directly identifying anomalies in finite-state targets such as gloves and equipment doors. For targets with infinite states, such as shoes, the improved LAB-YOLO model first outputs detection boxes, which are then combined with the improved ResLab-Net classification model to complete fine classification and determine anomalies. All results are filtered by NMS, associated with a unique frame_id, and stored in the result queue. After aggregation, detection boxes and labels are drawn on the corresponding frames.

[0202] 4) Service layer: Stream forwarding is implemented based on NGINX, with Java backend management interface and task scheduling; real-time data is cached in Redis, exception records are stored in MySQL, and exception video clips are saved locally.

[0203] 5) Application layer: The laboratory management platform supports real-time video viewing, anomaly record query and algorithm parameter configuration, and triggers an alarm after detecting a real anomaly (multi-frame verification).

[0204] The method flows disclosed in the above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. The computer program is written to a storage medium and runs on a corresponding system, thereby enabling the system to automatically execute the method flows disclosed in the above embodiments, which will not be elaborated further here. When the computer instructions or computer program are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0205] In summary, the laboratory intelligent monitoring system based on multi-model fusion disclosed in this invention can acquire real-time video streams from multiple cameras. After preprocessing, it utilizes a customized model built based on LAB-YOLO to accurately detect the attire and behavior of laboratory personnel and the opening and closing status of equipment doors. Different anomaly detection methods are employed for different monitored objects, and false detections are eliminated through multi-frame verification. A new real-time video stream processing module aggregates and pushes the results of multiple algorithms, ultimately generating and pushing anomaly information and alarms in a timely manner. This system combines high precision and real-time performance, effectively handling complex scenarios such as changes in laboratory lighting and target obstruction, significantly improving the automation and intelligence level of laboratory safety management, and providing strong support for efficient and safe laboratory management.

[0206] As described above, although the invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the invention itself. Various changes in form and detail may be made without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A laboratory intelligent monitoring method based on multi-model fusion, characterized in that, Includes the following steps: Deploy multiple cameras to collect real-time video streams of the target space, assign a unique frame identifier to each frame image in the real-time video stream, and store the frame images containing the frame identifiers into a shared frame queue. A sample judgment model is constructed, in which the LAB-YOLO network structure is used as the finite sample judgment model and the ResLab-Net network structure is used as the infinite sample judgment model; wherein, the finite samples include gloves, mutual inspection behavior between two people, fall behavior, and equipment door opening status; the infinite samples include shoe color; The LAB-YOLO network structure includes a Backbone network, in which the C3k2 module is replaced by a multi-scale adaptive feature extraction module (MSAFE), as shown in the following expression: In the formula, The adaptive weight parameters are dynamically adjusted through an attention weight allocation mechanism. Lab-SPPF, a spatial pyramid pooling module, is constructed for laboratory scenarios. Its expression is as follows: In the formula, For channel-dimensional splicing; Used to adjust the number of channels; In the ResLab-Net network structure, the 3×3 convolutions in ResNet-34 are replaced with multi-scale residual fusion blocks (MSRBs) composed of multiple convolutions of different sizes. In the formula, Input features for residual blocks, The weight parameters are adaptive; the output features are fused by element-wise addition. The LAB-YOLO network structure also includes a Neck network and a Head network; In the Neck network, the C3k2 module uses 2×2 convolutional kernels combined with depthwise separable convolutions to reduce computational complexity; a bidirectional feature fusion enhancement module (BFEM) is designed, employing a cross-attention mechanism, as shown in the following formula: In the formula, Represents element-wise multiplication; For cross-attention functions; The Head network employs a decoupled detection head designed for laboratory scenarios, introducing behavioral semantic understanding and confidence branches on top of the traditional classification and regression branches. The structure of the decoupled detection head, Lab-DetHead, is as follows: In the formula, ClassBranch is the traditional classification; BBoxBranch is the regression branch; BehaviorBranch is the behavior semantic understanding branch; and ConfidenceBranch is the confidence branch. The ResLab-Net network structure uses ResNet-34 as the backbone network, removes the original 1000-dimensional classification layer, and introduces a 128-dimensional fully connected layer as a bottleneck feature layer after global average pooling (GAP) for anomaly identification by fusing with the detection results; the bottleneck feature formula is as follows: In the formula, For global average pooling; It is a 128-dimensional fully connected layer that maps the pooling results to low-dimensional bottleneck features; Frame images containing the frame identifier are obtained from the frame queue and imported into the sample judgment model. For the categories of gloves, mutual inspection between two people, fall, and equipment door opening / closing status, the LAB-YOLO network is used to directly determine whether the category status is abnormal. For the shoe category, the shoes detected by the LAB-YOLO network are input into the ResLab-Net network. The classification model trained by the improved ResLab-Net network is used to determine whether the shoes are the specified color and model of slippers in the laboratory. Slippers of different colors and models are judged as abnormal. The abnormality detection results of the above two methods contain frame identifiers and are stored in the result queue. When an anomaly is detected, it is verified from several consecutive frames. If the number of times the anomaly occurs exceeds a preset value, it is confirmed as a real anomaly. A video clip containing the result of the real anomaly is generated, and the alarm annotation image and alarm original image are saved. The time node, location of occurrence and alarm content are recorded.

2. The laboratory intelligent monitoring method based on multi-model fusion according to claim 1, characterized in that, Each multi-scale residual fusion block (MSRB) is followed by a laboratory scene feature enhancement module (LSFEM), which models the relationship between channels and space based on a coordinate attention mechanism. Its structure is as follows: In the formula, For global average pooling; This is for global max pooling; For the sigmoid function; The coordinate attention generation process is as follows: Coordinate information embedding: In the formula, For the input feature of the first One channel; The height and width of the feature map; and These are aggregation features along the height and width directions, respectively; Channel attention generation: In the formula, For channel-dimensional splicing; It is a dimension-reduced fully connected layer; It is the sigmoid function; Attention weighting: In the formula, Let be the attention weight for the c-th channel; The weighted output features.

3. The laboratory intelligent monitoring method based on multi-model fusion according to claim 1, characterized in that, The frame queue and result queue are implemented based on Python's thread module and queue module. The frame queue manages frame images according to the first-in-first-out principle, and discards the earliest frame when the capacity is exceeded to prevent memory overflow.

4. The laboratory intelligent monitoring method based on multi-model fusion according to claim 1, characterized in that, Real-time video streams use a shared frame queue for frame synchronization during streaming. Processed frames are pushed to an RTMP server in FLV format using FFmpeg. Detection results are aggregated using the Multimodal Feature Fusion (MMFA) algorithm, as shown in the following formula: In the formula, The dynamic weight of the m-th algorithm is calculated using a confidence-weighted strategy; This represents the detection result of the m-th algorithm; At the same time, adaptive bitrate adjustment is used to ensure streaming quality: In the formula, Base bitrate; This is a complexity adjustment factor; This represents the complexity factor of the video content.

5. The laboratory intelligent monitoring method based on multi-model fusion according to claim 1, characterized in that, When an anomaly is detected, timing consistency verification is performed on several consecutive frames. If the number of anomalies exceeds a threshold, the verification will be terminated. If so, it is confirmed as a genuine anomaly; The formula for verifying timing consistency is as follows: In the formula, For the first Anomaly detection results for the frame; To verify the window size; For time series weights; This is an exception indicator function; when At that time, it was confirmed as a genuine anomaly. For dynamic thresholds; Take K frames and subsequent K frames from queue A to generate video clips, save alarm annotation images and original alarm images, and record time nodes, occurrence locations and alarm content; Queue A is used to store consecutive frame images in the video stream. Queue A is updated in real time when no anomalies are detected. When an anomaly is detected, historical frames are extracted from queue A to generate video clips.

6. A laboratory intelligent monitoring system based on multi-model fusion, used to execute the laboratory intelligent monitoring method based on multi-model fusion as described in any one of claims 1 to 5, characterized in that, include: Data acquisition module; The data acquisition module acquires real-time video streams of the target space through multiple cameras, assigns a unique frame identifier to each frame in the real-time video stream, and stores the frame images containing the frame identifiers into a shared frame queue. The algorithm processing module is used to construct a sample judgment model, wherein the LAB-YOLO network structure is used as a finite sample judgment model and the ResLab-Net network structure is used as an infinite sample judgment model. The module obtains a frame image containing the frame identifier from the frame queue, imports the frame image into the sample judgment model, first uses the LAB-YOLO network structure to determine whether the category of the finite samples is abnormal, then uses the ResLab-Net network structure to detect the features of the infinite samples, and determines whether the features meet preset conditions. If they do not meet the conditions, the abnormal result is stored in the result queue along with the frame identifier. The queue management module is used to coordinate the data flow between the two queues; The frame queue is used to store pre-processed frames; The results queue is used to store the algorithm detection results; a unique frame identifier ensures a one-to-one correspondence between the image and the result. The real-time streaming module is used to draw the detection results on the original screen and push the processed screen to the RTMP server as a video stream using FFmpeg. It supports real-time viewing on the laboratory management platform. The detection results include anomaly boxes, labels, and confidence levels. The alarm and storage module is used to perform multi-frame verification through the timing consistency verification module after an anomaly is detected, save the abnormal video clips, alarm original images and alarm annotation images, record the time node, occurrence location and alarm content, and push the alarm to the laboratory management platform.

7. An electronic device, characterized in that, include: At least one processor; and a memory communicatively connected to at least one of the processors; The memory stores instructions that can be executed by the processor to implement the laboratory intelligent monitoring method based on multi-model fusion as described in any one of claims 1 to 5.

Images