A dynamic security monitoring method and system based on computer vision
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-03-05
- Publication Date
- 2026-07-10
Smart Images
Figure CN122368879A_ABST
Abstract
Description
Technical Field
[0001] This application relates to security monitoring, and more specifically, to a dynamic security monitoring method and system based on computer vision. Background Technology
[0002] With the rapid development of intelligent manufacturing and industrial automation, real-time safety monitoring systems based on computer vision are increasingly used in modern factories. These systems typically rely on deep neural network models deployed on edge servers to perform real-time analysis of multiple monitoring video streams at key workstations, enabling accurate detection and tracking of moving targets such as personnel and robotic arms, and providing early warnings of potential collision risks.
[0003] Current dynamic safety monitoring technologies still rely on detection-tracking, with the default processing unit being the complete video frame. In industrial scenarios where targets are sparse, this leads to a significant consumption of computational resources on static backgrounds and non-critical areas, resulting in inefficient use of the limited computing power of edge servers. This, in turn, causes a sharp increase in analysis latency. Furthermore, under the high-concurrency monitoring requirements of multiple workstations and multiple cameras, millisecond-level latency prevents the system from providing timely warnings before collisions occur. Traditional region of interest (ROI) filtering strategies typically process single-frame images in isolation, completely ignoring the spatiotemporal continuity and regularity of human and robotic arm movements in the video sequence. Consequently, they cannot achieve forward-looking and high-precision region filtering, making it difficult to fundamentally resolve the contradiction between resource consumption and real-time performance. Therefore, how to reduce the computational load when conducting safety collision monitoring in dangerous areas where human-machine interaction collisions occur has become a challenge for the industry. Summary of the Invention
[0004] This application provides a dynamic safety monitoring method and system based on computer vision, which can reduce the computational load when performing safety collision monitoring in dangerous areas where human-computer interaction may occur.
[0005] Firstly, this application provides a computer vision-based dynamic safety monitoring method for monitoring human-machine collaborative areas in a factory using a dynamic safety monitoring system. The dynamic safety monitoring system includes multiple monitoring cameras. The method includes the following steps: Acquire real-time video streams from surveillance cameras within the human-machine collaboration area of the factory; The frequency of workers entering and exiting in historical video streams collected by surveillance cameras is analyzed to generate a heat map of the frequency of workers entering and exiting within the working range of the robotic arm. Based on the density of human-machine spatial interaction in the heat map, multiple danger zones within the human-machine collaboration area are determined. In the current frame of the real-time video stream, capture multiple personnel targets that re-enter all danger zones, activate the protective tracking zone corresponding to each personnel target, and update the corresponding protective tracking zone in the current frame based on the predicted motion state of each personnel target. Based on the overlap status of all updated protective tracking areas and all dangerous areas, collision determination is performed on each personnel target and all robotic arms in the current frame. Then, based on the collision determination results, risk marking is performed on all updated protective tracking areas to obtain the risk monitoring area. Based on the target detection verification results in the risk monitoring area, a safety alarm is issued for the movement of each personnel target in the current frame within the human-machine collaboration area.
[0006] In some embodiments, analyzing the frequency of workers entering and exiting in historical video streams captured by surveillance cameras to generate a heatmap of worker entry and exit frequencies within the working range of the robotic arm specifically includes: Extract the sequence of images from the historical surveillance video stream; Full-frame target detection is performed on the sequence of images to obtain the position and category of each worker in each frame; Cross-frame correlation and trajectory tracking are performed on each worker in consecutive frames to record the spatiotemporal trajectory of each worker from appearance to disappearance; Based on all locations, categories, and spatiotemporal trajectories, the coordinates of each worker's first appearance and last disappearance in the video footage are calculated. A heatmap of the frequency of worker entry and exit within the working range of the robotic arm is generated by gridding all the position coordinates that first appear and last disappear.
[0007] In some embodiments, determining multiple hazardous areas within the human-machine collaboration area based on the density of human-machine spatial interaction in the heat map specifically includes: The video footage of the historical surveillance video stream is divided into multiple grid units; Determine the density of human-computer interaction in each grid cell of the heat map; Based on all the density, multiple hazardous areas within the human-machine collaborative area are selected from all the grid cells.
[0008] In some embodiments, activating the protective tracking zone corresponding to each personnel target specifically includes: Determine the initial detection bounding box for each personnel target; Expand all initial detection frames to obtain the protection and tracking area for each personnel target; Each protection and tracking zone is allocated independent computing resources and storage space, and the personnel targets corresponding to each protection and tracking zone are continuously tracked.
[0009] In some embodiments, updating the corresponding protective tracking area in the current frame based on the predicted motion state of each person target specifically includes: A Kalman filter is initialized for each personnel target; Predict the motion state of each person target in the current frame based on all Kalman filters, and the motion state includes at least the position and velocity vector of the person target in the image coordinate system; The protective tracking area for each personnel target is moved based on their position in all motion states; The protective tracking area for each personnel target is expanded based on the velocity vector in all motion states; Determine the redundancy detection rate between the updated protection tracking areas; The updated protection tracking areas are merged based on all redundant detection rates; In addition, if no human target is detected within the protection tracking area for multiple consecutive frames, it is determined that the human target has been lost and the resources of the corresponding protection tracking area are released.
[0010] In some embodiments, collision determination of each person target and all robotic arms in the current frame based on the overlap state of all updated protective tracking areas and all danger zones specifically includes: Determine the area coverage and overlap distance between all updated protection tracking areas and all hazardous areas; All area coverage and overlap distance are used as characteristics for determining the collision risk between each updated protection tracking area and all danger areas; Based on all the judgment features, determine the collision risk level of personnel targets and robotic arms within each updated protection tracking area in the current frame.
[0011] In some embodiments, risk marking is performed on all updated protection tracking areas based on collision determination results, resulting in risk monitoring areas specifically including: Based on all the collision risk levels in the collision determination results, a risk marker is assigned to each updated protective tracking area; Protective tracking areas that are spatially adjacent and have the same risk markers are grouped and aggregated to form risk monitoring areas.
[0012] Secondly, this application provides a computer vision-based dynamic security monitoring system, which includes multiple surveillance cameras and further includes: The acquisition module is used to acquire real-time video streams from surveillance cameras within the human-machine collaboration area of the factory. The processing module is used to analyze the frequency of workers entering and exiting in the historical video streams collected by the surveillance camera, generate a heat map of the frequency of workers entering and exiting within the working range of the robotic arm, and determine multiple danger zones within the human-machine collaboration area based on the density of human-machine spatial interaction in the heat map. The processing module is also used to capture multiple personnel targets that re-enter all dangerous areas in the current frame of the real-time video stream, activate the protective tracking area corresponding to each personnel target, and update the corresponding protective tracking area in the current frame based on the predicted motion state of each personnel target. The processing module is also used to perform collision determination on each personnel target and all robotic arms in the current frame based on the overlap status of all updated protective tracking areas and all dangerous areas, and then to perform risk marking on all updated protective tracking areas based on the collision determination results to obtain risk monitoring areas. The execution module is used to issue safety alarms for the movement of each person target in the current frame within the human-machine collaboration area based on the target detection verification results of the risk monitoring area.
[0013] Thirdly, this application provides a computer device, the computer device including a memory and a processor, the memory storing code, and the processor being configured to acquire the code and execute the above-described computer vision-based dynamic security monitoring method.
[0014] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned computer vision-based dynamic security monitoring method.
[0015] The technical solutions provided by the embodiments disclosed in this application have the following beneficial effects: The computer vision-based dynamic safety monitoring method and system provided in this application first acquires real-time video streams from monitoring cameras within the human-machine collaboration area of a factory; analyzes the frequency of worker entry and exit in the historical video streams collected by the monitoring cameras to generate a heat map of worker entry and exit frequency within the working range of the robotic arm; and determines multiple hazardous areas within the human-machine collaboration area based on the density of human-machine spatial interaction in the heat map. In the current frame of the real-time video stream, multiple personnel targets re-entering all hazardous areas are captured, and protective tracking areas corresponding to each personnel target are activated. Based on the predicted motion state of each personnel target, the corresponding protective tracking areas are updated in the current frame. Collision determination is performed between each personnel target and all robotic arms in the current frame based on the overlap state of all updated protective tracking areas and all hazardous areas. Risk marking is then applied to all updated protective tracking areas based on the collision determination results to obtain risk monitoring areas. Finally, safety alarms are issued for the movement of each personnel target within the human-machine collaboration area in the current frame based on the target detection verification results of the risk monitoring areas.
[0016] Therefore, in the computer vision-based dynamic safety monitoring method of this application, firstly, real-time video streams from monitoring cameras within the human-machine collaboration area of the factory are acquired; the frequency of worker entry and exit in the historical video streams collected by the monitoring cameras is analyzed to generate a heat map of worker entry and exit frequency within the working range of the robotic arm; based on the density of human-machine spatial interaction in the heat map, multiple danger zones within the human-machine collaboration area are determined, wherein danger zones refer to potential collision risk areas within the human-machine collaboration area of the factory where the frequency of spatial interaction between workers and the robotic arm is high; multiple personnel targets re-entering all danger zones are captured in the current frame of the real-time video stream, and the protective tracking zone corresponding to each personnel target is activated; based on the predicted motion state of each personnel target, the tracking zone is updated in the current frame. The corresponding protective tracking area is a candidate image region dynamically created for continuous tracking of each independently detected personnel target during real-time video analysis. The protective tracking area transforms the traditional global analysis of the entire video frame into a focused analysis of local areas where the target may appear, thus achieving precise allocation of computational resources. Secondly, based on the overlap status of all updated protective tracking areas and all dangerous areas, collision determination is performed on each personnel target and all robotic arms in the current frame. Then, based on the collision determination results, risk marking is applied to all updated protective tracking areas to obtain a risk monitoring area. Safety alarms are issued for the movement of each personnel target within the human-machine collaboration area in the current frame based on the target detection verification results of the risk monitoring area. This scheme can reduce the computational load when dynamically monitoring safety collisions of personnel targets while focusing on collision risk areas during human-machine interaction. Attached Figure Description
[0017] Figure 1 This is an exemplary flowchart of a computer vision-based dynamic security monitoring method according to some embodiments of this application; Figure 2 This is an exemplary flowchart illustrating the determination of hazardous areas according to some embodiments of this application; Figure 3 This is an exemplary flowchart illustrating the determination of a protective tracking area according to some embodiments of this application; Figure 4 This is a schematic diagram of the structure of a computer vision-based dynamic security monitoring system according to some embodiments of this application; Figure 5 This is a schematic diagram of the structure of a computer device implementing a computer vision-based dynamic security monitoring method according to some embodiments of this application. Detailed Implementation
[0018] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.
[0019] refer to Figure 1 The figure is an exemplary flowchart of a computer vision-based dynamic safety monitoring method according to some embodiments of this application. This method is used by a dynamic safety monitoring system to monitor human-machine collaborative areas in a factory. The dynamic safety monitoring system includes multiple monitoring cameras. The computer vision-based dynamic safety monitoring method mainly includes the following steps: In step 101, real-time video streams from monitoring cameras within the human-machine collaboration area of the factory are acquired.
[0020] In specific implementation, obtaining real-time video streams from surveillance cameras within the human-machine collaboration area of the factory can be achieved in the following way: a video stream management module is deployed on an edge server. This module establishes a network connection with fixed cameras deployed in the human-machine collaboration area of the factory production line via RTSP or RTMP protocols, and continuously pulls real-time video streams encoded in H.264 or H.265. Subsequently, the real-time video stream is sent to a frame buffer for decoding and preprocessing. The preprocessing includes color space conversion and resolution normalization, thereby obtaining the real-time video streams from the surveillance cameras within the human-machine collaboration area of the factory. Other methods can also be used in other embodiments, which are not limited here.
[0021] In step 102, the frequency of workers entering and exiting in the historical video streams collected by the monitoring camera is analyzed to generate a heat map of the frequency of workers entering and exiting within the working range of the robotic arm. Based on the density of human-machine spatial interaction in the heat map, multiple danger zones within the human-machine collaboration area are determined.
[0022] In some embodiments, analyzing the frequency of workers entering and exiting in historical video streams captured by surveillance cameras to generate a heatmap of worker entry and exit frequencies within the working range of the robotic arm can be achieved through the following steps: Extract the sequence of images from the historical surveillance video stream; Full-frame target detection is performed on the sequence of images to obtain the position and category of each worker in each frame; Cross-frame correlation and trajectory tracking are performed on each worker in consecutive frames to record the spatiotemporal trajectory of each worker from appearance to disappearance; Based on all locations, categories, and spatiotemporal trajectories, the coordinates of each worker's first appearance and last disappearance in the video footage are calculated. A heatmap of the frequency of worker entry and exit within the working range of the robotic arm is generated by gridding all the position coordinates that first appear and last disappear.
[0023] In specific implementation, the extraction of the sequence images from the historical surveillance video stream can be achieved in the following way: using the VideoCapture function of the OpenCV library to read the historical surveillance video file at a sampling rate of 5 frames per second, performing size normalization processing on each frame image to uniformly scale the resolution to a uniform pixel, performing color space conversion, and normalizing the pixel values to the range of [0,1] to form a sequence image dataset; other methods can also be used in other embodiments, which are not limited here.
[0024] In specific implementation, full-frame target detection of the sequence images to obtain the position and category of each worker in each frame can be achieved in the following way: A YOLOv8s model pre-trained on the COCO dataset is used. This model contains three core components: a backbone network, a neck network, and a detection head. The backbone network uses CSPDarknet53 and extracts multi-scale features through cross-order local partial connections and CSP modules. The neck network uses a path aggregation network, fusing feature maps of different resolutions through bidirectional feature pyramids from top to bottom and bottom to top. The detection head uses a decoupled structure to predict the coordinates of the detection box at the worker's location and the worker's category probability. For full-frame target detection in factory monitoring, the model is fine-tuned using a dataset of labeled images in VOC format. The XML contains two specific categories: worker and robotic arm. Training uses the SGD optimizer with an initial learning rate of 0.01, momentum of 0.937, weight decay of 0.0005, batch size of 16, and training epochs of 100. The input image is unified in pixels and normalized to the range [0,1]. Thus, the output of the model is a feature map at three scales. Each anchor point in the feature map predicts 4 coordinate offsets, 1 object confidence, and 2 class probabilities. All coordinate offsets are converted into the boundaries of the detection boxes of each worker in the image coordinate system, i.e., the positions of each worker. The object confidence is multiplied by each class probability to obtain the confidence score, and the class with the highest confidence score is used as the individual's class label. All detection results are stored in the detection result database along with the corresponding frame number and timestamp. Other methods can be used in other embodiments, which are not limited here.
[0025] In specific implementation, cross-frame association and trajectory tracking are performed on each worker in consecutive frames. The spatiotemporal trajectory of each worker from appearance to disappearance can be recorded in the following way: First, a new trajectory is initialized for the detection box of each worker in the first frame. For each subsequent frame, an association algorithm based on appearance features and spatial consistency is used: all worker category detection boxes in the current frame and the previous frame are extracted from the detection result database. A pre-trained ReID network is used, whose structure includes a backbone network and a feature embedding layer with an output dimension of 512, to extract the appearance feature vector of each detection box. Then, the cosine distance matrix between the feature vectors of all detection boxes in the current frame and the previous frame is calculated and weighted and fused with the spatial distance matrix constructed based on the distance between the center points of the detection boxes to generate the final association cost matrix. Finally, the Hungarian algorithm is used to perform optimal matching on the cost matrix. The successfully matched detection boxes are updated to the corresponding existing trajectories, and the unmatched detection boxes are initialized to new trajectories, thereby obtaining the spatiotemporal trajectory of each worker from appearance to disappearance. Other methods can also be used in other embodiments, which are not limited here.
[0026] In specific implementation, the coordinates of the first appearance and the last disappearance of each worker in the video frame can be calculated based on all positions, categories, and spatiotemporal trajectories. This can be achieved as follows: traverse all successfully tracked spatiotemporal trajectories, and take all trajectories with a duration of more than 30 frames as valid trajectories. For each valid trajectory, extract the coordinates of the first successfully associated detection box in the starting frame of the trajectory as the first appearance position, and extract the coordinates of the last successfully associated detection box in the ending frame of the trajectory as the last disappearance position. Store these key position coordinates together with the corresponding worker, trajectory ID, and timestamp in the trajectory key point database. Other methods can also be used in other embodiments, which are not limited here.
[0027] In specific implementation, the heat map of the frequency of worker entry and exit within the working range of the robotic arm, based on the gridded position coordinates of all first-appearance and last-disappearance locations, can be achieved in the following way: The video frame in each frame is spatially divided into a regular grid array: Based on computational accuracy and storage overhead, the frame is divided into M equal parts horizontally and N equal parts vertically, forming M×N rectangular grid units. For example, M=50, N=50. Next, all first-appearance and last-disappearance detection frames are traversed, and the intersection-union ratio (IUU) of each detection frame with each grid unit is calculated. These IUU values are then accumulated to the corresponding grid unit. Finally, the accumulated values of all grid units are Gaussian smoothed to generate a heat map reflecting the spatial distribution of worker entry and exit frequencies. Other methods can be used in other embodiments, which are not limited here.
[0028] It should be noted that the heatmap in this application is a probability distribution map representing the frequency of workers entering and leaving the human-computer interaction space. It utilizes the spatiotemporal continuity of video sequences, statistically analyzes the start and end positions of personnel target trajectories in historical data, and uses the intersection-over-union ratio (IoU), a standard metric in computer vision, to accumulate grid density. This quantifies the movement pattern of people entering or leaving the robotic arm's work area along fixed paths into a two-dimensional probability map with different heat values overlaid on the video screen, thereby achieving the purpose of efficient filtering by pre-analyzing target movement patterns. This heatmap ensures that the system's computing resources can be focused to the real high-frequency interaction area to the maximum extent, which can improve resource utilization efficiency and early warning accuracy. On the other hand, it enables the system to have adaptive capabilities. When the production line layout or workers' working habits change, the heatmap and danger areas can be updated by re-analyzing the data.
[0029] In some embodiments, reference Figure 2 As shown, this figure is an exemplary flowchart for determining hazardous areas in some embodiments of this application. In this embodiment, determining multiple hazardous areas within the human-machine collaboration area based on the density of human-machine spatial interaction in the heat map can be achieved through the following steps: First, in step 1021, the video frame of the historical monitoring video stream is divided into multiple grid units; Secondly, in step 1022, the density of human-machine spatial interaction in each grid cell of the heat map is determined; Finally, in step 1023, multiple hazardous areas within the human-machine collaborative area are selected from all grid cells based on their density.
[0030] In specific implementation, dividing the video frame of the historical surveillance video stream into multiple grid units can be achieved in the following way: Establish a two-dimensional coordinate system for the video frames in the historical surveillance video stream, with the upper left corner as the origin, the rightward direction from the upper left corner as the positive x-axis, and the downward direction from the upper left corner as the positive y-axis; divide the coordinate system evenly into M parts along the x-axis and N parts evenly along the y-axis, forming M×N rectangular grid units, where M=50. N=50; The density of human-machine spatial interaction in each grid cell of the heat map can be determined by the following method: reading the cumulative value of the cross-union ratio of each grid cell in the heat map, and taking the cumulative value of all cross-union ratios as the density of human-machine spatial interaction in the corresponding grid cell; Based on all the densities, multiple dangerous areas in the human-machine collaboration area can be screened from all grid cells by the following method: calculating the gradient values of each grid cell in the x-axis and y-axis directions using the Sobel operator, and when the gradient magnitude exceeds a preset threshold, determining that the corresponding grid cell is in a region of drastic density change, and then clustering all the regions determined to be drastic density change into multiple connected regions by using a region growing algorithm, calculating the minimum bounding rectangle of each connected region, filtering out noise regions with an area less than 1000 pixels, and finally retaining the rectangular region as the dangerous area in the human-machine collaboration area; Other methods can also be used in other embodiments, which are not limited here.
[0031] It should be noted that the hazardous area in this application refers to the potential collision risk area in the human-machine collaboration area of the factory, where the spatial interaction between workers and robotic arms is frequent. Specifically, by statistically learning the frequency of personnel entering and exiting in historical videos, the spatiotemporal continuity characteristics of video data are utilized to combine the intersection-union ratio calculation in computer vision with gridded analysis, thereby identifying hazardous interaction areas. This allows the system to focus on monitoring and calculating within these specific areas, significantly reducing the amount of video data that needs to be analyzed and lowering the computational load on the edge server. The hazardous area facilitates improved analysis efficiency and enables millisecond-level real-time safety early warning in resource-constrained edge environments.
[0032] In step 103, multiple personnel targets that re-enter all danger zones are captured in the current frame of the real-time video stream, and the protective tracking area corresponding to each personnel target is activated. The corresponding protective tracking area is updated in the current frame based on the predicted motion state of each personnel target.
[0033] In specific implementation, capturing multiple personnel targets re-entering all danger zones in the current frame of the real-time video stream can be achieved in the following way: obtaining a list of marked danger zone coordinates and a tracking list of globally active targets; for the image of the real-time video stream in the current frame, using a lightweight YOLOv8s model for forward inference only within all danger zones, where the input of the model is an image patch cropped from all danger zones, and the output is the detection box coordinates and confidence score for each detected worker category; when a target with a confidence score higher than 0.8 is detected, if the cumulative intersection-union ratio (IUU) of the target's detection box with the current danger zone is greater than 0.1, and the target's position range in the previous frame is not in the aforementioned tracking list, then the target is determined to be a newly entered personnel target, thereby creating a new tracking profile for the personnel target, including a globally unique ID, a fixed-length trajectory buffer, and recording its initial detection box position, confidence score, and category label; other methods can also be used in other embodiments, which are not limited here.
[0034] In some embodiments, reference Figure 3 As shown, this figure is an exemplary flowchart for determining the protection and tracking area in some embodiments of this application. In this embodiment, activating the protection and tracking area corresponding to each personnel target can be achieved by the following steps: First, in step 1031, the initial detection box for each personnel target is determined; Secondly, in step 1032, all initial detection frames are expanded to obtain the protection and tracking area for each personnel target; Finally, in step 1033, independent computing resources and storage space are allocated to each protection and tracking area, and the personnel targets corresponding to each protection and tracking area are continuously tracked.
[0035] In specific implementation, the initial detection bounding box for each personnel target can be determined as follows: For each personnel target detected within the danger zone, the output vector of the YOLOv8s model is parsed, and the normalized coordinates, height, and width representing the initial detection bounding box position of each personnel target within the corresponding danger zone are extracted. Then, through inverse coordinate transformation, the coordinates are converted back to absolute pixel coordinates in the original real-time video stream's video frame coordinate system, thereby forming the initial detection bounding box for each personnel target. The initial detection bounding box is a rectangle enclosed by four coordinate points, including the maximum and minimum values of the detection bounding box for each personnel target on the x-axis and the maximum and minimum values on the y-axis. Other implementation methods can also be used in other embodiments, which are not limited here.
[0036] In specific implementation, expanding all initial detection frames to obtain the protection and tracking area for each person target can be achieved in the following way: taking the center point of the initial detection frame of each person target as the center, multiplying the four coordinate points of the initial detection frame of each person target by a preset expansion ratio, thereby using the rectangular area enclosed by the four new coordinates as the protection and tracking area of the corresponding person target; wherein, the expansion ratio in this application is set to 1.2, and at the same time, in order to ensure that all expanded protection and tracking areas do not exceed the effective spatial range of the video frame in the digital coordinate system, the excess protection and tracking areas are truncated, and finally all expanded rectangular areas are used as the protection and tracking area of the corresponding person target; other methods can also be used in other embodiments, which are not limited here.
[0037] In specific implementation, allocating independent computing resources and storage space to each protection and tracking area, and continuously tracking the personnel targets corresponding to each protection and tracking area can be achieved in the following way: creating an independent data structure for each protection and tracking area in the server's memory. This structure includes: a Kalman filter instance, a circular buffer for storing the motion trajectory of the most recent 15 frames, a target identifier, and the current status flag of the area; at the same time, allocating a fixed tensor space for each protection and tracking area in the GPU memory to store the image data of that area, so as to facilitate subsequent fast target detection inference; other methods can also be used in other embodiments, which are not limited here.
[0038] It should be noted that the protective tracking region in this application is a candidate image region dynamically created for continuous tracking of each independently detected person target during real-time video analysis. The role of the protective tracking region is to transform the traditional global analysis of the entire frame of video into a focused analysis of the local area where the target may appear, thereby achieving precise allocation of computing resources. By pre-setting a reasonable extension range around the initial detection box, a visual anchor point that can cover the maximum possible displacement of the target between adjacent frames is formed. Thus, the target can be continuously captured in subsequent frames without processing the entire frame image, which helps to reduce the computational overhead and latency of a single inference. Furthermore, by analyzing the local area containing the moving target, the interference of complex backgrounds on the tracking algorithm is effectively avoided.
[0039] In some embodiments, updating the corresponding protective tracking area in the current frame based on the predicted motion state of each person target can be achieved using the following steps: A Kalman filter is initialized for each personnel target; Predict the motion state of each person target in the current frame based on all Kalman filters, and the motion state includes at least the position and velocity vector of the person target in the image coordinate system; The protective tracking area for each personnel target is moved based on their position in all motion states; The protective tracking area for each personnel target is expanded based on the velocity vector in all motion states; Determine the redundancy detection rate between the updated protection and tracking zones; The updated protection tracking areas are merged based on all redundant detection rates; In addition, if no human target is detected within the protection tracking area for multiple consecutive frames, it is determined that the human target has been lost and the resources of the corresponding protection tracking area are released.
[0040] In specific implementation, initializing a Kalman filter for each personnel target can be achieved in the following way: Initialize an 8-dimensional state vector Kalman filter for each personnel target, where the state vector includes the center coordinates, width, and height of the candidate region corresponding to each personnel target, as well as the change velocity of the personnel target on the x-axis and y-axis. The change velocity on the x-axis and y-axis is initialized to 0, and the state transition matrix is set to an 8×8 matrix, where the relationship between position and velocity conforms to the uniform motion model; the observation matrix H is set to a 4×8 matrix, that is, only the position and size can be observed, but the velocity cannot be directly observed; other methods can also be used in other embodiments, which are not limited here.
[0041] In specific implementation, the prediction of the motion state of each person target in the current frame based on all Kalman filters can be achieved in the following way: For each person target, the prediction method in its corresponding Kalman filter instance is called: Based on the state estimation and state transition matrix of the previous frame, the predicted state of the current frame is calculated; The state transition matrix is designed according to the physical law of the near-uniform motion of the person target's movement next to the human-machine interaction area of the assembly line, so that it can associate the position and velocity of the previous frame with the predicted position of the current frame. The output is the prior state vector of the current frame, including the predicted center position coordinates of the target, width, height, the changing velocity of the person target and the prior estimate covariance, thereby forming a velocity vector from the changing velocities, and then using each velocity vector and the center position coordinates as the motion state of the corresponding person target in the current frame; Other methods can also be used in other embodiments, which are not limited here.
[0042] In specific implementation, the protective tracking area of each personnel target can be moved based on the position in all motion states in the following way: extract the center position coordinates from the predicted state vector of each personnel target as the new center point coordinates, translate the center of the corresponding protective tracking area to the new center point coordinates, and keep the width and height of the protective tracking area of each personnel target unchanged; other methods can also be used in other embodiments, which are not limited here.
[0043] In specific implementation, expanding the protective tracking area of each personnel target based on the velocity vectors in all motion states can be achieved in the following way: extract the velocity components from all predicted state vectors and calculate the velocity. Then, based on the velocity magnitude and the width and height of the protective tracking area of each personnel target, substitute them into the calculation formula to calculate the new width and height of the protective tracking area of each personnel target in the motion direction. The calculation formula can be, for example: new width = original width * (1 + velocity sensitivity coefficient * velocity), where the new height is calculated using the same formula as above, and the velocity sensitivity coefficient can be set to 0.1 based on empirical values. Other methods can also be used in other embodiments, which are not limited here.
[0044] In specific implementation, the redundancy detection degree between each updated protection tracking area can be determined in the following way: calculate the cross-union ratio (CUI) between each pair of protection tracking areas, and use all CUIs as the redundancy detection degree between the corresponding two protection tracking areas. This process is repeated for all protection tracking area pairs to construct an N×N redundancy detection degree matrix, where N is the number of currently active protection tracking areas. The threshold of the redundancy detection degree matrix is set to 0.3. When the redundancy detection degree of two protection tracking areas is greater than the above threshold, it is determined that there is detection redundancy. Other methods can also be used in other embodiments, which are not limited here.
[0045] In specific implementation, merging the updated protection tracking regions based on all redundancy detection degrees can be achieved in the following way: combining the redundancy detection degree matrix with a region merging algorithm based on connected components: first, marking protection tracking regions with redundancy detection degrees greater than the threshold of 0.3 as connected, resulting in multiple connected region groups; for each connected region group, calculating the minimum bounding rectangle containing all regions within the group, and replacing all regions within the original group with this bounding rectangle, using the resulting new region as the updated protection tracking region; simultaneously, creating a composite identity identifier for each new region, which records the IDs of personnel targets in all merged original regions; other methods can also be used in other embodiments, which are not limited here.
[0046] It should be noted that a loss counter is maintained for each updated protection tracking area. When target detection is performed in any protection tracking area but no personnel target with a confidence level higher than 0.5 is found, the counter is incremented by 1. If the counter value exceeds the preset maximum tolerance frame number, it is determined that the personnel target in the protection tracking area has been lost. Subsequently, the system reclaims all resources occupied by the area, including: releasing the tensor space in the GPU memory, destroying the Kalman filter, clearing the trajectory buffer, and marking the target identifier as reusable.
[0047] In step 104, collision determination is performed on each personnel target and all robotic arms in the current frame based on the overlap status of all updated protective tracking areas and all dangerous areas. Then, risk marking is performed on all updated protective tracking areas based on the collision determination results to obtain the risk monitoring area.
[0048] In some embodiments, collision determination of each person target and all robotic arms in the current frame based on the overlap status of all updated protective tracking areas and all hazardous areas can be achieved using the following steps: Determine the area coverage and overlap distance between all updated protection tracking areas and all hazardous areas; All area coverage and overlap distance are used as characteristics for determining the collision risk between each updated protection tracking area and all danger areas; Based on all the judgment features, determine the collision risk level of personnel targets and robotic arms within each updated protection tracking area in the current frame.
[0049] In specific implementation, determining the area coverage and overlap distance between all updated protective tracking areas and all dangerous areas can be achieved in the following way: For each updated protective tracking area, calculate its intersection-union ratio (IUGR) with each dangerous area, and simultaneously calculate the Euclidean distance between the center points of the two areas. Use all IUGRs as the area coverage between the corresponding protective tracking area and dangerous area, and use all Euclidean distances as the overlap distance between the corresponding protective tracking area and dangerous area. Form an N×M matrix with all IUGRs and all Euclidean distances, where N is the number of protective tracking areas and M is the number of dangerous areas. The area coverage is a parameter describing the degree to which the protective tracking area covers the dangerous area, and the overlap distance is a parameter describing the distance when the protective tracking area and the dangerous area overlap. Other methods can also be used in other embodiments, which are not limited here.
[0050] In specific implementation, the collision risk level of personnel targets and robotic arms within each updated protective tracking area in the current frame can be determined based on all the judgment features in the following way: Select a protective tracking area as the selected protective tracking area. When the largest area coverage in the judgment features of the selected protective tracking area is >0.5, the selected protective tracking area is marked as high risk; when 0.3 < the largest area coverage in the judgment features of the selected protective tracking area is ≤0.5 and the overlap distance is <100 pixels, it is marked as medium risk; when 0 < the largest area coverage in the judgment features of the selected protective tracking area is ≤0.3, it is marked as low risk. Other methods can also be used in other embodiments, which are not limited here.
[0051] It should be noted that in a fixed factory human-machine collaboration scenario, the range of motion of the robotic arm is pre-programmed and relatively fixed, which constitutes a relatively static danger zone; while the movement of the personnel is dynamically and continuously located through the protective tracking area. The essence of a collision is the intersection of the personnel and the robotic arm in physical space, which is then transformed into a two-dimensional projection in computer vision. This three-dimensional spatial positional intersection is directly reflected as the overlap of the corresponding two-dimensional bounding boxes (i.e., the protective tracking area and the danger zone) in the image coordinate system, namely the intersection-union ratio and the center distance. Therefore, the overlap state of the area becomes a reliable indicator for predicting and judging the collision risk. Secondly, the complex collision safety problem is transformed into a geometric calculation of the overlap relationship of a few key areas, making full use of the spatiotemporal continuity of the video sequence. That is, the protective tracking area is predicted and updated through Kalman filtering, which already includes the movement trend of the personnel target, so that the system can not only judge the current instantaneous overlap, but also predict the upcoming overlap based on the motion state.
[0052] In some embodiments, risk marking is performed on all updated protection tracking areas based on collision determination results to obtain risk monitoring areas, which can be achieved through the following steps: Based on all the collision risk levels in the collision determination results, a risk marker is assigned to each updated protective tracking area; Protective tracking areas that are spatially adjacent and have the same risk markers are grouped and aggregated to form risk monitoring areas.
[0053] In specific implementation, assigning a risk marker to each updated protective tracking area based on all collision risk levels in the collision determination results can be achieved in the following way: establishing a color-coded risk marker system: marking high-risk protective tracking areas as red, medium-risk protective tracking areas as yellow, and low-risk protective tracking areas as green; and attaching a priority value to each marker, which is set in this application as: red=3, yellow=2, green=1; other methods can also be used in other embodiments, which are not limited here.
[0054] In specific implementation, grouping and aggregating spatially adjacent protection tracking areas with the same risk label to form risk monitoring areas can be achieved in the following way: Based on the spatial relationship between protection tracking areas, a distance-based clustering algorithm (such as the DBSCAN algorithm) is used. The center coordinates and risk levels of all protection tracking areas are combined into a feature vector and input into the distance-based clustering algorithm. The neighborhood radius parameter is set to 50 pixels to ensure that spatially adjacent protection tracking areas can be merged. The minimum number of core point samples is set to 1, meaning that a single area can initiate aggregation. Then, the Euclidean distance between protection tracking areas is calculated using the algorithm, and protection tracking areas with an Euclidean distance less than the neighborhood radius parameter and the same risk label are clustered into the same connected cluster. Finally, multiple cluster lists are output, and each cluster contains one or more protection tracking areas. Subsequently, the minimum bounding rectangle of all protection tracking areas of each cluster is calculated, and this rectangle is the risk monitoring area. Other methods can also be used in other embodiments, which are not limited here.
[0055] It should be noted that the risk monitoring area in this application refers to a spatially composed area to be monitored, which is used to spatially integrate multiple local threats with the same risk level, forming a unified and large-scale monitoring and verification unit. This allows for risk monitoring of human-machine collisions in a single area, effectively reducing the contradiction between fragmented alarms and limited edge computing resources. Secondly, the above approach combines the high-precision detection inference that originally required multiple executions for small areas into a one-time batch inference for larger areas, thereby significantly improving the parallel processing efficiency and resource utilization of computing hardware such as GPUs. This enables a more refined resource scheduling strategy based on the unified risk level of the area, ensuring that limited edge computing power can be prioritized and fully allocated to the highest-risk areas. Finally, the risk monitoring area may include one or more, depending on the distribution of risk targets in the current frame.
[0056] In step 105, a safety alarm is issued for the movement of each person target in the current frame within the human-machine collaboration area based on the target detection verification results of the risk monitoring area.
[0057] In some embodiments, the following steps can be used to issue safety alerts for the movement of various personnel targets within the human-machine collaboration area in the current frame based on the target detection verification results of the risk monitoring area: Real-time collection of computing power status of each edge server; The risk monitoring area will be dynamically scheduled to a designated edge server based on the computing power status. Parallel inference is performed on the received risk monitoring areas on the edge server; Based on the reasoning results, an emergency braking warning is issued to personnel targets within the risk area described in the current frame.
[0058] In specific implementation, dynamically scheduling the risk monitoring area to designated edge servers based on the computing power status can be achieved in the following way: First, monitor the GPU utilization, video memory usage, GPU temperature, and system memory usage of each edge server in real time using NVIDIA's DCGM toolkit. Then, obtain CPU load and network bandwidth using Linux system commands to calculate the real-time computing power coefficient for each server. The calculation formula for the computing power coefficient can be, for example: Computing Power Coefficient = (1 - GPU Utilization) × Available Video Memory Ratio × (1 - CPU Load) × Temperature attenuation factor; secondly, based on computing power coefficient, all risk monitoring areas are prioritized by color, for example: red > yellow > green, and a weighted round-robin scheduling algorithm is used to allocate high-priority areas to the server with the highest computing power coefficient to ensure that emergency tasks receive the best computing resources; at the same time, considering data transmission cost, servers with low network latency are prioritized for scheduling; parallel inference of the received risk monitoring areas on the edge server can be implemented in the following way, that is: on the scheduled server, the YOLOv8x model is used to perform single forward propagation inference on the image blocks of the received risk monitoring areas. This model extracts features through the backbone network, then performs multi-scale feature fusion through the neck network, and finally outputs the bounding box coordinates, class confidence and instance segmentation mask of each person target in the risk monitoring area in the detection head part; wherein, the confidence threshold of the class confidence is set to 0.9 to filter the detection results; other methods can also be used in other embodiments, which are not limited here.
[0059] In specific implementation, the emergency braking alarm for personnel targets within the risk area in the current frame based on the inference result can be implemented in the following way: when a personnel target with a confidence level higher than 0.9 is detected in the red risk monitoring area, an emergency braking command is immediately sent to the robotic arm control system via the MQTT protocol. The emergency braking command includes the target location, risk level, and timestamp. At the same time, a red flashing alarm and an audible prompt are triggered on the monitoring interface, and the alarm event is recorded in the safety log database. Other methods can also be used in other embodiments, which are not limited here.
[0060] Furthermore, in another aspect of this application, in some embodiments, this application provides a dynamic security monitoring system based on computer vision, referencing... Figure 4The figure is a schematic diagram of the structure of a computer vision-based dynamic security monitoring system according to some embodiments of this application. The dynamic security monitoring system includes multiple monitoring cameras. In addition, the computer vision-based dynamic security monitoring system also includes: an acquisition module 401, a processing module 402, and an execution module 403, which are described below: The acquisition module 401 in this application is mainly used to acquire the real-time video stream of the monitoring camera in the human-machine collaboration area of the factory; Processing module 402 in this application is used to analyze the frequency of workers entering and exiting in the historical video stream collected by the monitoring camera, generate a heat map of the frequency of workers entering and exiting within the working range of the robotic arm, and determine multiple dangerous areas within the human-machine collaboration area based on the density of human-machine spatial interaction in the heat map. It should be noted that the processing module 402 in this application is also used to capture multiple personnel targets that re-enter all dangerous areas in the current frame of the real-time video stream, activate the protective tracking area corresponding to each personnel target, and update the corresponding protective tracking area in the current frame based on the predicted motion state of each personnel target. Additionally, it should be noted that the processing module 402 in this application is also used to perform collision determination on each personnel target and all robotic arms in the current frame based on the overlap status of all updated protective tracking areas and all dangerous areas, and then to perform risk marking on all updated protective tracking areas based on the collision determination results to obtain risk monitoring areas. The execution module 403 in this application is mainly used to issue a safety alarm for the movement of each person target in the current frame within the human-machine collaboration area based on the target detection verification results of the risk monitoring area.
[0061] In addition, this application also provides a computer device, the computer device including a memory and a processor, the memory storing code, and the processor being configured to acquire the code and execute the above-described computer vision-based dynamic security monitoring method.
[0062] In some embodiments, reference Figure 5 The figure is a schematic diagram of the structure of a computer device implementing a computer vision-based dynamic security monitoring method according to some embodiments of this application. The computer vision-based dynamic security monitoring method in the above embodiments can... Figure 5 The computer device shown is used to implement this, and the computer device includes at least one processor 501, a communication bus 502, a memory 503, and at least one communication interface 504.
[0063] Processor 501 can be a general-purpose central processing unit (CPU) or an application-specific integrated circuit (ASIC).
[0064] The communication bus 502 can be used to transmit information between the aforementioned components.
[0065] Memory 503 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CDROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile optical discs, Blu-ray discs, etc.), magnetic disks or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory 503 may exist independently and be connected to processor 501 via communication bus 502. Memory 503 may also be integrated with processor 501.
[0066] The memory 503 stores program code for executing the scheme of this application, and its execution is controlled by the processor 501. The processor 501 executes the program code stored in the memory 503. The program code may include one or more software modules. The method used in the above embodiments can be implemented by the processor 501 and one or more software modules in the program code in the memory 503.
[0067] Communication interface 504 uses any transceiver-like device to communicate with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.
[0068] In a specific implementation, as one example, a computer device may include multiple processors, each of which may be a single-core (single CPU) processor or a multi-core (multi CPU) processor. Here, a processor may refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).
[0069] The aforementioned computer device can be a general-purpose computer device or a special-purpose computer device. In specific implementations, the computer device can be a desktop computer, a portable computer, a network server, a handheld digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. This application does not limit the type of computer device.
[0070] In addition, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described computer vision-based dynamic security monitoring method.
[0071] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.
[0072] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.
Claims
1. A computer vision-based dynamic safety monitoring method for monitoring human-machine collaborative areas in a factory using a dynamic safety monitoring system, wherein the dynamic safety monitoring system includes multiple monitoring cameras, characterized in that... The method includes the following steps: Acquire real-time video streams from surveillance cameras within the human-machine collaboration area of the factory; The frequency of workers entering and exiting in historical video streams collected by surveillance cameras is analyzed to generate a heat map of the frequency of workers entering and exiting within the working range of the robotic arm. Based on the density of human-machine spatial interaction in the heat map, multiple danger zones within the human-machine collaboration area are determined. In the current frame of the real-time video stream, capture multiple personnel targets that re-enter all danger zones, activate the protective tracking zone corresponding to each personnel target, and update the corresponding protective tracking zone in the current frame based on the predicted motion state of each personnel target. Based on the overlap status of all updated protective tracking areas and all dangerous areas, collision determination is performed on each personnel target and all robotic arms in the current frame. Then, based on the collision determination results, risk marking is performed on all updated protective tracking areas to obtain the risk monitoring area. Based on the target detection verification results in the risk monitoring area, a safety alarm is issued for the movement of each personnel target in the current frame within the human-machine collaboration area.
2. The method as described in claim 1, characterized in that, Analyzing the frequency of worker entry and exit in historical video streams captured by surveillance cameras to generate a heatmap of worker entry and exit frequency within the robotic arm's working range specifically includes: Extract the sequence of images from the historical surveillance video stream; Full-frame target detection is performed on the sequence of images to obtain the position and category of each worker in each frame; Cross-frame correlation and trajectory tracking are performed on each worker in consecutive frames to record the spatiotemporal trajectory of each worker from appearance to disappearance; Based on all locations, categories, and spatiotemporal trajectories, the coordinates of each worker's first appearance and last disappearance in the video footage are calculated. A heatmap of the frequency of worker entry and exit within the working range of the robotic arm is generated by gridding all the position coordinates that first appear and last disappear.
3. The method as described in claim 1, characterized in that, Based on the density of human-machine spatial interaction in the heat map, multiple hazardous areas within the human-machine collaboration area are identified, specifically including: The video footage of the historical surveillance video stream is divided into multiple grid units; Determine the density of human-computer interaction in each grid cell of the heat map; Based on all the density, multiple hazardous areas within the human-machine collaborative area are selected from all the grid cells.
4. The method as described in claim 1, characterized in that, The specific protective tracking zones activated for each personnel target include: Determine the initial detection bounding box for each personnel target; Expand all initial detection frames to obtain the protection and tracking area for each personnel target; Each protection and tracking zone is allocated independent computing resources and storage space, and the personnel targets corresponding to each protection and tracking zone are continuously tracked.
5. The method as described in claim 1, characterized in that, The corresponding protection tracking area is updated in the current frame based on the predicted motion state of each person target, specifically including: A Kalman filter is initialized for each personnel target; Predict the motion state of each person target in the current frame based on all Kalman filters, and the motion state includes at least the position and velocity vector of the person target in the image coordinate system; The protective tracking area for each personnel target is moved based on their position in all motion states; The protective tracking area for each personnel target is expanded based on the velocity vector in all motion states; Determine the redundancy detection rate between the updated protection tracking areas; The updated protection tracking areas are merged based on all redundant detection rates; In addition, if no human target is detected within the protection tracking area for multiple consecutive frames, it is determined that the human target has been lost and the resources of the corresponding protection tracking area are released.
6. The method as described in claim 1, characterized in that, Based on the overlap status of all updated protective tracking areas and all danger zones, collision detection is performed on each personnel target and all robotic arms in the current frame, specifically including: Determine the area coverage and overlap distance between all updated protection tracking areas and all hazardous areas; All area coverage and overlap distance are used as characteristics for determining the collision risk between each updated protection tracking area and all danger areas; Based on all the judgment features, determine the collision risk level of personnel targets and robotic arms within each updated protection tracking area in the current frame.
7. The method as described in claim 1, characterized in that, Based on the collision determination results, all updated protection tracking areas are marked with risks, resulting in risk monitoring areas including: Based on all the collision risk levels in the collision determination results, a risk marker is assigned to each updated protective tracking area; Protective tracking areas that are spatially adjacent and have the same risk markers are grouped and aggregated to form risk monitoring areas.
8. A computer vision-based dynamic security monitoring system, the dynamic security monitoring system comprising multiple surveillance cameras, characterized in that, The dynamic security monitoring system also includes: The acquisition module is used to acquire real-time video streams from surveillance cameras within the human-machine collaboration area of the factory. The processing module is used to analyze the frequency of workers entering and exiting in the historical video streams collected by the surveillance camera, generate a heat map of the frequency of workers entering and exiting within the working range of the robotic arm, and determine multiple danger zones within the human-machine collaboration area based on the density of human-machine spatial interaction in the heat map. The processing module is also used to capture multiple personnel targets that re-enter all dangerous areas in the current frame of the real-time video stream, activate the protective tracking area corresponding to each personnel target, and update the corresponding protective tracking area in the current frame based on the predicted motion state of each personnel target. The processing module is also used to perform collision determination on each personnel target and all robotic arms in the current frame based on the overlap status of all updated protective tracking areas and all dangerous areas, and then to perform risk marking on all updated protective tracking areas based on the collision determination results to obtain risk monitoring areas. The execution module is used to issue safety alarms for the movement of each person target in the current frame within the human-machine collaboration area based on the target detection verification results of the risk monitoring area.
9. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing code, and the processor being configured to retrieve the code and execute the computer vision-based dynamic security monitoring method as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the computer vision-based dynamic security monitoring method as described in any one of claims 1 to 7.