A method and system for dynamic detection of intrusion in a warning area

Abstract

The present invention belongs to the field of computer vision, and discloses a method and system for dynamic detection of intrusion in a warning zone. Based on the warning zone annotation results, the warning zone intrusion judgment and intrusion prediction are realized by performing target detection and target tracking on the target; the moving target is extracted based on the improved ViBe algorithm; the YOLOv7 target detection result is differentiated from the improved ViBe algorithm extraction result to obtain the real-time target detection result; the real-time target detection result is tracked in real time according to the improved ECO‑HC algorithm, the trajectory of the target is recorded as training data, the LSTM trajectory prediction model is trained to predict the future trajectory of the target, and the intrusion prediction is performed based on the future trajectory of the target and the warning zone annotation information. The moving target detection effect of the improved ViBe algorithm is significantly improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method and system for dynamic detection of intrusion into a warning zone. Background Art

[0002] The development of manufacturing industry is inseparable from production. The production workshop is the main battlefield of the manufacturing industry, and the safe production of the workshop is the foundation of all work. With the promotion of digital and intelligent production, the safety management of the production workshop has also transformed from manual supervision to intelligent management based on video surveillance. In the intelligent monitoring system, warning zone intrusion detection is one of the core functions. In the daily production activities of the production workshop, there are incidents where the safety of life and property is threatened due to illegal entry into the warning area. The traditional management model uses manual patrols for safety management, but this method is difficult to detect illegal intrusion incidents in time, so there are a lot of omissions, which makes it difficult to achieve strict management. With the development of monitoring technology, most production workshops are managed by installing monitoring equipment, but a large number of monitoring videos cannot be managed in real time and effectively by manual viewing. Therefore, it is very important to study intelligent safety management technology based on monitoring videos. At present, based on the application needs of various fields, some intelligent monitoring equipment can not only provide basic functions such as real-time monitoring, video storage and playback, but also perform license plate recognition, tripwire intrusion, intrusion detection, etc.

[0003] Warning zone intrusion detection refers to the detection and alarm of target intrusion in a specific area through various devices. Warning zone intrusion detection has a wide range of application scenarios, and some research results have been achieved. For example, "Yu Xiaomang, Luo Guangming, Zhu Zhenmin, et al. Multi-target recognition of perimeter security intrusion signals by distributed optical fiber sensors [J]. Optoelectronic Engineering, 2014, 41(01): 36-41." proposed to use distributed optical fiber sensors for perimeter intrusion security detection. Although this method can detect abnormal conditions in the surrounding area, the false positive rate for warning zone intrusion detection is high, and the actual application cost is high and is easily affected by environmental changes. "Ling Ren, Lian Qixing, He Yuehai, et al. Research on foreign body detection between subway platform doors and trains based on laser scanning [J]. Locomotive Electric Drive, 2020, (03): 60-62." The laser beam device was used for foreign body intrusion detection between subway platforms and trains. This method has a long detection distance and high sensitivity, but the installation conditions are relatively harsh and not suitable for complex scenes. In addition, due to the size of the laser sensor, the false positive rate of foreign body intrusion detection in actual applications is high. With the popularization of monitoring equipment and the development of computer vision technology, video-based warning zone intrusion detection research has gradually become mainstream.

[0004] At present, most intelligent monitoring devices are based on the detection of moving targets in the video to achieve warning zone intrusion detection, but traditional moving target detection methods (optical flow method, frame difference method, mixed Gaussian modeling, etc.) have certain limitations in the application of intrusion detection. With the development and progress of target detection technology, the current target detection method based on deep learning can overcome the shortcomings of slow reasoning speed while ensuring high-precision detection, and well balance the accuracy and speed of target detection, bringing new directions for the research of warning zone intrusion detection technology. In addition, in the target tracking research, the correlation filter target tracking algorithm breaks through the limitation of slow detection speed of traditional target tracking algorithms, and improves the accuracy and speed of target tracking to a new level. Due to the great improvement of target tracking speed, it is possible to introduce target tracking technology into the research of dynamic detection of warning zone intrusion.

[0005] At present, video-based warning zone intrusion technology generally has problems such as high application cost, high false detection rate and low accuracy, and the intrusion detection function often cannot meet the real-time requirements in actual applications. Single target detection methods such as motion target detection or deep learning target detection have their own limitations. Due to the high mobility of staff in the production workshop, frequent occlusion of targets due to the large number of production equipment, and the low clarity of some monitoring equipment, there are still technical problems that need to be solved in the actual application of warning zone intrusion detection. The current warning zone intrusion detection can only alarm after the intrusion occurs, but for the actual scenes with high danger or special scenes, it is necessary not only to make intrusion judgments but also to prevent intrusions before intrusions occur. The intrusion detection system needs more functions to meet the needs of various scenarios. Summary of the invention

[0006] The research on video-based warning zone intrusion dynamic detection technology for production workshops can improve the efficiency of workshop safety management and reduce the frequency of safety accidents. The purpose of this invention is to address the problems existing in the actual application of the current warning zone intrusion detection technology, and propose a warning zone intrusion determination method combining target detection and warning zone marking, and a warning zone intrusion prediction method combining target tracking and trajectory prediction.

[0007] The technical solution of the present invention is as follows: A method for dynamic detection of intrusion in a warning zone, based on the warning zone marking result, realizes intrusion determination and intrusion prediction in the warning zone by performing target detection and target tracking on the target, comprising the following steps:

[0008] Step 1. Warning zone marking for different monitoring scenarios: Different methods are used to mark the warning zone range according to the monitoring scenarios: if the monitoring scenario is a production workshop, the warning zone is automatically marked based on the landmark line extraction; if the monitoring scenario is the perspective of monitoring equipment distributed in different places, the warning zone is manually marked after scene splicing; if there are safety cones in the monitoring scene, the corner points of the warning zone are identified according to the safety cones and the warning zone is marked;

[0009] Step 2: Storage of warning zone annotation information: the warning zone annotation information includes attribute information and file information; the attribute information includes current scene information, warning zone number, warning zone intersection coordinates, and warning zone remark information; the file information includes a screenshot of the warning zone annotation result, image size, and image name;

[0010] Step 3: Target detection in real-time video: perform basic recognition of staff, safety cones, and transport carts using the YOLOv7 target detection algorithm; extract moving targets based on the improved ViBe algorithm; perform differential analysis between the YOLOv7 target detection result and the improved ViBe algorithm extraction result to obtain real-time target detection results;

[0011] Step 4: Warning zone intrusion determination: Based on the real-time target detection results of step 3 and the warning zone marking information loaded in step 2, determine the relative position relationship between the target and the warning zone, determine the target category authority, determine the time when the target enters the warning zone, and report the intrusion determination result;

[0012] Step 5: Warning zone intrusion prediction: According to the improved ECO-HC algorithm, the real-time target detection results of step 3 are tracked in real time, the trajectory of the target is recorded as training data, and the LSTM trajectory prediction model is trained; the future trajectory of the target is predicted by the LSTM trajectory prediction model; and intrusion prediction is performed based on the future trajectory of the target and the warning zone annotation information.

[0013] The improved ViBe algorithm includes ghost phenomenon elimination based on histogram similarity, foreground shadow elimination based on YCbCr space color features and texture features, and target integrity improvement based on merging adjacent target bounding rectangles, as follows:

[0014] (1) Ghost elimination based on histogram similarity;

[0015] 1.1) Use the ViBe algorithm to perform background modeling on the initial frame image and establish a background model;

[0016] 1.2) Detect moving targets in subsequent frames, and perform corrosion processing on the detection results to remove the noise points of the panoramic target adhesion, obtain the foreground target areas of the current frame, and obtain the foreground block F i , i=1,2,…,N, i is the sequence number, N is the total number of foreground target areas;

[0017] 1.3) Calculate the foreground block F i The histogram FH of the current frame i ;

[0018] 1.4) Take the minimum enclosing rectangle R of each foreground target area i , and its corresponding neighborhood background area B i The calculation formula is B i =R i -F i , get B i The histogram BH of the current frame i ;

[0019] 1.5) By calculating FH i and BH i The ghost area is further determined by the similarity; after the ghost area is identified, the pixel value of the ghost area is assigned to 0 when the background model of the ViBe algorithm is updated, and the background model is updated; when the ghost area is not identified, the foreground shadow is eliminated based on the color features and texture features of the YCbCr space;

[0020] (2) Foreground shadow removal based on YCbCr spatial color features and texture features;

[0021] First, the image without ghost area identified in step 1.5) is converted from RGB color space to YCbCr color space. The specific conversion formula is as follows:

[0022] Y=0.299·R+0.587·G+0.114·B

[0023] Cr=(RY)·0.713+delta

[0024] Cb=(BY)·0.564+delta

[0025]

[0026] Y is the brightness component of the color, R is the red component of the color, G is the green component of the color, B is the blue component of the color, Cb and Cr are the concentration offset component of blue and the concentration offset component of red respectively; in the YCbCr color space, the brightness and chromaticity information of the pixel points are used to screen out the candidate area of ​​the foreground shadow; the specific steps are as follows:

[0027] 2.1) According to the neighborhood concept of the image, the corresponding Y, Cb, and Cr components are calculated based on the R, G, and B components of the pixel point P(x, y) in the foreground target area and the above conversion formula, and the 8-neighborhood weighted average calculation method is used when calculating the Y, Cb, and Cr components of the point;

[0028] 2.2) Based on the fact that the foreground shadow pixels show the characteristic of reduced brightness and unchanged chrominance in the YCbCr color space, the candidate foreground shadow area is preliminarily detected by using the common characteristics of the chrominance signal and the luminance signal of the foreground shadow area in the image. The pixel determination method of the foreground shadow is shown in the following formula;

[0029]

[0030] Where, T Y is the brightness threshold, T Cb is the blue chromaticity threshold, T Cr is the red chromaticity threshold, I represents the foreground shadow area, and I Y (x,y),B Y (x,y),I Cb (x, y) is the three components of the foreground shadow area in the YCbCr color space; B is the background model, B Cb (x,y),I Cr (x,y),B Cr (x, y) is the three components of the background area in the YCbCr color space.

[0031] 2.3) Perform image morphological processing on the initially identified candidate foreground shadow region, separate the adhered regions and fill the internal holes, and define the extracted candidate foreground shadow region as S;

[0032] 2.4) Traverse each candidate area of ​​foreground shadow, and use the complete local binary pattern algorithm to extract texture features from the candidate area of ​​foreground shadow; calculate the histogram similarity between the candidate area of ​​foreground shadow and the corresponding area of ​​background model according to the chi-square distance, further judge the shadow area, and update the pixel value of the area identified as foreground shadow to the background to eliminate the shadow of the moving target;

[0033] (3) Target integrity improvement based on merging the bounding rectangles of adjacent targets;

[0034] The complete target is obtained by merging the bounding rectangles of adjacent targets. The method for determining whether adjacent targets belong to the same moving target is as follows;

[0035] A(x1, y1) and B(x2, y2) are the center points of the two foreground image detection frames, and θ is the horizontal angle between the two vectors; when θ is less than the angle threshold T θ And the horizontal distance between points A and B is less than the distance threshold T dWhen the two regions in the detection result are classified as the same moving target, the detection frames are merged based on the geometric center positions of the two detection frames and the detection frame width (w) and height (h) attributes, and finally the complete moving target outline is obtained;

[0036] θ≤Tθ

[0037] |x1-x2|≤T d

[0038] Where, T θ and T d The value of is determined by the width (w), height (h) of the detection box and the width (W), height (H) of the image; the overall outline of the non-rigid target is obtained by merging adjacent small target detection frames.

[0039] The warning zone intrusion determination method includes intrusion determination based on target location information and intrusion determination based on target classification information;

[0040] The intrusion determination method based on target position information obtains the coordinates of each corner point of each warning zone in the picture during the warning zone marking process; obtains the detection frame of each target during the real-time target recognition process; uses the "ray method" to determine the inclusion relationship between the two corner points at the bottom of the detection frame and the warning zone polygon to determine whether the target is intruded; if one corner point is within the target area, it is determined that there is an intrusion behavior; if both corner points are not within the target area, it is determined that there is no intrusion behavior;

[0041] The intrusion determination method based on target classification information is based on an application scenario in which some areas in a production workshop are not allowed to be entered by unrelated personnel except staff. The determination of intrusion into some warning areas is combined with the classification information of the target. When the target entering the warning area is an authorized target, it is not determined as an intrusion behavior. The time when the target enters the warning area is analyzed in combination with actual application conditions, and it is stipulated that an alarm will only be issued when the time when the target enters the warning area reaches a threshold.

[0042] The warning zone intrusion prediction is performed by tracking the target through the improved ECO-HC algorithm; a model update supervision mechanism based on the confidence of the target tracking response graph is proposed;

[0043] (1) Add the evaluation of tracking results to the target tracking process; calculate the tracking response map of the target, and obtain the confidence of the tracking result of the current frame by tracking the target response map; the confidence calculation method is as follows:

[0044]

[0045] Where V max (n) and V min (n) respectively represent the maximum and minimum values ​​of the tracking response graph of the nth frame; V (x,y)(n) represents the response value corresponding to the pixel with coordinate (x, y) in the tracking response map; N represents the total number of response values ​​in the tracking response map;

[0046] (2) Tracking results that meet the confidence threshold are used to update the filter;

[0047] (3) For tracking results that do not meet the confidence threshold, the positioning of the detection box is weighted and corrected using the Kalman filter motion prediction results; a Kalman filter is constructed based on the motion information of position and speed to predict the target trajectory, and the filter is updated based on the predicted results and the actual results.

[0048] The confidence threshold T c =0.65.

[0049] T Y =50, T Cb =10, T Cr =15.

[0050] A warning zone intrusion dynamic detection system includes a system management module, a device management module, a warning zone marking module, a warning zone intrusion determination module, a warning zone intrusion prediction module, and an intrusion information management module;

[0051] The system management module manages system information, completes user login, user information management and authority management;

[0052] The device management module switches the video data source, checks the real-time images of the monitoring devices and manages the monitoring devices;

[0053] The warning zone marking module includes different warning zone marking methods. Manual marking uses human-computer interaction to directly record the warning zone corners in the monitoring screen by puncturing points; when the warning zones are distributed in different monitoring device viewing angles, the warning zones are marked after scene splicing; the warning zone marking method that obtains the warning zone corner points by identifying safety cones is used for marking temporary warning zones;

[0054] The warning zone intrusion determination module performs real-time target detection, intrusion rule setting and intrusion event recording;

[0055] The warning zone intrusion prediction module performs LSTM trajectory prediction model training and warning zone intrusion prediction;

[0056] The intrusion information management module manages all warning zone intrusion detection results, and processes the alarm information of warning zone intrusion determination and the early warning information of warning zone intrusion prediction; by selecting the intrusion event type, query start time, and query end time to search for alarm information, the intrusion detection history information can be added, deleted, modified, and checked.

[0057] The beneficial effects of the present invention are as follows: (1) A production workshop scene data set was constructed based on actual application needs, and a YOLOv7 target detection model was trained to achieve basic recognition of staff, transport carts and safety cones. By improving the ViBe algorithm, the accuracy and recall rate were improved compared to the traditional ViBe algorithm. (2) A production workshop target detection method integrating the YOLOv7 and ViBe algorithms was proposed. By integrating the YOLOv7 and ViBe algorithms, the advantages of the two algorithms are combined to reduce the missed detection phenomenon of target detection in the production workshop scenario. (3) A warning zone intrusion determination method combining target detection and warning zone marking is proposed. (4) A warning zone intrusion prediction method combining target tracking and trajectory prediction is proposed. (5) A video-based production workshop warning zone intrusion dynamic detection system is designed and implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is the architecture diagram of the video-based production workshop warning area intrusion dynamic detection method;

[0059] Figure 2 To improve the ViBe algorithm flow chart;

[0060] Figure 3 is the result of moving target detection, where Figure 3(a) is the original image of each data set; Figure 3(b) is the standard result image of each data set; Figure 3(c) is the frame difference method result image of each data set; Figure 3(d) is the GMM algorithm result image; Figure 3(e) is the ViBe algorithm result image; Figure 3(f) is the improved ViBe algorithm result image;

[0061] Figure 4 To improve the target tracking framework of the ECO-HC algorithm;

[0062] Figure 5 shows the target tracking results under the OTB100 dataset, where (a) is the accuracy graph; Figure 5(b) is the success rate graph;

[0063] Figure 6 This is the overall architecture diagram of the system functional modules;

[0064] Figure 7 This is the overall flow chart of the system;

[0065] Figure 8 Schematic diagram for determining the same moving target. DETAILED DESCRIPTION

[0066] Based on the strict management requirements for safe production in production workshops and in view of the problems existing in the practical application of the current warning zone intrusion detection technology, the present invention is a warning zone intrusion determination method combining target detection and warning zone marking and a warning zone intrusion prediction method combining target tracking and trajectory prediction. Figure 1 It is the main flow chart of the technical solution of the present invention, such as Figure 1As shown, the warning zone intrusion dynamic detection method proposed by the present invention includes the following steps:

[0067] (1) Object detection in real-time video: YOLOv7 realizes basic recognition of staff, safety cones, and transport carts; the improved ViBe algorithm is used to extract moving targets.

[0068] (2) Warning zone marking for different monitoring scenarios: automatic marking of warning zones based on landmark line recognition in the production workshop; manual marking of warning zones based on scene stitching; automatic marking of temporary warning zones based on safety cone recognition.

[0069] (3) Storage of warning zone annotation information: Attribute information is stored, including current scene information, warning zone number, warning zone corner point coordinates, and warning zone remarks; image information is stored, and screenshots of warning zone annotation results are stored.

[0070] (4) Warning zone intrusion determination: real-time target detection; loading warning zone labeling related information to determine the relative position of the target and the warning zone; determining the target category authority; determining the time when the target enters the warning zone; and reporting the intrusion determination results.

[0071] (5) Warning zone intrusion prediction: real-time target detection; collecting historical target tracking trajectories; LSTM trajectory prediction model training; real-time target tracking, obtaining target trajectories; LSTM trajectory prediction; intrusion prediction based on prediction estimation.

[0072] (6) Intrusion event processing: screenshots of intrusion events; intrusion alarms; triggering of intrusion warning alarms; storage of intrusion event related information; and intrusion event statistics and publication.

[0073] The present invention is developed based on Python language, the compiler is Pycharm 2021, the front-end implementation uses PyQt5, the back-end is developed based on JAVA language, the back-end framework uses Springboot, and the mongoDB database is used to store screenshot files of intrusion events, and the MySQL database is used to store the alarm information of intrusion events.

[0074] The video-based dynamic detection system for intrusion in the warning zone of the production workshop can mark the warning zone of the scenes captured by the fixed monitoring equipment installed in the production workshop, and determine and predict the intrusion behavior based on the marked warning zone. When the judgment result of the warning zone intrusion is true, the intrusion event information is recorded and the alarm information is issued to remind the supervisor. For the prediction of intrusion in the warning zone of the key monitoring area, if it is predicted that a target is about to enter the warning zone, the on-site sound and light alarm equipment is activated in advance to achieve early warning of the target that is about to enter the warning zone.

[0075] The improved algorithm flow is as follows Figure 2The experimental results are shown in Figure 3. As shown in Figure 3, the moving target detection effect of the improved ViBe algorithm is significantly improved.

[0076] The video sequences in the highway dataset in the figure all have obvious light changes. The experimental results show that the traditional ViBe algorithm will lead to low integrity of the detected target due to light changes. The improved algorithm of the present invention can effectively eliminate internal pores and edge defects through the integrity improvement strategy.

[0077] The video sequences in the overpass dataset have dynamic backgrounds (swaying leaves). The experimental results show that the frame difference method has the best robustness in such scenes, but there are internal holes in the moving targets. The detection results of the GMM method have a lot of noise, and the method has poor adaptability to dynamic backgrounds. The traditional ViBe algorithm has a lot of noise due to the changes in the background. In comparison, the detection results of the improved ViBe algorithm have been significantly improved.

[0078] The video sequences in the subway monitoring dataset contain foreground shadows and slow-moving targets. The experimental results show that the four algorithms tested by the present invention have poor effects on slow-moving targets. The foreground shadow has a more significant impact on the GMM algorithm and the ViBe algorithm. The improved ViBe algorithm can effectively identify and eliminate the foreground shadow area.

[0079] In the video sequence of the pedestrian dataset, moving targets appear in the initial frame. This type of scene has the most serious impact on the ViBe algorithm, and there are obvious ghost areas in the detection results. The experimental results show that the improved ViBe algorithm can completely eliminate ghosts in a short time.

[0080] In the video sequence of the climber dataset, the moving target and the background color are similar. In the experimental results, the moving target extracted by ViBe is divided into three parts and three detection frames are drawn. The improved ViBe algorithm of the present invention extracts the complete target by merging the detection frames. Compared with the traditional ViBe algorithm, the precision and recall rate of the improved ViBe algorithm are improved by 0.182 and 0.181 respectively.

[0081] The present invention improves the ECO-HC algorithm. The improved algorithm flow is as follows: Figure 4 As shown in the figure. Aiming at the deficiency of the algorithm in evaluating the tracking results, a model updating supervision mechanism based on the confidence of the target tracking response graph is proposed. The main improvements include:

[0082] (1) Add the evaluation of tracking results to the target tracking process, obtain the confidence of the tracking result of the current frame by tracking the target response map, and only use the high confidence results to update the filter;

[0083] (2) For low confidence results that cause the detection frame position to shift, the Kalman filter motion prediction result is used to weight and correct the positioning of the detection frame.

[0084] The rationality of the confidence threshold determines whether the ECO-HC model update supervision mechanism of the present invention can achieve better results. Therefore, 8 groups of different confidence thresholds T are set. C Experiments are conducted on the OBT100 dataset. C The larger the value, the stricter the supervision mechanism. The tracking effect of the improved ECO-HC algorithm under different thresholds is evaluated by precision and success rate. C When =0.65, the tracking accuracy and success rate of the improved ECO-HC algorithm of the present invention are the highest.

[0085] The experimental results are shown in Figure 5. It can be concluded that the average tracking accuracy of the algorithm of the present invention under the OBT100 data set reached 86.9%, and the average tracking success rate reached 77.6%. Compared with the traditional ECO-HC algorithm, the accuracy was improved by 0.9%, and the success rate was improved by 1.5%. Compared with other advanced algorithms, it also has better performance. Table 1 is a statistical analysis of the experimental results of different algorithms. It can be seen from the table that the average frame number of the target tracking of the improved algorithm of the present invention is 31 (FPS=31), which can meet the real-time requirements.

[0086] Table 1 Comparison of tracking effects of different tracking algorithms

[0087]

[0088] This system is based on image processing technology, target detection technology, target tracking technology and trajectory prediction technology to achieve the core functions of the system, and record and save the detection results. The overall architecture of the system function module is as follows: Figure 6 The overall system architecture consists of six modules: system management, equipment management, warning zone marking, warning zone intrusion determination, warning zone intrusion prediction, and intrusion information management. The specific functions of each module are as follows:

[0089] (1) The system management module mainly implements functions such as user login, user management, and authority management;

[0090] (2) The equipment management module is mainly used to realize the real-time viewing of production workshop monitoring videos and the attribute information management of monitoring equipment;

[0091] (3) The warning area marking module is mainly used to realize the marking of the warning area and the storage of warning area related information (number, corner point coordinates, etc.);

[0092] (4) The warning zone intrusion determination module mainly implements the setting of intrusion determination rules (intrusion alarm time threshold, intrusion authority based on target classification), intrusion determination of warning zones, and release of intrusion alarm information;

[0093] (5) The regional intrusion prediction module is mainly used to realize the selection of tracking targets, the generation of LSTM trajectory prediction models, the prediction of intrusion in warning areas, and the release of intrusion warning information;

[0094] (6) The intrusion information management module is mainly used to view intrusion events (information such as intrusion time and video screenshots), process intrusion alarm information, process intrusion warning information (activate on-site sound and light alarm equipment), and add, delete, modify, and query information related to intrusion events.

[0095] The specific application process of this system is:

[0096] (1) The staff logs in to the software using their account and password;

[0097] (2) Enter the system management module to manage user information and set user permissions;

[0098] (3) Enter the device management module, select the system data source (local data or real-time monitoring data), select the monitoring device from the device list for real-time preview, and manage the device attribute information through the device management function;

[0099] (4) Enter the warning zone marking module, select different monitoring devices and use different warning zone marking methods to add or modify the relevant information of the current scene warning zone;

[0100] (5) Enter the warning zone intrusion determination module, select the monitoring device to automatically load the warning zone range information, set the intrusion determination rules, and start target detection to perform warning zone intrusion determination;

[0101] (6) Enter the warning zone intrusion prediction module, you can train the LSTM trajectory prediction model or directly load the historical model, enable target tracking and trajectory prediction to predict the intrusion of the warning zone;

[0102] (7) Enter the intrusion information management module to query information related to historical intrusion events and preview screenshots, and automatically issue alarm instructions or turn on on-site sound and light alarm equipment based on the intrusion information.

[0103] The overall process of the system is as follows Figure 7 shown.

[0104] The warning zone marking module includes different warning zone marking methods. Three warning zone marking methods are mainly implemented according to the characteristics of the production workshop. Manual marking uses human-computer interaction to directly record the corner points of the warning zone in the monitoring screen by puncturing points. When the warning zone is distributed in different monitoring equipment viewing angles, the warning zone can be marked after scene splicing. The warning zone marking method that obtains the corner points of the warning zone by identifying safety cones is often used for the marking of temporary warning zones. This method is more flexible and efficient. Since a large number of landmark lines are set up in the production workshop for easy management, the warning zone marking method based on landmark line extraction is the most commonly used.

[0105] The warning zone intrusion judgment module mainly includes functions such as real-time target detection, intrusion rule setting and intrusion event recording. This module needs to select the monitoring equipment that performs the warning zone intrusion judgment, automatically load the corresponding warning zone related information from the database according to the monitoring equipment serial number, call the packaged improved target detection program of the present invention when the target detection function is turned on, and output the target detection result for intrusion judgment. The intrusion rule setting mainly includes different types of intrusion authority settings and intrusion time threshold settings. When the warning zone intrusion judgment rules are met, the system will automatically mark (marked in red) the intrusion target, automatically take screenshots and publish intrusion event related information.

[0106] The warning zone intrusion prediction module mainly includes functions such as LSTM trajectory prediction model training and warning zone intrusion prediction. When this module is used for the first time, it is necessary to train the LSTM trajectory prediction model. The training data of the model is the historical trajectory of pedestrians in the current scene. The trained LSTM model is saved and stored in the local project file for direct selection and use in subsequent warning zone intrusion prediction. When predicting the intrusion of the warning zone, the system will automatically load the warning zone information according to the monitoring equipment information. After selecting the LSTM trajectory prediction model, you can manually select the tracking target, or you can automatically select the target closest to the warning zone through the automatic labeling button. After turning on the intrusion prediction function, the system will automatically give the predicted trajectory for warning zone intrusion prediction based on the real-time trajectory information, and issue intrusion warning information in a timely manner.

[0107] The intrusion information management module can manage all warning zone intrusion detection results, and process the alarm information of warning zone intrusion determination and warning information of warning zone intrusion prediction (monitoring room alarm or activation of sound and light alarm). Alarm information search can be performed by selecting the intrusion event type, query start time, and query end time, and the functions of adding, deleting, modifying and checking the intrusion detection history information are realized.

[0108] Based on the detection requirements of targets such as workers, transport carts, and safety cones in the production workshop, this paper constructs a target detection dataset for the production workshop and trains the YOLOv7 target detection model. In view of the problems of ghost phenomenon, motion foreground, and incomplete target extraction in the traditional ViBe algorithm when performing moving target detection, a ghost area elimination method based on histogram similarity judgment, a foreground shadow detection method based on color features combined with texture features in the YCbCr color space, and a target integrity improvement method based on the CodeBook algorithm color distortion measurement are proposed.

[0109] The workshop target detection method that integrates YOLOv7 and ViBe algorithms combines the advantages of the two algorithms to reduce the missed detection of target detection in the workshop scenario. In the algorithm fusion strategy of the present invention, the difference between the YOLOv7 target detection result and the ViBe algorithm detection result is used to eliminate a large amount of redundant information in the ViBe detection result, effectively reducing the computing time consumption caused by the subsequent ViBe improvement strategy, and ensuring the real-time performance of target detection.

[0110] Based on the characteristics of the actual application scenarios in the production workshop, different warning zone labeling methods based on scene splicing, landmark line extraction and safety cone recognition are proposed to clarify the scope of the warning zone in different scenarios. A more flexible and reasonable warning zone intrusion judgment method is achieved by comprehensively considering the relative position relationship between the target and the warning zone, target classification information, and the time of entering the warning zone.

[0111] In the production workshop scenario, in order to solve the problem of tracking drift or loss caused by the obstruction of the tracking target by the production equipment in the traditional ECO-HC algorithm, this paper introduces an improved ECO-HC method that introduces a model update supervision mechanism based on the confidence of the response graph. In order to solve the problem of detection frame offset caused by the model update supervision mechanism, the detection frame positioning correction method based on Kalman filter motion prediction is further studied to achieve high-precision target tracking. On this basis, the LSTM trajectory prediction model is used to predict the trajectory according to the real-time trajectory, and finally the warning zone intrusion prediction is realized according to the relative position relationship between the predicted trajectory and the warning zone.

[0112] The present invention improves the ViBe algorithm and proposes a ghost discrimination method based on the similarity measurement of the target foreground area histogram and the corresponding neighborhood background area histogram. The method can effectively identify the ghost area in the ViBe detection result. After the ghost area is identified, the pixel value of the ghost area is assigned to 0 when the ViBe algorithm background model is updated to achieve the effect of quickly eliminating the ghost.

[0113] A foreground shadow detection method combining color features and texture features in YCbCr color space is proposed. Combining the two features can effectively improve the accuracy of shadow recognition. Combining the two features can effectively improve the accuracy of shadow recognition.

[0114] Aiming at the influence of noise on the extraction results in ViBe moving target detection, the present invention mainly uses image processing related technologies for denoising. Aiming at the phenomenon of incomplete moving targets in the detection results due to the similarity between the foreground color and the background color, the present invention will improve the detection effect by improving the foreground detection strategy in the ViBe algorithm, and by drawing on the color distortion measurement method in the CodeBook algorithm, a new distance function for pixel similarity determination is proposed, and an adaptive threshold is generated according to the background complexity.

Claims

1. A method for dynamic detection of intrusion in a warning area, characterized in that: Based on the warning zone marking results, the warning zone intrusion determination and intrusion prediction are realized by performing target detection and target tracking on the target, including the following steps: Step 1. Warning zone marking for different monitoring scenarios: Different methods are used to mark the warning zone range according to the monitoring scenarios: if the monitoring scenario is a production workshop, the warning zone is automatically marked based on the landmark line extraction; if the monitoring scenario is the perspective of monitoring equipment distributed in different places, the warning zone is manually marked after scene splicing; if there are safety cones in the monitoring scene, the corner points of the warning zone are identified according to the safety cones and the warning zone is marked; Step 2: Storage of warning zone annotation information: the warning zone annotation information includes attribute information and file information; the attribute information includes current scene information, warning zone number, warning zone intersection coordinates, and warning zone remark information; the file information includes a screenshot of the warning zone annotation result, image size, and image name; Step 3: Target detection in real-time video: perform basic recognition of staff, safety cones, and transport carts using the YOLOv7 target detection algorithm; extract moving targets based on the improved ViBe algorithm; perform differential analysis between the YOLOv7 target detection result and the improved ViBe algorithm extraction result to obtain real-time target detection results; Step 4: Warning zone intrusion determination: Based on the real-time target detection results of step 3 and the warning zone marking information loaded in step 2, determine the relative position relationship between the target and the warning zone, determine the target category authority, determine the time when the target enters the warning zone, and report the intrusion determination result; Step 5: Warning zone intrusion prediction: According to the improved ECO-HC algorithm, the real-time target detection results of step 3 are tracked in real time, the trajectory of the target is recorded as training data, and the LSTM trajectory prediction model is trained; the future trajectory of the target is predicted by the LSTM trajectory prediction model; Intrusion prediction based on the target's future trajectory and warning zone annotation information; The improved ViBe algorithm includes ghost phenomenon elimination based on histogram similarity, foreground shadow elimination based on YCbCr space color features and texture features, and target integrity improvement based on merging adjacent target bounding rectangles, as follows: (1) Ghost elimination based on histogram similarity; 1.1) Use the ViBe algorithm to perform background modeling on the initial frame image and establish a background model; 1.2) Detect moving targets in subsequent frames, and perform corrosion processing on the detection results to remove the noise points of the panoramic target adhesion, obtain the foreground target areas of the current frame, and obtain the foreground block F i , i=1,2,…,N, i is the sequence number, N is the total number of foreground target areas; 1.3) Calculate the foreground block F i The histogram FH of the current frame i ; 1.4) Take the minimum enclosing rectangle R of each foreground target area i , and its corresponding neighborhood background area B i The calculation formula is B i =R i -F i , get B i The histogram BH of the current frame i ; 1.5) By calculating FH i and BH i The similarity is further used to determine the Ghost area; After the Ghost area is identified, the pixel value of the Ghost area is assigned to 0 when the ViBe algorithm background model is updated, and the background model is updated; when the Ghost area is not identified, the foreground shadow is eliminated based on the YCbCr space color features and texture features; (2) Foreground shadow removal based on YCbCr spatial color features and texture features; First, the image without ghost area identified in step 1.5) is converted from RGB color space to YCbCr color space. The specific conversion formula is as follows: Y=0.299·R+0.587·G+0.114·B Cr=(RY)·0.713+delta Cb=(BY)·0.564+delta Y is the brightness component of the color, R is the red component of the color, G is the green component of the color, B is the blue component of the color, Cb and Cr are the concentration offset component of blue and the concentration offset component of red respectively; In the YCbCr color space, the brightness and chromaticity information of the pixels are used to filter out the candidate areas of the foreground shadow. The specific steps are as follows: 2.1) According to the neighborhood concept of the image, the corresponding Y, Cb, and Cr components are calculated based on the R, G, and B components of the pixel point P(x, y) in the foreground target area and the above conversion formula, and the 8-neighborhood weighted average calculation method is used when calculating the Y, Cb, and Cr components of the point; 2.2) Based on the fact that the foreground shadow pixels show the characteristic of reduced brightness and unchanged chrominance in the YCbCr color space, the candidate foreground shadow area is preliminarily detected by using the common characteristics of the chrominance signal and the luminance signal of the foreground shadow area in the image. The pixel determination method of the foreground shadow is shown in the following formula; Where, T Y is the brightness threshold, T Cb is the blue chromaticity threshold, T Cr is the red chromaticity threshold, I represents the foreground shadow area, and I Y (x,y),B Y (x,y),I Cb (x, y) is the three components of the foreground shadow area in the YCbCr color space; B is the background model, B Cb (x,y),I Cr (x,y),B Cr (x, y) is the three components of the background area in the YCbCr color space; 2.3) Perform image morphological processing on the initially identified candidate foreground shadow region, separate the adhered regions and fill the internal holes, and define the extracted candidate foreground shadow region as S; 2.4) Traverse each candidate area of ​​the foreground shadow and use the complete local binary pattern algorithm to extract texture features from the candidate area of ​​the foreground shadow; The histogram similarity is calculated based on the chi-square distance between the candidate area of ​​the foreground shadow area and the corresponding area of ​​the background model, and the shadow area is further determined, and the pixel value of the area identified as the foreground shadow is updated to the background to eliminate the shadow of the moving target; (3) Target integrity improvement based on merging the bounding rectangles of adjacent targets; The complete target is obtained by merging the bounding rectangles of adjacent targets. The method for determining whether adjacent targets belong to the same moving target is as follows; A(x1, y1) and B(x2, y2) are the center points of the two foreground image detection frames, and θ is the horizontal angle between the two vectors; when θ is less than the angle threshold T θ And the horizontal distance between points A and B is less than the distance threshold T d When the two regions in the detection result are classified as the same moving target, the detection frames are merged based on the geometric center positions of the two detection frames and the detection frame width (w) and height (h) attributes, and finally the complete moving target outline is obtained; θ≤T θ |x1-x2|≤T d Where, T θ and T d The value of is determined by the width (w), height (h) of the detection box and the width (W), height (H) of the image; the overall contour of the non-rigid target is obtained by merging adjacent small target detection boxes; The warning zone intrusion prediction is performed by tracking the target through the improved ECO-HC algorithm; a model update supervision mechanism based on the confidence of the target tracking response graph is proposed; (1) Add the evaluation of the tracking results to the target tracking process; calculate the tracking response map of the target, and obtain the confidence of the tracking result of the current frame by tracking the target response map; the confidence calculation method is as follows: Where V max (n) and V min (n) respectively represent the maximum and minimum values ​​of the tracking response graph of the nth frame; V (x,y) (n) represents the response value corresponding to the pixel with coordinate (x, y) in the tracking response map; N represents the total number of response values ​​in the tracking response map; (2) Tracking results that meet the confidence threshold are used to update the filter; (3) For tracking results that do not meet the confidence threshold, the positioning of the detection box is weighted and corrected using the Kalman filter motion prediction results; a Kalman filter is constructed based on the motion information of position and speed to predict the target trajectory, and the filter is updated based on the predicted results and the actual results.

2. The method for dynamic detection of intrusion into a warning zone according to claim 1, characterized in that: The warning zone intrusion determination method includes intrusion determination based on target location information and intrusion determination based on target classification information; The intrusion determination method based on target position information obtains the coordinates of each corner point of each warning zone in the screen during the warning zone marking process; obtains the detection frame of each target during the real-time target recognition process; and determines whether the target is intruded by using the "ray method" to determine the inclusion relationship between the two corner points at the bottom of the detection frame and the warning zone polygon; if one corner point is within the target area, it is determined that there is an intrusion behavior; if both corner points are not within the target area, it is determined that there is no intrusion behavior; The intrusion determination method based on target classification information is based on an application scenario in which some areas in a production workshop are not allowed to be entered by unrelated personnel except staff. The determination of intrusion into some warning areas is combined with the classification information of the target. When the target entering the warning area is an authorized target, it is not determined as an intrusion behavior. The time when the target enters the warning area is analyzed in combination with actual application conditions, and it is stipulated that an alarm will only be issued when the time when the target enters the warning area reaches a threshold.

3. The method for dynamic detection of intrusion into a warning zone according to claim 1, characterized in that: The confidence threshold T c =0.

65.

4. The method for dynamic detection of intrusion into a warning zone according to claim 1, characterized in that: T Y =50、T Cb =10、T Cr =15。 5. A warning zone intrusion dynamic detection system, characterized in that: The warning zone intrusion dynamic detection system includes a system management module, a device management module, a warning zone marking module, a warning zone intrusion determination module, a warning zone intrusion prediction module, and an intrusion information management module; The system management module manages system information, completes user login, user information management and authority management; The device management module switches the video data source, checks the real-time images of the monitoring devices and manages the monitoring devices; The warning zone marking module includes different warning zone marking methods. Manual marking uses human-computer interaction to directly record the warning zone corners in the monitoring screen by puncturing points; when the warning zones are distributed in different monitoring device viewing angles, the warning zones are marked after scene splicing; the warning zone marking method that obtains the warning zone corner points by identifying safety cones is used for marking temporary warning zones; The warning zone intrusion determination module performs real-time target detection, intrusion rule setting and intrusion event recording; The warning zone intrusion prediction module performs LSTM trajectory prediction model training and warning zone intrusion prediction; The intrusion information management module manages all warning zone intrusion detection results, and processes the alarm information of warning zone intrusion determination and the early warning information of warning zone intrusion prediction; by selecting the intrusion event type, query start time, and query end time to search for alarm information, the intrusion detection history information can be added, deleted, modified, and checked.

Images