Abnormal intrusion monitoring method and system based on video stream analysis in combination with Internet of Things

Abstract

The invention relates to the technical field of image processing, in particular to an abnormal intrusion monitoring method and system based on video stream analysis in combination with the Internet of Things, and the method comprises the steps: obtaining a regional monitoring video set, sequentially extracting regional monitoring videos, carrying out the target detection, obtaining a detection target if the detection result is that the detection exists, and carrying out the analysis of the detection target, calculating a target area and a target proportion, performing target judgment to obtain a target judgment result, if the target judgment result is a target to be detected, obtaining a detection time set and a detection center set, calculating a movement speed, setting decision nodes, and constructing a target decision map by using the decision nodes; and judging the detection target according to the movement speed and the target decision map to obtain map judgment data, and completing abnormal intrusion monitoring according to the map judgment data. According to the invention, the accuracy of abnormal intrusion monitoring can be improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an abnormal intrusion monitoring method and system based on video stream analysis combined with the Internet of Things. Background Art

[0002] With the increasing demand for urban security construction and public place management, abnormal intrusion monitoring plays a vital role in protecting people's lives and property and maintaining social stability. Through real-time collection and analysis of regional surveillance videos, targets of abnormal intrusions can be discovered in a timely manner. Efficient and reliable abnormal intrusion monitoring can not only effectively curb criminal activities, but also provide important support for personnel evacuation and emergency rescue in emergency situations.

[0003] Traditional methods for detecting abnormal intrusions mostly rely on single motion detection, which has significant shortcomings in practical applications. These methods lack the ability to accurately detect abnormal stops, abnormal obscurations, and abnormal speeds, making it difficult to accurately distinguish between normal activity and potential threats. Therefore, improving the accuracy of abnormal intrusion detection is a pressing issue. Summary of the Invention

[0004] The present invention provides an abnormal intrusion monitoring method and system based on video stream analysis combined with the Internet of Things, the main purpose of which is to improve the accuracy of abnormal intrusion monitoring.

[0005] To achieve the above objectives, the present invention provides an abnormal intrusion monitoring method based on video stream analysis combined with the Internet of Things, comprising:

[0006] Obtaining a regional surveillance video set, sequentially extracting regional surveillance videos from the regional surveillance video set, and performing target detection based on the extracted regional surveillance videos to obtain a detection result, wherein the detection result is detection presence or detection absence;

[0007] If the detection result is that the detection does not exist, return to the above step of obtaining the regional surveillance video set until the detection result is that the detection exists;

[0008] If the detection result is existence, the detection target is obtained based on the detection result and the regional surveillance video corresponding to the detection result;

[0009] Analyze the detected target to obtain the target bounding box feature coordinates, and calculate the target area and target ratio based on the target bounding box feature coordinates;

[0010] Performing target determination based on the target area and target ratio to obtain a target determination result, wherein the target determination result is a target to be verified or a non-target to be verified;

[0011] If the target determination result is not the target to be verified, return to the above step of using the pre-built surveillance camera to obtain the regional surveillance video set until the target determination result is the target to be verified;

[0012] If the target is determined to be a target to be tested, a preset target following algorithm is used to continuously collect data on the target to obtain a detection time set and a detection center set, wherein the detection time in the detection time set corresponds to the detection center in the detection center set one-to-one;

[0013] Calculate the motion speed according to the detection time set and the detection center set, and set a decision node, wherein the decision node includes: a face occlusion node, an abnormal cover node, an abnormal entry node, an abnormal stay node, an abnormal speed node, and an alarm triggering node;

[0014] The decision nodes are used to construct a target decision map, and the detection target is judged according to the movement speed and the target decision map to obtain map judgment data, and abnormal intrusion monitoring is completed according to the map judgment data.

[0015] Optionally, analyzing the detected target to obtain target bounding box feature coordinates includes:

[0016] Acquire a detection image according to the detection target, extract a reference point from the detection image, and establish a boundary detection coordinate system with the reference point as the origin, wherein the reference point is the upper left corner vertex of the detection image;

[0017] Detect the detection target using a preset target detection algorithm and boundary detection coordinate system to obtain a detection bounding box set, a bounding box starting point set, and a detection confidence set, wherein the detection bounding boxes in the detection bounding box set correspond one-to-one to the bounding box starting points in the bounding box starting point set, and the bounding box starting points in the bounding box starting point set correspond one-to-one to the detection confidences in the detection confidence set;

[0018] Extracting detection confidences from the detection confidence set in sequence, and comparing the detection confidences with a preset confidence threshold;

[0019] If the detection confidence is less than the confidence threshold, return to the above step of detecting the target using the preset target detection algorithm until the detection confidence is greater than or equal to the confidence threshold;

[0020] If the detection confidence is not uniformly less than the confidence threshold, the detection confidences that are greater than or equal to the confidence threshold are aggregated to obtain the judgment confidence set;

[0021] The judgment confidence set is screened to obtain the target confidence, and the target bounding box is obtained from the detection bounding box set according to the target confidence. The bounding box starting point is matched from the bounding box starting point set according to the target bounding box, and the starting point coordinates are obtained according to the bounding box starting point, and the starting point coordinates are confirmed as the target bounding box feature coordinates.

[0022] Optionally, calculating the target area and target ratio according to the target bounding box feature coordinates includes:

[0023] Obtaining a bounding box width and a bounding box height according to the target bounding box and the target bounding box feature coordinates;

[0024] Calculate the target area using the bounding box width and the bounding box height, where the target area is the product of the bounding box width and the bounding box height;

[0025] The target ratio is calculated based on the bounding box width and bounding box height. The calculation formula is as follows:

[0026]

[0027] Here, s refers to the target scale, g refers to the bounding box height, and k refers to the bounding box width.

[0028] Optionally, calculating the motion speed according to the detection time set and the detection center set includes:

[0029] Extracting detection centers from the detection center set in sequence, obtaining detection center coordinates based on the extracted detection centers, and collecting the detection center coordinates to obtain a detection center coordinate set;

[0030] The last detection time is extracted from the detection time set, and the initial center coordinates and the final center coordinates are extracted from the detection center coordinate set. The detection displacement is calculated based on the last detection time, the initial center coordinates, and the final center coordinates. The initial center coordinates include: the initial center horizontal coordinate and the initial center vertical coordinate, and the final center coordinates include: the final center horizontal coordinate and the final center vertical coordinate. The calculation formula of the detection displacement is as follows:

[0031]

[0032] Where β refers to the detection displacement, f refers to the last detection time, x1 refers to the last center horizontal coordinate, x2 refers to the initial center horizontal coordinate, y1 refers to the last center vertical coordinate, and y2 refers to the initial center vertical coordinate.

[0033] Optionally, setting a decision node includes:

[0034] Construct a decision node determination method, wherein the decision node determination method includes: a face occlusion method, an abnormality concealment method, an abnormal entry method, a stay abnormality method, a speed abnormality method, and an alarm triggering method. The abnormality concealment method is as follows:

[0035] Get target detection frame;

[0036] Continuous detection is performed based on a preset number of continuous detection frames and target detection frames to obtain a continuous image set;

[0037] Extracting continuous images from the continuous image set in sequence, and determining whether the detection target exists in the extracted continuous images;

[0038] If there is a detection target in the continuous image, the target detection frame is reset according to the number of continuous detection frames to obtain a reset detection frame, the reset detection frame is used to update the target detection frame, and the updated target detection frame is used to return to the above step of continuous detection based on the preset number of continuous detection frames and target detection frames;

[0039] If the detection target does not exist in the continuous image, the detection target is marked as an abnormal target;

[0040] Obtain training data, and use the training data to train a pre-built segmentation learning model to obtain a training learning model;

[0041] Decision nodes are set according to the decision node determination method, training learning model and abnormal targets.

[0042] Optionally, obtaining training data includes:

[0043] Obtain a historical surveillance video set, extract historical surveillance videos from the historical surveillance video set in sequence, extract facial image frame sequences from the extracted historical surveillance videos, and aggregate the facial image frame sequences to obtain a facial image frame sequence set;

[0044] Setting a facial annotation and an occlusion annotation, and annotating the facial image frame sequence set using the facial annotation to obtain an annotated image frame sequence set, wherein the facial annotation is 1 or 0;

[0045] Acquire an occluded image frame set from the annotated image frame sequence set, wherein the occluded image frame set consists of annotated image frames whose face is annotated as 1 in the annotated image frame sequence set;

[0046] Using the occlusion annotation to mark the occlusion image frame set, and collecting the occlusion image frames with the same occlusion annotation in the occlusion image frame set to obtain an object occlusion image set and a body occlusion image set;

[0047] The labeled image frame sequence set, the object occlusion image set, and the body occlusion image set are identified as training data.

[0048] Optionally, the face occlusion method is as follows:

[0049] Obtain a target facial image, perform facial detection on the target facial image using a preset face detection algorithm, and obtain a facial bounding box;

[0050] Segmenting the target facial image based on the facial bounding box to obtain a facial image;

[0051] Calculating the actual facial area based on the facial image, and performing forward propagation on the facial image using the trained learning model to obtain a pixel occlusion probability set, wherein each pixel in the facial image corresponds to a pixel occlusion probability;

[0052] Extracting pixel occlusion probabilities from the pixel occlusion probability set in sequence, and comparing a preset demarcation threshold with the pixel occlusion probability;

[0053] If it is confirmed that the pixel occlusion probability is greater than or equal to the demarcation threshold, the pixel point corresponding to the pixel occlusion probability is marked as an occluded pixel point, and the occluded pixel points are collected to obtain an occluded pixel point set; the occluded pixel point set is weighted to obtain a first pixel point set and a second pixel point set;

[0054] The first pixel number and the second pixel number are counted according to the first pixel point set and the second pixel point set, and the facial occlusion rate is calculated using the actual facial area, the first pixel number and the second pixel number:

[0055]

[0056] Among them, j refers to the facial occlusion rate, Refers to the preset first weight, a dy Refers to the first pixel number, Refers to the preset second weight, a de Refers to the second pixel number, S am Refers to the actual facial area;

[0057] Comparing the facial occlusion rate with a preset occlusion rate threshold;

[0058] If it is confirmed that the facial occlusion rate is greater than the occlusion rate threshold, the target facial image is marked as an abnormal target.

[0059] Optionally, performing weighted division on the occluded pixel point set to obtain the first pixel point set and the second pixel point set includes:

[0060] Obtaining facial reference points based on the facial image, and calculating a first facial segmentation area according to the actual facial area, wherein the first facial segmentation area is 66% of the actual facial area;

[0061] Segmenting the facial image using the facial reference points and the first facial segmentation area to obtain a first facial segmentation image and a second facial segmentation image;

[0062] The first face segmentation image is filtered from the occluded pixel point set to obtain a first pixel point set, and the second face segmentation image is filtered from the occluded pixel point set to obtain a second pixel point set.

[0063] Optionally, constructing a target decision graph using the decision nodes includes:

[0064] Construct an entry-stay edge from the abnormal entry node to the stay-abnormal node, and construct a stay-alarm edge from the stay-abnormal node to the alarm triggering node;

[0065] Construct an entry speed edge from the abnormal entry node to the speed abnormality node, and construct a speed alarm edge from the speed abnormality node to the alarm triggering node;

[0066] Constructing a masked face edge from the abnormal masking node to the face occlusion node, and constructing a face alarm edge from the face occlusion node to the alarm triggering node;

[0067] Constructing a mask-entry edge from the face occlusion node to the abnormal entry node, and constructing an entry-alarm edge from the abnormal entry node to the alarm trigger node;

[0068] Construct a stop-speed edge from the stop-abnormal node to the speed-abnormal node;

[0069] A target decision graph is constructed according to the entry and stop edges, stop warning edges, entry speed edges, speed warning edges, masked face edges, face warning edges, masked entry edges, entry warning edges, stop speed edges and decision nodes.

[0070] To achieve the above objectives, the present invention further provides an abnormal intrusion monitoring system based on video stream analysis combined with the Internet of Things, comprising:

[0071] A detection target acquisition module is used to obtain a regional surveillance video set, sequentially extract regional surveillance videos from the regional surveillance video set, and perform target detection based on the extracted regional surveillance videos to obtain a detection result, wherein the detection result is detection presence or detection absence;

[0072] If the detection result is that the detection does not exist, return to the above step of obtaining the regional surveillance video set until the detection result is that the detection exists;

[0073] If the detection result is existence, the detection target is obtained based on the detection result and the regional surveillance video corresponding to the detection result;

[0074] The determination result acquisition module is used to analyze the detected target, obtain the target bounding box feature coordinates, and calculate the target area and target ratio based on the target bounding box feature coordinates;

[0075] Performing target determination based on the target area and target ratio to obtain a target determination result, wherein the target determination result is a target to be verified or a non-target to be verified;

[0076] If the target determination result is not the target to be verified, return to the above step of using the pre-built surveillance camera to obtain the regional surveillance video set until the target determination result is the target to be verified;

[0077] If the target is determined to be a target to be tested, a preset target following algorithm is used to continuously collect data on the target to obtain a detection time set and a detection center set, wherein the detection time in the detection time set corresponds to the detection center in the detection center set one-to-one;

[0078] a continuous data acquisition module, configured to calculate the motion speed based on the detection time set and the detection center set, and set decision nodes, wherein the decision nodes include: a face occlusion node, an abnormal occlusion node, an abnormal entry node, an abnormal stay node, an abnormal speed node, and an alarm triggering node;

[0079] The decision map construction module is used to use the decision nodes to construct a target decision map, determine the detection target according to the movement speed and the target decision map, obtain map determination data, and complete abnormal intrusion monitoring according to the map determination data.

[0080] In order to solve the above problem, the present invention further provides an electronic device, comprising:

[0081] a memory storing at least one instruction;

[0082] The processor executes the instructions stored in the memory to implement the above-mentioned abnormal intrusion monitoring method based on video stream analysis combined with the Internet of Things.

[0083] In order to solve the above problems, the present invention also provides a computer-readable storage medium, which stores at least one instruction. The at least one instruction is executed by a processor in an electronic device to implement the above-mentioned abnormal intrusion monitoring method based on video stream analysis combined with the Internet of Things.

[0084] In order to solve the problems described in the background technology, the present invention first obtains a regional monitoring video set, which includes monitoring videos of all divided areas in the monitoring area, providing an all-round field of view for subsequent detection, reducing the risk of missed detection, and ensuring the continuity and integrity of data sources through unified management and call, laying the foundation for real-time and rapid response; secondly, extracting regional monitoring videos in turn and performing target detection to obtain detection results, and traversing the regional monitoring videos of each divided area in order can ensure that each area is inspected without omission. The binary result of target detection simplifies the decision logic and helps to quickly determine whether to enter subsequent deep processing, which not only saves computing resources but also can quickly locate suspected intrusion activities. When the detection result is that the detection does not exist, the regional monitoring video set is re-acquired. When the detection result is that the detection exists, the corresponding video segment is immediately extracted for deep analysis to avoid wasting resources on irrelevant videos and improve the efficiency and accuracy of judgment; thirdly, the detection target is detected. The target area and target ratio are calculated, and the geometric features of the target are accurately extracted to assist in distinguishing different subjects and filter out small noise targets or false detections. The target area and target ratio are used to determine whether the target is to be tested or not. By judging the target area and target ratio, the targets that are not related to the intrusion are quickly eliminated, reducing the false alarm rate. Afterwards, if the target is determined to be a target to be tested, the target following algorithm is used to continuously collect data on the detected target. The target following algorithm can establish a stable spatiotemporal trajectory for a single target, record its detection center and detection time, and provide accurate continuous data for subsequent analysis. Finally, a decision node is set, and a target decision map is constructed using the decision node. By setting multi-dimensional decision nodes, multiple abnormal behaviors of the detected target can be monitored in parallel. The multi-node parallel design enhances the coverage of complex intrusion methods, avoids the failure of a single rule, and significantly improves the accuracy of abnormal intrusion monitoring. Therefore, the present invention can improve the accuracy of abnormal intrusion monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 A flowchart of an abnormal intrusion monitoring method based on video stream analysis combined with the Internet of Things provided by one embodiment of the present invention;

[0086] Figure 2 This is a functional module diagram of an abnormal intrusion monitoring system based on video stream analysis combined with the Internet of Things provided by one embodiment of the present invention;

[0087] Figure 3 A schematic diagram of the structure of an electronic device for implementing the abnormal intrusion monitoring method based on video stream analysis combined with the Internet of Things provided in one embodiment of the present invention.

[0088] Description of reference numerals:

[0089] 1. Electronic device; 10. Processor; 11. Memory; 12. Bus.

[0090] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0091] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0092] The embodiments of the present application provide a method for monitoring abnormal intrusions based on video stream analysis combined with the Internet of Things. The execution subject of the method for monitoring abnormal intrusions based on video stream analysis combined with the Internet of Things includes but is not limited to at least one of electronic devices such as a server, a terminal, etc. that can be configured to execute the method provided by the embodiments of the present application. In other words, the method for monitoring abnormal intrusions based on video stream analysis combined with the Internet of Things can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server, or a cloud server cluster, etc.

[0093] Reference Figure 1 FIG. 1 is a flow chart of an abnormal intrusion monitoring method based on video stream analysis combined with the Internet of Things provided by one embodiment of the present invention. In this embodiment, the abnormal intrusion monitoring method based on video stream analysis combined with the Internet of Things includes:

[0094] S1. Obtain a regional surveillance video set, extract regional surveillance videos from the regional surveillance video set in sequence, and perform target detection based on the extracted regional surveillance videos to obtain a detection result, wherein the detection result is detection presence or detection absence.

[0095] Explainably, obtaining a regional surveillance video set refers to obtaining a regional surveillance video set using a surveillance camera group. A surveillance camera group refers to a combination of a fixed-view camera and a panoramic-view camera. A surveillance camera refers to a camera used to capture images within a surveillance area in real time. A regional surveillance video set refers to a set of regional surveillance videos. The surveillance area is divided into multiple partitioned areas by the surveillance camera group. A regional surveillance video refers to a surveillance video of a partitioned area. For example, the surveillance area is divided into five partitioned areas, and each partitioned area is provided with several fixed-view cameras and panoramic-view cameras. The surveillance videos corresponding to the partitioned areas are the regional surveillance videos, and the set of surveillance videos of all partitioned areas is the regional surveillance video set. The detection result refers to the result obtained after performing target detection on the regional surveillance video. Performing target detection on the regional surveillance video refers to detecting dynamic objects in the regional surveillance video. Optionally, the dynamic objects in the regional surveillance video can be detected using background modeling and foreground difference methods. Each frame of the regional surveillance video is compared with the background of the regional surveillance video. The foreground difference is the dynamic object. This is a prior art and will not be described in detail here. The dynamic objects detected in the regional surveillance video are. Existence detection means that after target detection is performed on the regional surveillance video, a dynamic object is found in the regional surveillance video. Non-existence detection means that after target detection is performed on the regional surveillance video, a dynamic object is found in the regional surveillance video.

[0096] S2. If the detection result is that the detection does not exist, return to the above step of obtaining the regional monitoring video set until the detection result is that the detection exists. If the detection result is that the detection exists, obtain the detection target according to the detection result and the regional monitoring video corresponding to the detection result.

[0097] It can be understood that the detection target refers to the dynamic object in the regional monitoring video.

[0098] S3. Analyze the detected target to obtain the target bounding box feature coordinates, and calculate the target area and target ratio based on the target bounding box feature coordinates.

[0099] Specifically, analyzing the detected target to obtain the target bounding box feature coordinates includes:

[0100] Acquire a detection image according to the detection target, extract a reference point from the detection image, and establish a boundary detection coordinate system with the reference point as the origin, wherein the reference point is the upper left corner vertex of the detection image;

[0101] Detect the detection target using a preset target detection algorithm and boundary detection coordinate system to obtain a detection bounding box set, a bounding box starting point set, and a detection confidence set, wherein the detection bounding boxes in the detection bounding box set correspond one-to-one to the bounding box starting points in the bounding box starting point set, and the bounding box starting points in the bounding box starting point set correspond one-to-one to the detection confidences in the detection confidence set;

[0102] Extracting detection confidences from the detection confidence set in sequence, and comparing the detection confidences with a preset confidence threshold;

[0103] If the detection confidence is less than the confidence threshold, return to the above step of detecting the target using the preset target detection algorithm until the detection confidence is greater than or equal to the confidence threshold;

[0104] If the detection confidence is not uniformly less than the confidence threshold, the detection confidences that are greater than or equal to the confidence threshold are aggregated to obtain the judgment confidence set;

[0105] The judgment confidence set is screened to obtain the target confidence, and the target bounding box is obtained from the detection bounding box set according to the target confidence. The bounding box starting point is matched from the bounding box starting point set according to the target bounding box, and the starting point coordinates are obtained according to the bounding box starting point, and the starting point coordinates are confirmed as the target bounding box feature coordinates.

[0106] Explainably, the detection image refers to the image of the detection target in the regional monitoring video, the boundary detection coordinate system refers to the coordinate system established based on the reference point, the vertical axis direction of the boundary detection coordinate system is the direction from the origin to directly below the origin, and the horizontal axis direction of the boundary detection coordinate system is the direction from the origin to directly to the right of the origin, and the target detection algorithm refers to the algorithm used to obtain the detection bounding box set, bounding box information set and detection confidence set of the detection target. Optionally, the target detection algorithm is the YOLO algorithm. The detection bounding box set refers to the set composed of detection bounding boxes, the bounding box starting point set refers to the set composed of bounding box starting points, the detection confidence set refers to the set composed of detection confidences, the detection bounding box refers to the bounding box output by the target detection algorithm after detecting the detection target, the detection confidence refers to the confidence output by the target detection algorithm after detecting the detection target, only when the detection confidence is greater than or equal to the confidence threshold, the detection bounding box corresponding to the detection confidence is considered to be able to correctly reflect the information of the detection target, the bounding box starting point refers to the upper left corner of the detection bounding box, the confidence threshold refers to the threshold set manually for judging the detection confidence, the judgment confidence set refers to the set composed of detection confidences greater than or equal to the confidence threshold, the target confidence refers to the confidence with the largest value in the judgment confidence set, the target bounding box refers to the detection bounding box in the detection bounding box set corresponding to the target confidence, and the starting point coordinates refer to the coordinates of the starting point of the bounding box corresponding to the target bounding box in the boundary detection coordinate system.

[0107] Specifically, the calculation of the target area and target ratio based on the target bounding box feature coordinates includes:

[0108] Obtaining a bounding box width and a bounding box height according to the target bounding box and the target bounding box feature coordinates;

[0109] Calculate the target area using the bounding box width and the bounding box height, where the target area is the product of the bounding box width and the bounding box height;

[0110] The target ratio is calculated based on the bounding box width and bounding box height. The calculation formula is as follows:

[0111]

[0112] Here, s refers to the target scale, g refers to the bounding box height, and k refers to the bounding box width.

[0113] Interpretably, bounding box width refers to the width of the target bounding box, bounding box height refers to the height of the target bounding box, and target ratio refers to the ratio of the bounding box height to the bounding box width of the target bounding box.

[0114] S4. Perform target determination based on the target area and target ratio to obtain a target determination result, wherein the target determination result is a target to be verified or a target not to be verified.

[0115] It can be explained that target judgment based on the target area and target ratio refers to judging whether the target area and target ratio are within the area range and ratio range respectively. If the target area is within the area range and the target ratio is within the ratio range, the target judgment result is confirmed as a target to be tested. If the target area is not within the area range or the target ratio is not within the ratio range, the target judgment result is confirmed as not a target to be tested. The area range and ratio range are both artificially set, and are used to determine whether the detection target is a human range. Optionally, the ratio range is greater than or equal to 1.5 and less than or equal to 4.0.

[0116] S5. If the target determination result is not the target to be verified, return to the above step of using the pre-built surveillance camera to obtain the regional surveillance video set until the target determination result is the target to be verified.

[0117] S6. If the target determination result is a target to be tested, a preset target following algorithm is used to continuously collect data on the detection target to obtain a detection time set and a detection center set, wherein the detection time in the detection time set corresponds one-to-one to the detection center in the detection center set.

[0118] Explainably, the target following algorithm refers to an algorithm that continuously tracks the detected target. Optionally, the target following algorithm is the DeepSORT algorithm. The detection time set refers to the set of detection times within the tracking time. The detection center set refers to the set of detection centers within the tracking time. The tracking time refers to the manually set duration for continuously tracking the detected target. The detection time refers to the time at which the detected target is recorded. The detection time starts at 0.0s. The detection center refers to the geometric center of the detected target. The detection center of the detected target is recorded once every time interval. The time interval is determined by the video frame rate of the area surveillance video. For example, if the video frame rate of the area surveillance video is 30fps, the time interval is 1 / 30s.

[0119] S7. Calculate the motion speed based on the detection time set and the detection center set, and set a decision node, wherein the decision node includes: a face occlusion node, an abnormal concealment node, an abnormal entry node, an abnormal stay node, an abnormal speed node, and an alarm triggering node.

[0120] In detail, the calculating of the motion speed according to the detection time set and the detection center set includes:

[0121] Extracting detection centers from the detection center set in sequence, obtaining detection center coordinates based on the extracted detection centers, and collecting the detection center coordinates to obtain a detection center coordinate set;

[0122] The last detection time is extracted from the detection time set, and the initial center coordinates and the final center coordinates are extracted from the detection center coordinate set. The detection displacement is calculated based on the last detection time, the initial center coordinates, and the final center coordinates. The initial center coordinates include: the initial center horizontal coordinate and the initial center vertical coordinate, and the final center coordinates include: the final center horizontal coordinate and the final center vertical coordinate. The calculation formula of the detection displacement is as follows:

[0123]

[0124] Where β refers to the detection displacement, f refers to the last detection time, x1 refers to the last center horizontal coordinate, x2 refers to the initial center horizontal coordinate, y1 refers to the last center vertical coordinate, and y2 refers to the initial center vertical coordinate.

[0125] It can be explained that the detection center coordinates refer to the coordinates of the detection center in the boundary detection coordinate system, the detection center coordinate set refers to the set composed of the detection center coordinates, the initial center coordinates refer to the detection center coordinates corresponding to the initial detection time, the initial detection time refers to the minimum detection time in the detection time set, the last center coordinates refer to the detection center coordinates corresponding to the last detection time, the last detection time refers to the maximum detection time in the detection time set, the detection displacement refers to the displacement of the detection target during the tracking time, the initial center horizontal coordinate refers to the horizontal coordinate of the initial center coordinate, the initial center vertical coordinate refers to the vertical coordinate of the initial center coordinate, the last center horizontal coordinate refers to the horizontal coordinate of the last center coordinate, and the last center vertical coordinate refers to the vertical coordinate of the last center coordinate.

[0126] In detail, the setting of the decision node includes:

[0127] Construct a decision node determination method, wherein the decision node determination method includes: a face occlusion method, an abnormality concealment method, an abnormal entry method, a stay abnormality method, a speed abnormality method, and an alarm triggering method. The abnormality concealment method is as follows:

[0128] Get target detection frame;

[0129] Continuous detection is performed based on a preset number of continuous detection frames and target detection frames to obtain a continuous image set;

[0130] Extracting continuous images from the continuous image set in sequence, and determining whether the detection target exists in the extracted continuous images;

[0131] If there is a detection target in the continuous image, the target detection frame is reset according to the number of continuous detection frames to obtain a reset detection frame, the reset detection frame is used to update the target detection frame, and the updated target detection frame is used to return to the above step of continuous detection based on the preset number of continuous detection frames and target detection frames;

[0132] If the detection target does not exist in the continuous image, the detection target is marked as an abnormal target;

[0133] Obtain training data, and use the training data to train a pre-built segmentation learning model to obtain a training learning model;

[0134] Decision nodes are set according to the decision node determination method, training learning model and abnormal targets.

[0135] Explainably, the decision node determination method refers to a method for determining face occlusion nodes, abnormal masking nodes, abnormal entry nodes, abnormal stay nodes, abnormal speed nodes, and alarm triggering nodes. After the determination is passed, the detection target will be marked as an abnormal target. The face occlusion method refers to a method for determining whether the detection target meets the face occlusion node, the abnormal masking method refers to a method for determining whether the detection target meets the abnormal masking node, and the abnormal entry method refers to a method for determining whether the detection target meets the abnormal masking node. The abnormal entry method includes: when the detection target enters the prohibited entry area without passing the formal detection procedure, the detection target is marked as an abnormal target. Optionally, the formal detection procedure is access control card swiping. The abnormal stay method refers to a method for determining whether the detection target meets the abnormal stay node. The abnormal stay method includes: obtaining an area residence time set, obtaining a target residence area according to the detection target, matching the corresponding time from the area residence time set according to the target residence area, recording the continuous residence time of the detection target, and marking the detection target as an abnormal target when the continuous residence time is greater than or equal to twice the corresponding time.

[0136] It should be explained that the regional dwell time set refers to the set of regional dwell times, the regional dwell time refers to the average dwell time of all personnel within the divided area over the past month, the target dwell area refers to the divided area where the detection target exists, the corresponding time refers to the time spent in the corresponding target dwell area within the regional dwell time set, and the continuous dwell time refers to the time the detection target stays in the target dwell area. The speed anomaly method refers to the method used to determine whether the detection target meets the speed anomaly node. The speed anomaly method includes: obtaining the current speed of the detection target, and when the current speed is greater than or equal to twice the normal speed, marking the detection target as an abnormal target.

[0137] Optionally, the normal speed is 1.5m / s. The alarm triggering method refers to the method used to trigger the alarm, which is obtained through the target decision map. The target detection frame refers to a frame of image when the target appears. Optionally, the number of continuous detection frames is 10 frames. The continuous image set refers to a set of continuous images obtained by continuous detection frames. For example, the target detection frame is the first frame, and the number of continuous detection frames is 10 frames, then the continuous image set is a set of images from the first frame to the eleventh frame. Training data refers to data used to train the segmentation learning model. The segmentation learning model refers to a model used to obtain a pixel occlusion probability set. Optionally, the segmentation learning model is a Mask R-CNN model. The training learning model refers to the model obtained after training the segmentation learning model. The decision node refers to the node for constructing the target decision map.

[0138] In detail, the obtaining of training data includes:

[0139] Obtain a historical surveillance video set, extract historical surveillance videos from the historical surveillance video set in sequence, extract facial image frame sequences from the extracted historical surveillance videos, and aggregate the facial image frame sequences to obtain a facial image frame sequence set;

[0140] Setting a facial annotation and an occlusion annotation, and annotating the facial image frame sequence set using the facial annotation to obtain an annotated image frame sequence set, wherein the facial annotation is 1 or 0;

[0141] Acquire an occluded image frame set from the annotated image frame sequence set, wherein the occluded image frame set consists of annotated image frames whose face is annotated as 1 in the annotated image frame sequence set;

[0142] Using the occlusion annotation to mark the occlusion image frame set, and collecting the occlusion image frames with the same occlusion annotation in the occlusion image frame set to obtain an object occlusion image set and a body occlusion image set;

[0143] The labeled image frame sequence set, the object occlusion image set, and the body occlusion image set are identified as training data.

[0144] It can be explained that the historical surveillance video set refers to the set of historical surveillance videos, and historical surveillance videos refer to surveillance videos within the past month. For example, if the historical surveillance video set is obtained on April 16, then the historical surveillance video set is all surveillance videos within March. The facial image frame sequence refers to the sequence of images corresponding to facial image frames, and the facial image frame refers to the frame corresponding to the image containing the face. The facial image frame sequence set refers to the set of facial image frame sequences. The facial annotation refers to the parameter used to mark the facial image frame sequence set. When the facial annotation is 1, it means that the face is occluded. When the facial annotation is 0, it means that the face is not occluded. The occlusion annotation refers to the parameter used to mark the occlusion image frame set. The occlusion annotation is 3 or 4. When the occlusion annotation is 3, it means that the image corresponding to the occluded image frame is an object occlusion image. When the occlusion annotation is 4, it means that the image corresponding to the occluded image frame is a body occlusion image. The "annotated image frame sequence set" refers to the set obtained by annotating the facial image frame sequence set using facial annotation. The "object-occluded image set" refers to the set consisting of object-occluded images, where an object-occluded image is an image in which the face is obscured by an external object. An external object refers to an object other than the body, such as a mask. The "body-occluded image set" refers to the set consisting of body-occluded images, where an image in which the face is obscured by the body, such as an arm.

[0145] In detail, the face occlusion method is as follows:

[0146] Obtain a target facial image, perform facial detection on the target facial image using a preset face detection algorithm, and obtain a facial bounding box;

[0147] Segmenting the target facial image based on the facial bounding box to obtain a facial image;

[0148] Calculating the actual facial area based on the facial image, and performing forward propagation on the facial image using the trained learning model to obtain a pixel occlusion probability set, wherein each pixel in the facial image corresponds to a pixel occlusion probability;

[0149] Extracting pixel occlusion probabilities from the pixel occlusion probability set in sequence, and comparing a preset demarcation threshold with the pixel occlusion probability;

[0150] If it is confirmed that the pixel occlusion probability is greater than or equal to the demarcation threshold, the pixel point corresponding to the pixel occlusion probability is marked as an occluded pixel point, and the occluded pixel points are collected to obtain an occluded pixel point set; the occluded pixel point set is weighted to obtain a first pixel point set and a second pixel point set;

[0151] The first pixel number and the second pixel number are counted according to the first pixel point set and the second pixel point set, and the facial occlusion rate is calculated using the actual facial area, the first pixel number and the second pixel number:

[0152]

[0153] Among them, j refers to the facial occlusion rate, Refers to the preset first weight, a dy Refers to the first pixel number, Refers to the preset second weight, a de Refers to the second pixel number, S am Refers to the actual facial area;

[0154] Comparing the facial occlusion rate with a preset occlusion rate threshold;

[0155] If it is confirmed that the facial occlusion rate is greater than the occlusion rate threshold, the target facial image is marked as an abnormal target.

[0156] In the interpretation, the target facial image refers to the image used to determine facial occlusion nodes, the face detection algorithm refers to the algorithm used to detect the target facial image, and optionally, the face detection algorithm is MTCNN. The facial bounding box refers to the bounding box output after detecting the target facial image using the face detection algorithm, the facial image refers to the image enclosed by the facial bounding box, the actual facial area refers to the total number of pixels contained in the facial image, and the pixel occlusion probability set refers to the set of values for each pixel in the facial image output by the trained learning model. This value ranges from 0 to 1 and is used to determine whether the pixel is occluded. The demarcation threshold refers to a manually set threshold used to determine the pixel occlusion probability. An occluded pixel refers to a pixel whose occlusion probability is greater than or equal to a predetermined threshold; an occluded pixel set refers to a set of occluded pixels; an open pixel refers to a pixel whose occlusion probability is less than a predetermined threshold; the first number of pixels refers to the number of pixels in the occluded pixel set; the second number of pixels refers to the number of open pixels; the facial occlusion rate is a value calculated based on the first number of pixels and the second number of pixels, and is used to reflect the occlusion situation of the facial image; the first weight refers to a parameter used to control the degree of influence of the first number of pixels on the facial occlusion rate; the second weight refers to a parameter used to control the degree of influence of the second number of pixels on the facial occlusion rate; the first occlusion rate has a greater influence on the facial occlusion rate than the second occlusion rate, so the first weight is set to be greater than the second weight; the occlusion rate threshold refers to an artificially set threshold for determining the facial occlusion rate.

[0157] In detail, the weighted partitioning of the occluded pixel point set to obtain the first pixel point set and the second pixel point set includes:

[0158] Obtaining facial reference points based on the facial image, and calculating a first facial segmentation area according to the actual facial area, wherein the first facial segmentation area is 66% of the actual facial area;

[0159] Segmenting the facial image using the facial reference points and the first facial segmentation area to obtain a first facial segmentation image and a second facial segmentation image;

[0160] The first face segmentation image is filtered from the occluded pixel point set to obtain a first pixel point set, and the second face segmentation image is filtered from the occluded pixel point set to obtain a second pixel point set.

[0161] To be explained, the facial reference point refers to the geometric center point of the facial image, the first facial segmentation image refers to the circular image on the facial image with the facial reference point as the origin and the first facial segmentation area as the area, and the second facial segmentation image refers to the image on the facial image outside the first facial segmentation image. The first pixel point set refers to the set consisting of the first pixel points, and the first pixel points refer to the pixels in the first facial segmentation image. The second pixel point set refers to the set consisting of the second pixel points, and the second pixel points refer to the pixels in the second facial segmentation image.

[0162] S8. Utilize the decision nodes to construct a target decision graph, determine the detection target according to the movement speed and the target decision graph, obtain graph determination data, and complete abnormal intrusion monitoring according to the graph determination data.

[0163] In detail, the use of the decision nodes to construct a target decision graph includes:

[0164] Construct an entry-stay edge from the abnormal entry node to the stay-abnormal node, and construct a stay-alarm edge from the stay-abnormal node to the alarm triggering node;

[0165] Construct an entry speed edge from the abnormal entry node to the speed abnormality node, and construct a speed alarm edge from the speed abnormality node to the alarm triggering node;

[0166] Constructing a masked face edge from the abnormal masking node to the face occlusion node, and constructing a face alarm edge from the face occlusion node to the alarm triggering node;

[0167] Constructing a mask-entry edge from the face occlusion node to the abnormal entry node, and constructing an entry-alarm edge from the abnormal entry node to the alarm trigger node;

[0168] Construct a stop-speed edge from the stop-abnormal node to the speed-abnormal node;

[0169] A target decision graph is constructed according to the entry and stop edges, stop warning edges, entry speed edges, speed warning edges, masked face edges, face warning edges, masked entry edges, entry warning edges, stop speed edges and decision nodes.

[0170] The target decision graph is an interpretable graph consisting of decision nodes and the directed edges between them. An entry-stop edge refers to a directed edge from an abnormal entry node to a speed abnormality node. For example, after an abnormal entry node marks a detected target as abnormal, the speed abnormality node is used to determine the detected target. A stop-alarm edge refers to a directed edge from an abnormal stop node to an alarm trigger node. An entry-speed edge refers to a directed edge from an abnormal entry node to a speed abnormality node. A speed alarm edge refers to a directed edge from a speed abnormality node to an alarm trigger node. A face-covering edge refers to a directed edge from an abnormal stop node to a face-occlusion node. A face-alarm edge refers to a directed edge from a face-occlusion node to an alarm trigger node. A face-covering edge refers to a directed edge from a face-occlusion node to an abnormal entry node. An entry-alarm edge refers to a directed edge from an abnormal entry node to an alarm trigger node. A stop-speed edge refers to a directed edge from an abnormal stop node to a speed abnormality node. The graph decision data is either te or fa. When the alarm trigger node is triggered, the graph decision data is fa. When the alarm trigger node is not triggered, the graph decision data is te.

[0171] In order to solve the problems described in the background technology, the present invention first obtains a regional monitoring video set, which includes monitoring videos of all divided areas in the monitoring area, providing an all-round field of view for subsequent detection, reducing the risk of missed detection, and ensuring the continuity and integrity of data sources through unified management and call, laying the foundation for real-time and rapid response; secondly, extracting regional monitoring videos in turn and performing target detection to obtain detection results, and traversing the regional monitoring videos of each divided area in order can ensure that each area is inspected without omission. The binary result of target detection simplifies the decision logic and helps to quickly determine whether to enter subsequent deep processing, which not only saves computing resources but also can quickly locate suspected intrusion activities. When the detection result is that the detection does not exist, the regional monitoring video set is re-acquired. When the detection result is that the detection exists, the corresponding video segment is immediately extracted for deep analysis to avoid wasting resources on irrelevant videos and improve the efficiency and accuracy of judgment; thirdly, the detection target is detected. The target area and target ratio are calculated, and the geometric features of the target are accurately extracted to assist in distinguishing different subjects and filter out small noise targets or false detections. The target area and target ratio are used to determine whether the target is to be tested or not. By judging the target area and target ratio, the targets that are not related to the intrusion are quickly eliminated, reducing the false alarm rate. Afterwards, if the target is determined to be a target to be tested, the target following algorithm is used to continuously collect data on the detected target. The target following algorithm can establish a stable spatiotemporal trajectory for a single target, record its detection center and detection time, and provide accurate continuous data for subsequent analysis. Finally, a decision node is set, and a target decision map is constructed using the decision node. By setting multi-dimensional decision nodes, multiple abnormal behaviors of the detected target can be monitored in parallel. The multi-node parallel design enhances the coverage of complex intrusion methods, avoids the failure of a single rule, and significantly improves the accuracy of abnormal intrusion monitoring. Therefore, the present invention can improve the accuracy of abnormal intrusion monitoring.

[0172] like Figure 2 , which is a functional module diagram of an abnormal intrusion monitoring system based on video stream analysis combined with the Internet of Things provided by one embodiment of the present invention.

[0173] The abnormal intrusion monitoring system 100 based on video stream analysis and the Internet of Things described in the present invention can be installed in an electronic device. Depending on the functionality implemented, the abnormal intrusion monitoring system 100 based on video stream analysis and the Internet of Things can include a detection target acquisition module 101, a judgment result acquisition module 102, a continuous data acquisition module 103, and a decision graph construction module 104. A module, also referred to as a unit, is a series of computer program segments that can be executed by an electronic device processor and perform a fixed function. These are stored in the electronic device's memory.

[0174] The detection target acquisition module 101 is used to obtain a regional surveillance video set, sequentially extract regional surveillance videos from the regional surveillance video set, and perform target detection based on the extracted regional surveillance videos to obtain a detection result, wherein the detection result is detection presence or detection absence;

[0175] If the detection result is that the detection does not exist, return to the above step of obtaining the regional surveillance video set until the detection result is that the detection exists;

[0176] If the detection result is existence, the detection target is obtained based on the detection result and the regional surveillance video corresponding to the detection result;

[0177] The determination result acquisition module 102 is used to analyze the detected target, obtain the target bounding box feature coordinates, and calculate the target area and target ratio based on the target bounding box feature coordinates;

[0178] Performing target determination based on the target area and target ratio to obtain a target determination result, wherein the target determination result is a target to be verified or a non-target to be verified;

[0179] If the target determination result is not the target to be verified, return to the above step of using the pre-built surveillance camera to obtain the regional surveillance video set until the target determination result is the target to be verified;

[0180] If the target is determined to be a target to be tested, a preset target following algorithm is used to continuously collect data on the target to obtain a detection time set and a detection center set, wherein the detection time in the detection time set corresponds to the detection center in the detection center set one-to-one;

[0181] The continuous data acquisition module 103 is used to calculate the motion speed based on the detection time set and the detection center set, and set decision nodes, wherein the decision nodes include: face occlusion node, abnormal cover node, abnormal entry node, abnormal stay node, abnormal speed node and alarm trigger node;

[0182] The decision graph construction module 104 is used to construct a target decision graph using the decision nodes, determine the detection target according to the movement speed and the target decision graph, obtain graph determination data, and complete abnormal intrusion monitoring based on the graph determination data.

[0183] In detail, the modules in the abnormal intrusion monitoring system 100 based on video stream analysis combined with the Internet of Things in the embodiment of the present invention are used in the same manner as above. Figure 1 The technical means described in the abnormal intrusion monitoring method based on video stream analysis combined with the Internet of Things are the same and can produce the same technical effects, so they will not be repeated here.

[0184] like Figure 3, which is a structural diagram of an electronic device for implementing an abnormal intrusion monitoring method based on video stream analysis combined with the Internet of Things, provided by one embodiment of the present invention.

[0185] The electronic device 1 may include a processor 10, a memory 11 and a bus 12, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as an abnormal intrusion monitoring method program based on video stream analysis combined with the Internet of Things.

[0186] The memory 11 includes at least one type of readable storage medium, including a flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of the electronic device 1, such as a mobile hard disk of the electronic device 1. In other embodiments, the memory 11 may also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 1. Furthermore, the memory 11 also includes an internal storage unit of the electronic device 1 and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device 1, such as the code of the abnormal intrusion monitoring method program based on video stream analysis combined with the Internet of Things, but can also be used to temporarily store data that has been output or is about to be output.

[0187] In some embodiments, the processor 10 may be composed of an integrated circuit, such as a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 10 is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and lines, and executing programs or modules stored in the memory 11 (such as an abnormal intrusion detection method program based on video stream analysis combined with the Internet of Things, etc.), as well as calling data stored in the memory 11, to perform various functions of the electronic device 1 and process data.

[0188] The bus 12 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 12 may be divided into an address bus, a data bus, a control bus, etc. The bus 12 is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0189] Figure 3 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the electronic device 1 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0190] For example, although not shown, the electronic device 1 may further include a power source (such as a battery) for powering the various components. Preferably, the power source may be logically connected to the at least one processor 10 via a power management device, thereby implementing functions such as charging management, discharging management, and power consumption management through the power management device. The power source may further include any components such as one or more DC or AC power sources, a recharging device, a power failure detection circuit, a power converter or inverter, a power status indicator, etc. The electronic device 1 may further include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0191] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device 1 and other electronic devices.

[0192] Optionally, the electronic device 1 may further include a user interface, which may be a display or an input unit (such as a keyboard). Optionally, the user interface may also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touch device. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visual user interface.

[0193] The abnormal intrusion monitoring method program based on video stream analysis combined with the Internet of Things stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When running in the processor 10, it can achieve the following:

[0194] Obtaining a regional surveillance video set, sequentially extracting regional surveillance videos from the regional surveillance video set, and performing target detection based on the extracted regional surveillance videos to obtain a detection result, wherein the detection result is detection presence or detection absence;

[0195] If the detection result is that the detection does not exist, return to the above step of obtaining the regional surveillance video set until the detection result is that the detection exists;

[0196] If the detection result is existence, the detection target is obtained based on the detection result and the regional surveillance video corresponding to the detection result;

[0197] Analyze the detected target to obtain the target bounding box feature coordinates, and calculate the target area and target ratio based on the target bounding box feature coordinates;

[0198] Performing target determination based on the target area and target ratio to obtain a target determination result, wherein the target determination result is a target to be verified or a non-target to be verified;

[0199] If the target determination result is not the target to be verified, return to the above step of using the pre-built surveillance camera to obtain the regional surveillance video set until the target determination result is the target to be verified;

[0200] If the target is determined to be a target to be tested, a preset target following algorithm is used to continuously collect data on the target to obtain a detection time set and a detection center set, wherein the detection time in the detection time set corresponds to the detection center in the detection center set one-to-one;

[0201] Calculate the motion speed according to the detection time set and the detection center set, and set a decision node, wherein the decision node includes: a face occlusion node, an abnormal cover node, an abnormal entry node, an abnormal stay node, an abnormal speed node, and an alarm triggering node;

[0202] The decision nodes are used to construct a target decision map, and the detection target is judged according to the movement speed and the target decision map to obtain map judgment data, and abnormal intrusion monitoring is completed according to the map judgment data.

[0203] Specifically, the specific implementation method of the processor 10 for the above instructions can refer to Figures 1 to 3 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0204] Furthermore, if the modules / units integrated into the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0205] The present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed by a processor of an electronic device, the computer program can implement:

[0206] Obtaining a regional surveillance video set, sequentially extracting regional surveillance videos from the regional surveillance video set, and performing target detection based on the extracted regional surveillance videos to obtain a detection result, wherein the detection result is detection presence or detection absence;

[0207] If the detection result is that the detection does not exist, return to the above step of obtaining the regional surveillance video set until the detection result is that the detection exists;

[0208] If the detection result is existence, the detection target is obtained based on the detection result and the regional surveillance video corresponding to the detection result;

[0209] Analyze the detected target to obtain the target bounding box feature coordinates, and calculate the target area and target ratio based on the target bounding box feature coordinates;

[0210] Performing target determination based on the target area and target ratio to obtain a target determination result, wherein the target determination result is a target to be verified or a non-target to be verified;

[0211] If the target determination result is not the target to be verified, return to the above step of using the pre-built surveillance camera to obtain the regional surveillance video set until the target determination result is the target to be verified;

[0212] If the target is determined to be a target to be tested, a preset target following algorithm is used to continuously collect data on the target to obtain a detection time set and a detection center set, wherein the detection time in the detection time set corresponds to the detection center in the detection center set one-to-one;

[0213] Calculate the motion speed according to the detection time set and the detection center set, and set a decision node, wherein the decision node includes: a face occlusion node, an abnormal cover node, an abnormal entry node, an abnormal stay node, an abnormal speed node, and an alarm triggering node;

[0214] The decision nodes are used to construct a target decision map, and the detection target is judged according to the movement speed and the target decision map to obtain map judgment data, and abnormal intrusion monitoring is completed according to the map judgment data.

[0215] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, systems and methods can be implemented in other ways. For example, the system embodiments described above are only exemplary, and actual implementations may have other division methods.

[0216] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected to achieve the purpose of the solution of this embodiment according to actual needs.

[0217] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or hardware plus software functional modules.

[0218] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0219] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. An abnormal intrusion monitoring method based on video stream analysis combined with the Internet of Things is characterized by: The method comprises: Obtaining a regional surveillance video set, sequentially extracting regional surveillance videos from the regional surveillance video set, and performing target detection based on the extracted regional surveillance videos to obtain a detection result, wherein the detection result is detection presence or detection absence; If the detection result is that the detection does not exist, return to the above step of obtaining the regional surveillance video set until the detection result is that the detection exists; If the detection result is existence, the detection target is obtained based on the detection result and the regional surveillance video corresponding to the detection result; Analyze the detected target to obtain the target bounding box feature coordinates, and calculate the target area and target ratio based on the target bounding box feature coordinates; Performing target determination based on the target area and target ratio to obtain a target determination result, wherein the target determination result is a target to be verified or a non-target to be verified; If the target determination result is not the target to be verified, return to the above step of using the pre-built surveillance camera to obtain the regional surveillance video set until the target determination result is the target to be verified; If the target is determined to be a target to be tested, a preset target following algorithm is used to continuously collect data on the target to obtain a detection time set and a detection center set, wherein the detection time in the detection time set corresponds to the detection center in the detection center set one-to-one; Calculate the motion speed according to the detection time set and the detection center set, and set a decision node, wherein the decision node includes: a face occlusion node, an abnormal cover node, an abnormal entry node, an abnormal stay node, an abnormal speed node, and an alarm triggering node; The decision nodes are used to construct a target decision map, and the detection target is judged according to the movement speed and the target decision map to obtain map judgment data, and abnormal intrusion monitoring is completed according to the map judgment data.

2. The abnormal intrusion monitoring method based on video stream analysis combined with the Internet of Things according to claim 1 is characterized in that: Analyzing the detected target to obtain the target bounding box feature coordinates includes: Acquire a detection image according to the detection target, extract a reference point from the detection image, and establish a boundary detection coordinate system with the reference point as the origin, wherein the reference point is the upper left corner vertex of the detection image; Detect the detection target using a preset target detection algorithm and boundary detection coordinate system to obtain a detection bounding box set, a bounding box starting point set, and a detection confidence set, wherein the detection bounding boxes in the detection bounding box set correspond one-to-one to the bounding box starting points in the bounding box starting point set, and the bounding box starting points in the bounding box starting point set correspond one-to-one to the detection confidences in the detection confidence set; Extracting detection confidences from the detection confidence set in sequence, and comparing the detection confidences with a preset confidence threshold; If the detection confidence is less than the confidence threshold, return to the above step of detecting the target using the preset target detection algorithm until the detection confidence is greater than or equal to the confidence threshold; If the detection confidence is not uniformly less than the confidence threshold, the detection confidences that are greater than or equal to the confidence threshold are aggregated to obtain the judgment confidence set; The judgment confidence set is screened to obtain the target confidence, and the target bounding box is obtained from the detection bounding box set according to the target confidence. The bounding box starting point is matched from the bounding box starting point set according to the target bounding box, and the starting point coordinates are obtained according to the bounding box starting point, and the starting point coordinates are confirmed as the target bounding box feature coordinates.

3. The abnormal intrusion monitoring method based on video stream analysis combined with the Internet of Things as claimed in claim 2 is characterized in that: Calculating the target area and target ratio based on the target bounding box feature coordinates includes: Obtaining a bounding box width and a bounding box height according to the target bounding box and the target bounding box feature coordinates; Calculate the target area using the bounding box width and the bounding box height, where the target area is the product of the bounding box width and the bounding box height; The target ratio is calculated based on the bounding box width and bounding box height. The calculation formula is as follows: Here, s refers to the target scale, g refers to the bounding box height, and k refers to the bounding box width.

4. The abnormal intrusion monitoring method based on video stream analysis combined with the Internet of Things as claimed in claim 3 is characterized in that: The calculating of the motion speed according to the detection time set and the detection center set includes: Extracting detection centers from the detection center set in sequence, obtaining detection center coordinates based on the extracted detection centers, and collecting the detection center coordinates to obtain a detection center coordinate set; The last detection time is extracted from the detection time set, and the initial center coordinates and the final center coordinates are extracted from the detection center coordinate set. The detection displacement is calculated based on the last detection time, the initial center coordinates, and the final center coordinates. The initial center coordinates include: the initial center horizontal coordinate and the initial center vertical coordinate, and the final center coordinates include: the final center horizontal coordinate and the final center vertical coordinate. The calculation formula of the detection displacement is as follows: Where β refers to the detection displacement, f refers to the last detection time, x1 refers to the last center horizontal coordinate, x2 refers to the initial center horizontal coordinate, y1 refers to the last center vertical coordinate, and y2 refers to the initial center vertical coordinate.

5. The abnormal intrusion monitoring method based on video stream analysis combined with the Internet of Things as claimed in claim 4 is characterized in that: The setting decision node includes: Construct a decision node determination method, wherein the decision node determination method includes: a face occlusion method, an abnormality concealment method, an abnormal entry method, a stay abnormality method, a speed abnormality method, and an alarm triggering method. The abnormality concealment method is as follows: Get target detection frame; Continuous detection is performed based on a preset number of continuous detection frames and target detection frames to obtain a continuous image set; Extracting continuous images from the continuous image set in sequence, and determining whether the detection target exists in the extracted continuous images; If there is a detection target in the continuous image, the target detection frame is reset according to the number of continuous detection frames to obtain a reset detection frame, the reset detection frame is used to update the target detection frame, and the updated target detection frame is used to return to the above step of continuous detection based on the preset number of continuous detection frames and target detection frames; If the detection target does not exist in the continuous image, the detection target is marked as an abnormal target; Obtain training data, and use the training data to train a pre-built segmentation learning model to obtain a training learning model; Decision nodes are set according to the decision node determination method, training learning model and abnormal targets.

6. The abnormal intrusion monitoring method based on video stream analysis combined with the Internet of Things according to claim 5 is characterized in that: The obtaining of training data includes: Obtain a historical surveillance video set, extract historical surveillance videos from the historical surveillance video set in sequence, extract facial image frame sequences from the extracted historical surveillance videos, and aggregate the facial image frame sequences to obtain a facial image frame sequence set; Setting a facial annotation and an occlusion annotation, and annotating the facial image frame sequence set using the facial annotation to obtain an annotated image frame sequence set, wherein the facial annotation is 1 or 0; Acquire an occluded image frame set from the annotated image frame sequence set, wherein the occluded image frame set consists of annotated image frames whose face is annotated as 1 in the annotated image frame sequence set; Using the occlusion annotation to mark the occlusion image frame set, and collecting the occlusion image frames with the same occlusion annotation in the occlusion image frame set to obtain an object occlusion image set and a body occlusion image set; The labeled image frame sequence set, the object occlusion image set, and the body occlusion image set are identified as training data.

7. The abnormal intrusion monitoring method based on video stream analysis combined with the Internet of Things according to claim 6 is characterized in that: The face occlusion method is as follows: Obtain a target facial image, perform facial detection on the target facial image using a preset face detection algorithm, and obtain a facial bounding box; Segmenting the target facial image based on the facial bounding box to obtain a facial image; Calculating the actual facial area based on the facial image, and performing forward propagation on the facial image using the trained learning model to obtain a pixel occlusion probability set, wherein each pixel in the facial image corresponds to a pixel occlusion probability; Extracting pixel occlusion probabilities from the pixel occlusion probability set in sequence, and comparing a preset demarcation threshold with the pixel occlusion probability; If it is confirmed that the pixel occlusion probability is greater than or equal to the demarcation threshold, the pixel point corresponding to the pixel occlusion probability is marked as an occluded pixel point, and the occluded pixel points are collected to obtain an occluded pixel point set; the occluded pixel point set is weighted to obtain a first pixel point set and a second pixel point set; The first pixel number and the second pixel number are counted according to the first pixel point set and the second pixel point set, and the facial occlusion rate is calculated using the actual facial area, the first pixel number and the second pixel number: Among them, j refers to the facial occlusion rate, Refers to the preset first weight, a dy Refers to the first pixel number, Refers to the preset second weight, a de Refers to the second pixel number, S am Refers to the actual facial area; Comparing the facial occlusion rate with a preset occlusion rate threshold; If it is confirmed that the facial occlusion rate is greater than the occlusion rate threshold, the target facial image is marked as an abnormal target.

8. The abnormal intrusion monitoring method based on video stream analysis combined with the Internet of Things according to claim 7 is characterized in that: The weighted division of the occluded pixel point set to obtain the first pixel point set and the second pixel point set includes: Obtaining facial reference points based on the facial image, and calculating a first facial segmentation area according to the actual facial area, wherein the first facial segmentation area is 66% of the actual facial area; Segmenting the facial image using the facial reference points and the first facial segmentation area to obtain a first facial segmentation image and a second facial segmentation image; The first face segmentation image is filtered from the occluded pixel point set to obtain a first pixel point set, and the second face segmentation image is filtered from the occluded pixel point set to obtain a second pixel point set.

9. The abnormal intrusion monitoring method based on video stream analysis combined with the Internet of Things according to claim 8, characterized in that: The method of constructing a target decision graph using the decision nodes includes: Construct an entry-stay edge from the abnormal entry node to the stay-abnormal node, and construct a stay-alarm edge from the stay-abnormal node to the alarm triggering node; Construct an entry speed edge from the abnormal entry node to the speed abnormality node, and construct a speed alarm edge from the speed abnormality node to the alarm triggering node; Constructing a masked face edge from the abnormal masking node to the face occlusion node, and constructing a face alarm edge from the face occlusion node to the alarm triggering node; Constructing a mask-entry edge from the face occlusion node to the abnormal entry node, and constructing an entry-alarm edge from the abnormal entry node to the alarm trigger node; Construct a stop-speed edge from the stop-abnormal node to the speed-abnormal node; A target decision graph is constructed according to the entry and stop edges, stop warning edges, entry speed edges, speed warning edges, masked face edges, face warning edges, masked entry edges, entry warning edges, stop speed edges and decision nodes.

10. The abnormal intrusion monitoring system based on video stream analysis combined with the Internet of Things is characterized by: The system comprises: A detection target acquisition module is used to obtain a regional surveillance video set, sequentially extract regional surveillance videos from the regional surveillance video set, and perform target detection based on the extracted regional surveillance videos to obtain a detection result, wherein the detection result is detection presence or detection absence; If the detection result is that the detection does not exist, return to the above step of obtaining the regional surveillance video set until the detection result is that the detection exists; If the detection result is existence, the detection target is obtained based on the detection result and the regional surveillance video corresponding to the detection result; The determination result acquisition module is used to analyze the detected target, obtain the target bounding box feature coordinates, and calculate the target area and target ratio based on the target bounding box feature coordinates; Performing target determination based on the target area and target ratio to obtain a target determination result, wherein the target determination result is a target to be verified or a non-target to be verified; If the target determination result is not the target to be verified, return to the above step of using the pre-built surveillance camera to obtain the regional surveillance video set until the target determination result is the target to be verified; If the target is determined to be a target to be tested, a preset target following algorithm is used to continuously collect data on the target to obtain a detection time set and a detection center set, wherein the detection time in the detection time set corresponds to the detection center in the detection center set one-to-one; a continuous data acquisition module, configured to calculate the motion speed based on the detection time set and the detection center set, and set decision nodes, wherein the decision nodes include: a face occlusion node, an abnormal occlusion node, an abnormal entry node, an abnormal stay node, an abnormal speed node, and an alarm triggering node; The decision map construction module is used to use the decision nodes to construct a target decision map, determine the detection target according to the movement speed and the target decision map, obtain map determination data, and complete abnormal intrusion monitoring according to the map determination data.

Images