A real-time integrated multi-video stream multi-level analysis and monitoring system

Abstract

The present invention provides a real-time integrated multi-video stream multi-level analysis and monitoring system, which relates to the technical field of video stream monitoring and distribution. The present invention introduces a warning area demarcation module and a coupling analysis module, enabling the system to focus on monitoring and analyzing key areas, thereby improving the accuracy and efficiency of anomaly detection. Secondly, the Kalman filter algorithm is used to construct a time series model to achieve effective coupling analysis of micro-motion characteristics and background changes, thereby enabling more accurate assessment of target abnormal behavior. Finally, through a multi-level priority alarm mechanism, the system can automatically adjust the alarm strategy according to different abnormal behaviors, thereby improving the intelligence level of the monitoring system.

Description

Technical Field

[0001] The present invention relates to the technical field of video stream monitoring and distribution, and in particular to a real-time integrated multi-video stream multi-level analysis and monitoring system. Background Art

[0002] In actual security or behavior monitoring scenarios, key locations are often framed within surveillance video to create "warning zones" (e.g., warehouse entrances, specific passages, restricted areas, etc.). When a person or object exhibits abnormal behavior (e.g., unusual movement, suspicious gestures, etc.) within this warning zone, the system must be able to immediately identify it and automatically highlight it within the massive real-time video stream or recommend it to human operators, thereby improving processing efficiency.

[0003] In the prior art, the publication number is CN114357243A, and the name is a multi-level analysis and monitoring system for massive real-time video streams. The solution processes massive videos in a hierarchical manner. First, two low-complexity algorithms based on video image motion characteristics and a low-cost embedded edge computing terminal with video hard decoding are used to perform two-level screening of a large number of videos; then, the artificial intelligence-based YOLO algorithm and a GPU-equipped server are used to perform target detection and recommended display on the few screened videos. Compared with a system that uses the same artificial intelligence analysis algorithm for all videos, it saves more than 90% of computing power configuration and hardware costs, making the investment required for real-time analysis and monitoring of all videos through intelligent analysis within the user's affordable range.

[0004] However, in scenarios with complex lighting, screen switching, or significant background changes (such as frequent light flickering, dramatic changes in light outside a window, and frequent content switching on large LED screens), relying solely on traditional object detection algorithms is often prone to false or missed detections. Therefore, the technical challenge that needs to be addressed is how to couple "target micro-motion characteristics" with "background motion field stability" to improve detection accuracy in complex environments by adaptively adjusting algorithm thresholds or integrating multi-frame timing information.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the Invention

[0006] The object of the present invention is to provide a real-time integrated multi-video stream multi-level analysis and monitoring system to solve the problems raised in the above background technology.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A real-time integrated multi-video stream multi-level analysis and monitoring system, specifically comprising:

[0009] Warning area demarcation module: used to predetermine at least one warning area in the surveillance video image, record the coordinate information or mask information of the warning area, and match each warning area with its associated impact area;

[0010] The affected area refers to a specific range outside the warning area, which involves the safety of monitored equipment or personnel;

[0011] Target detection and micro-motion feature extraction module: used to detect the target in the warning area and obtain key point information of the detected target in the warning area;

[0012] Input the acquired key point information into a micro-motion analysis module, which includes a frequency domain analysis algorithm;

[0013] By using a sliding time window and frequency domain analysis algorithm, the micro-motion of the detected target is analyzed to extract its main frequency and amplitude.

[0014] Combine the extracted micro-motion main frequency and amplitude to generate the micro-motion characteristic index of the detected target;

[0015] Background motion field stability index generation module: used to perform background modeling on the warning area and its surrounding areas to obtain a background model;

[0016] Continuously calculate the likelihood value fluctuation amplitude between the current frame and the background model;

[0017] By calculating the standard deviation of the likelihood value fluctuation amplitude, the background variation amplitude is measured to obtain the stability index of the background motion field;

[0018] Micro-motion and background field index coupling evaluation module: used for performing nonlinear coupling analysis on the generated micro-motion characteristic index and the generated stability index;

[0019] The coupling analysis uses Kalman filtering to construct a time series model to obtain a comprehensive evaluation index for evaluating the abnormal suspicion of the detected target, and the comprehensive evaluation index is used to provide a first abnormal behavior evaluation result for the warning area;

[0020] Impact area status evaluation module: used to collect real-time status data of the impact area associated with each warning area, analyze and process these real-time status data, and generate a regional status evaluation index. The regional status evaluation index is used to provide a second abnormal behavior evaluation result for the impact area;

[0021] Abnormal detection and alarm triggering module: Based on the first abnormal behavior evaluation result of the warning area, the abnormal behavior of the detected target in the warning area is prioritized; then, based on the second abnormal behavior evaluation result of the affected area, the corresponding level of alarm mechanism is triggered under different priority division results, and the monitoring video channel is pushed to the monitoring center or automatically highlighted in the interface.

[0022] Compared with the existing technology, the beneficial effects of the present invention are: by introducing the warning area demarcation module and the coupling analysis module, the system can focus on monitoring and analyzing key areas, thereby improving the accuracy and efficiency of anomaly detection; secondly, the Kalman filter algorithm is used to construct a time series model to achieve effective coupling analysis of micro-motion characteristics and background changes, thereby being able to more accurately evaluate the abnormal behavior of the detected target; finally, through the multi-level priority alarm mechanism, the system can automatically adjust the alarm strategy according to different abnormal behaviors, thereby improving the intelligence level of the monitoring system. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the overall system module flow of the present invention. DETAILED DESCRIPTION

[0024] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to specific embodiments.

[0025] It should be noted that, unless otherwise defined, the technical or scientific terms used in the present invention should have the usual meanings understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative position relationships. When the absolute position of the object being described changes, the relative position relationship may also change accordingly.

[0026] Example 1:

[0027] See also Figure 1 , the present invention provides a technical solution:

[0028] A real-time integrated multi-video stream multi-level analysis and monitoring system, specifically including:

[0029] Warning area demarcation module: used to predetermine at least one warning area in the surveillance video image, record the coordinate information or mask information of the warning area, and match each warning area with its associated impact area;

[0030] Target detection and micro-motion feature extraction module: used to detect the target in the warning area and obtain key point information of the detected target in the warning area;

[0031] Input the acquired key point information into a micro-motion analysis module, which includes a frequency domain analysis algorithm;

[0032] By using a sliding time window and frequency domain analysis algorithm, the micro-motion of the detected target is analyzed to extract its main frequency and amplitude.

[0033] Combine the extracted micro-motion main frequency and amplitude to generate the micro-motion characteristic index of the detected target ;

[0034] Background motion field stability index generation module: used to perform background modeling on the warning area and its surrounding areas to obtain a background model;

[0035] Continuously calculate the likelihood value fluctuation amplitude between the current frame and the background model;

[0036] By calculating the standard deviation of the likelihood value fluctuation amplitude, the background change amplitude is measured to obtain the stability index of the background motion field. ;

[0037] Micro-motion and background field index coupling evaluation module: used to generate the micro-motion feature index The stability index generated with Perform nonlinear coupling analysis;

[0038] The coupling analysis uses Kalman filtering to construct a time series model to obtain a comprehensive evaluation index for evaluating the abnormal suspicion of the detected target, and the comprehensive evaluation index is used to provide a first abnormal behavior evaluation result for the warning area;

[0039] Impact area status evaluation module: used to collect real-time status data of the impact area associated with each warning area, analyze and process these real-time status data, and generate a regional status evaluation index. The regional status evaluation index is used to provide a second abnormal behavior evaluation result for the impact area;

[0040] Abnormal detection and alarm triggering module: Based on the first abnormal behavior evaluation result of the warning area, the abnormal behavior of the detected target in the warning area is prioritized; then, based on the second abnormal behavior evaluation result of the affected area, the corresponding level of alarm mechanism is triggered under different priority division results, and the monitoring video channel is pushed to the monitoring center or automatically highlighted in the interface.

[0041] Further explanation: the warning area demarcation module specifically includes:

[0042] 1.1) Video input preprocessing:

[0043] Access multiple video streams and obtain real-time data through video acquisition devices;

[0044] Adopting a coding and decoding algorithm, this embodiment adopts the H.264 coding algorithm to perform real-time coding and decoding on the video stream to ensure synchronous processing;

[0045] H.264 encoding is a widely used video compression standard and the name of a video encoding algorithm. Its full name is Advanced Video Coding (AVC). The purpose of H.264 encoding is to significantly reduce the size of video files while maintaining high image quality, thereby improving transmission and storage efficiency. Due to its good compression efficiency and applicability, H.264 is widely used in various video applications, including video streaming, storage, and broadcasting.

[0046] Perform split processing on the encoded video frames to ensure that each video stream can be analyzed independently;

[0047] Use the video decoder FFmpeg to decode the compressed video stream and convert it into a processable frame format;

[0048] Create a video processing workflow that distributes each decoded video frame to a separate analysis pipeline.

[0049] Specifically, multi-threading or multi-processing is used to achieve parallel processing, and each thread or process is responsible for processing an independent video stream;

[0050] Deploy a separate video analysis module in each independent pipeline and perform motion detection and object recognition operations;

[0051] Ensure that analysis modules can run in parallel without interfering with each other, thus improving processing efficiency;

[0052] Use synchronization mechanisms to ensure the coordination of multiple video stream processing to avoid data competition and conflicts;

[0053] Integrate analysis results to generate unified reports or early warning information;

[0054] Integrate the results processed by each pipeline and output them to a centralized management platform.

[0055] 1.2) Video stream background modeling;

[0056] Use Gaussian mixture models to create dynamic background models for video streams;

[0057] Continuously update the background model to adapt to environmental changes and identify foreground objects and distinguish them from the background;

[0058] 1.3) Warning area selection:

[0059] Pre-select the warning areas that need to be monitored in the video stream to generate a warning area list, marked as {1,2,…,i,…,n}, where i represents the index of the warning area i; n represents the total number of warning areas;

[0060] 1.4) Coordinate and mask information recording:

[0061] Determine the boundaries of each warning area i, extract its coordinate information (x, y) and mask information for subsequent processing, and store these data in the spatial database;

[0062] The record format is: region index i->(coordinate information, mask information);

[0063] 1.5) Matching of warning area and impact area:

[0064] Match the information of each warning area i with the predefined impact area, and use the impact area of ​​warning area i as Indicates. The affected area refers to a specific range outside the warning area, which involves the safety of monitored equipment or personnel;

[0065] In this embodiment, the warning area is the transition area between the aircraft boarding bridge and the aircraft door interface. The impact area includes the impact area where boarding personnel can throw objects at the aircraft in the warning area. The impact area is bounded and delineated using the visual image acquired by the monitoring system.

[0066] A weighted association analysis algorithm was used to score the matching results based on geographical and functional relatedness;

[0067] Confirm and store the relationship between the warning area and the affected area to provide a basis for subsequent analysis.

[0068] Further explanation: the target detection and micro-motion feature extraction module specifically includes:

[0069] The key point information is set to include the small displacement of the arms and legs of the detected target within a preset unit time;

[0070] In the target detection phase of the warning area i, the skeleton point tracking algorithm is used to count the small displacements of the human arms and legs per unit time;

[0071] 2.1) Object Detection:

[0072] In warning area i, the trained YOLOv5 model is used to process the video stream in real time to detect all objects that appear. A convolutional neural network is used to analyze the pixels in warning area i, identify dynamic objects, and output the bounding box and category label of the detected object.

[0073] 2.2) Keypoint Tracking:

[0074] Once the detected objects are identified, the OpenPose algorithm is applied to track the skeleton key points of these detected objects. The specific operations include:

[0075] Apply pose estimation to the detected target and extract the coordinates of specific skeletal key points, which are arms and legs;

[0076] Record these data in real time for subsequent motion analysis;

[0077] The YOLOv5 model and the OpenPose algorithm are both advanced technical tools for computer vision tasks, each with unique capabilities and application scenarios.

[0078] Description of the YOLOv5 model:

[0079] The YOLOv5 model is a real-time object detection model that belongs to the YOLO (You Only Look Once) family. Its main features are fast speed and high accuracy, and it can complete object detection tasks in a single forward propagation of the convolutional neural network.

[0080] Used to identify and locate objects in images or video frames; YOLOv5 effectively balances detection accuracy and speed by processing the entire image once.

[0081] Real-time: Due to its efficient architecture design, YOLOv5 can achieve real-time target detection on resource-constrained devices.

[0082] Accuracy: The optimized model can recognize objects from multiple categories and output detection bounding boxes and confidence scores;

[0083] The YOLOv5 model uses the following principles:

[0084] Input processing: YOLOv5 divides the input image into SxS grids, and each grid is responsible for detecting the target in the area.

[0085] Convolution operation: Feature extraction of images through convolutional neural networks (CNN).

[0086] YOLOv5 extracts multi-level features based on the input image to adapt to the detection of targets of different sizes.

[0087] Bounding box prediction: Each grid predicts a fixed number of bounding box parameters (center point coordinates, width and height) and a confidence score that indicates the likelihood of the object existing in the predicted box and its accuracy.

[0088] Category prediction: At the same time, for each bounding box, the probability distribution of the category to which it belongs is also predicted.

[0089] Non-maximum suppression (NMS): Among all predictions, NMS is used to remove duplicate or overlapping predictions, retain the detection results with the highest scores, and improve prediction accuracy.

[0090] Description of the OpenPose algorithm:

[0091] The OpenPose algorithm is an open source posture detection library that focuses on detecting key points of the human body, hands, face, and feet. It can recognize human posture and movements in two-dimensional images.

[0092] Usage: Used for tracking and pose estimation of skeletal key points. OpenPose can detect the pose information of multiple detected targets at the same time.

[0093] Multi-target tracking: It can simultaneously detect and track multiple human bodies in an image or video, and output the spatial coordinates of each key point.

[0094] Rich applications: It can be used in many fields such as sports analysis, human-computer interaction, and motion recognition.

[0095] The OpenPose algorithm uses the following steps:

[0096] Input image: Receives a two-dimensional image or video frame from a video stream as input.

[0097] Feature extraction: Use convolutional neural networks to extract image features, significantly reducing the complexity of the image and retaining effective information.

[0098] Part Affinity Field (PAF): OpenPose builds a point-to-point connection field to show the relationship between different parts of the human body.

[0099] Heatmap generation: Generate a heatmap for each human keypoint to determine the relative position and pose of the keypoint.

[0100] Graph Convolutional Network (GCN): Combine PAF and heat map, use GCN to associate detected key points and reconstruct the human skeleton.

[0101] Multi-person detection: OpenPose uses a parsing algorithm to group detected key points to simultaneously identify multiple individuals in an image.

[0102] When used together, these two tools can be very effective in complex video analysis scenarios. In surveillance systems, object detection is performed using YOLOv5, followed by detailed pose analysis using OpenPose, for accurate behavior recognition and analysis.

[0103] Combined with empirical thresholds or machine learning training, an upper threshold for the small displacement of the arms and legs within a preset unit time is set, and micro-movements exceeding the corresponding upper threshold are marked as "suspicious small movements";

[0104] The first evaluation index of warning area i is defined as , the calculation formula is as follows:

[0105]

[0106] in, is the small displacement of the skeleton key point in the warning area i, is the upper threshold of the skeleton key point in the warning area i; It is the first evaluation index of the skeleton key point in the warning area i; GG represents SB and TB, SB and TB represent the arm and leg respectively; The selection is determined by the expert group through experimental data; The determination strategy is to imitate the detected target in the warning area i for the affected area When a suspicious small action is performed, the small displacement corresponding to the key point of the skeleton is ;

[0107] like When the micro-movement is detected, it is marked as a "suspicious micro-movement";

[0108] The small displacements of the arms and legs within a preset unit time are placed in a sliding time window. The frequency components and corresponding amplitudes of the "suspicious small movements" of the detected target are extracted using a frequency domain analysis algorithm such as Fourier transform or wavelet analysis.

[0109] Continuously collect the coordinates of the skeleton key points of each target in the warning area i and sample them at a fixed time interval to form a time series data set to reflect the spatial movement of the key points; in this embodiment, the time interval is 0.1 seconds;

[0110] Input the displacement data of key points into the frequency domain analysis module and process it using Fourier transform:

[0111] Perform fast Fourier transform on the collected time series data to extract the frequency components and corresponding amplitudes of the data;

[0112] The reason for choosing Fourier transform is its efficiency and accuracy, which can quickly extract rich frequency domain information;

[0113] Generate micro-motion feature indicators for each detected target based on the extracted frequency components and amplitude information ;

[0114] Combined with experience threshold or machine learning training, pre-set the micro-motion characteristic indicators of the detected target The comparison threshold, if within any time window T, If the comparison threshold is exceeded, it is considered suspicious behavior. The characterization formula is: ;in is the comparison threshold of warning area i; The determination strategy is to imitate the detected target in the warning area i for the affected area When engaging in suspicious behavior, micro-movement characteristic indicators The corresponding output value.

[0115] Micro-motion characteristic indicators The calculation formula is as follows:

[0116]

[0117] in, is the frequency component of the “suspicious small movement” in the warning area i. The frequency component represents the frequency value associated with the signal. is the amplitude of the frequency component; is the maximum displacement in the time window T; are weighting factors for frequency, amplitude, and displacement, The values ​​of are all in the interval (0,1), and . It needs to be determined through experiments or model training; this embodiment sets .

[0118] In the context of warning area i, the feature parameter set is analyzed using a preset comparison threshold or machine learning model:

[0119] The pre-set thresholds are developed based on historical data and field experiment results, specific to the possible behavior of the warning area i;

[0120] If within a certain time window, the generated If the characteristic exceeds the preset threshold, an alarm is triggered and the event is recorded as a potential suspicious behavior for subsequent investigation.

[0121] Further explanation: the background motion field stability index generation module specifically includes:

[0122] The background modeling is constructed using an algorithm based on the Gaussian Mixture Model (GMM). The specific modeling steps are as follows:

[0123] Data preprocessing: Use a low-pass filter to remove high-frequency noise from the input video stream and normalize the pixel intensity to the range of [0,1] to obtain smooth video data;

[0124] Background initialization: Extract the first N1 frames from the video and use the data of these frames to initialize the parameters of the Gaussian mixture model (GMM), including the mean, variance, and weight of each Gaussian component, thereby capturing the multimodal characteristics of the background;

[0125] The background model of each pixel is composed of K Gaussian components, each of which is described by the following parameters:

[0126] mean ;variance ; Weight ,in ;

[0127] Real-time update: For each new input current frame t, calculate the matching degree with the current background model and adjust the learning rate according to the preset learning rate. , adjust the Gaussian model parameters to adapt to background changes:

[0128]

[0129]

[0130] in, Represents the mean value of any pixel in the background model before the current frame is updated; is the mean value of any pixel in the updated background model; is the pixel value of the corresponding position in the current frame;

[0131] is the variance of any pixel in the background model before the current frame update, is the variance of any pixel in the updated background model; is the square difference between the current frame pixel value and the updated mean;

[0132] Make sure the warning area contains pixels, and each pixel has an independent background model;

[0133] Each pixel in the warning area is marked as , Represents the pixel at the jth row and zth column, and according to The corresponding background model calculates the likelihood of the current frame t pixel value as ;

[0134]

[0135] is the grayscale value of the pixel in row j and column z in the current frame; is the mean of the kth Gaussian component in the background model; is the variance of the kth Gaussian component in the background model; is the weight of the kth Gaussian component; is the likelihood value of the pixel at row j and column z in the current frame t; The index representing the current frame;

[0136] In the current time window T, it is assumed that the time window T contains N1 frames, and these frames are represented as {1, 2, ..., t-1, t, ..., N1}, where t represents the current frame. The likelihood values ​​of all pixels in the warning area are integrated into the total likelihood value of the current frame. :

[0137]

[0138] is the average likelihood value of the entire warning area in the current t-th frame; M is the total number of pixel rows in the warning area; N is the total number of pixel columns in the warning area;

[0139] The total likelihood value difference between two consecutive frames is taken as the likelihood value fluctuation amplitude, which is defined as:

[0140]

[0141] is the fluctuation amplitude of the likelihood value between the t-th frame and the t-1-th frame;

[0142] If the video frame rate is fixed, Directly reflects the rate of background change;

[0143] In the time window T, the stability index of the background motion field is defined as ; and Defined as the fluctuation amplitude of the likelihood value The standard deviation of:

[0144]

[0145] in, is the average value of all likelihood value fluctuations within the time window T, The larger the value, the more dramatic the background change;

[0146] Based on experimental demonstration or expert group analysis, simulate light flashing, screen content switching, or rapid shadow changes in the background area corresponding to the warning area. Determine the fluctuation range of the likelihood value in these simulation data. The acceptable upper limit value is set, under which the micro-motion video monitoring of the detected target in the warning area will not be affected, and the standard deviation of the likelihood value fluctuation amplitude corresponding to this upper limit value is used as The value of

[0147] Set the threshold of the stability index to ,when When the background changes greatly, it indicates that the background changes greatly. The background changes greatly may be caused by flickering lights, switching of screen content, or rapid changes in shadows.

[0148] Adjustment strategy: During anomaly detection, increase the number of frames accumulated in micro-motion detection or reduce the confidence level.

[0149] Further explanation: the micro-motion and background field index coupling evaluation module specifically includes:

[0150] For the Kalman filter: the state transition equation of the Kalman filter is used to describe the evolution of the warning area i from the current t-th frame state to the next t+1-th frame state;

[0151] Set the initial state vector : represents the initial micro-motion parameters and background stability;

[0152] The calculation formula of the state covariance matrix is ​​as follows:

[0153]

[0154] is the state vector, including and ;

[0155] A is the state transition matrix, which describes the evolution of the system from the t-th frame to the t+1-th frame;

[0156] B is the control input matrix, taking into account external influences;

[0157] u(t) is the control variable vector, representing the system input;

[0158] w(t) is the process noise, reflecting the uncertainty of the model; the observation equation is expressed as:

[0159]

[0160] is the observation vector, i.e. the measured micro-motion characteristic index ; H is the observation matrix, which maps the state to the observation space; v(t) is the observation noise, which reflects the uncertainty of the observation process;

[0161] Use nonlinear function to update the state of the t+1th frame , introducing the stability index The influence is calculated as follows:

[0162]

[0163] is the state process noise, simulating environmental uncertainty; It is a nonlinear function that reflects the complex relationship between micro-movements and background changes;

[0164] Using filtered To calculate the abnormal suspicion of the warning area, and obtain the following comprehensive evaluation index for the warning area i:

[0165]

[0166] Using the updated micro-motion feature indicators and the stability index of the previous moment To define the comprehensive evaluation index This combination allows the calculation formula to fully utilize the latest dynamic state information while maintaining a certain degree of rationality in the historical stability effect. Even if the stability index is not updated in real time, it can still reflect the inertial characteristics of the system. The specific description is as follows:

[0167] Balanced anomaly detection: A comprehensive evaluation index combines the latest micro-motion parameters with historical stability. This balance allows the system to respond promptly to unusual changes while avoiding overreactions to brief or occasional changes. For example, in the event of sudden environmental disturbances (such as changes in lighting), the system can mitigate fluctuations in anomaly detection through the stability index.

[0168] Enhanced robustness and tolerance: Using a historical stability index is a built-in mechanism for system tolerance, providing a more stable evaluation in environments with frequent or irregular motion. This not only reduces the possibility of false positives, but also improves the system's adaptability to long-term observations.

[0169] according to Selection and standardization of The valid range of is [0,1);

[0170] is the comprehensive evaluation index of warning area i; is the output relationship function, which maps the filter state to a comprehensive evaluation index; and It has the following selection features:

[0171] When there is suspicious behavior within the time window T and the background changes greatly, the corresponding The closer the value is to 1;

[0172] When there is no suspicious behavior within the time window T, The output value is 0.

[0173] Further explanation: A way to choose, and then get The specific calculation formula is:

[0174]

[0175] The results of the first abnormal behavior evaluation are as follows:

[0176] When there is suspicious behavior within the time window T and the background changes greatly, The output value is ; At this time, the micro-motion characteristic index and stability index The larger the value, the The closer the value is to 1, the more severe the abnormal behavior in warning area i is.

[0177] When there is suspicious behavior within the time window T and the background change amplitude is small, The output value is , at this time the micro-motion characteristic index The larger the value, the The closer the value is to 1, the more severe the abnormal behavior in warning area i is.

[0178] Among them, the severity of the first-level abnormal behavior is greater than the severity of the second-level abnormal behavior;

[0179] When there is no suspicious behavior within the time window T, The output value is 0.

[0180] Further explanation: the impact area status evaluation module specifically includes:

[0181] Set the warning area i and the associated impact area The real-time status data includes abnormal status indicators and correlation factors, where the correlation factor is used to describe the degree of abnormal correlation between the abnormal status indicator and the warning area i;

[0182] The abnormal status indicators and correlation factors are combined for analysis to generate the regional status evaluation index, which is defined as follows:

[0183]

[0184] in, The affected area Abnormal status indicators, Through normalization, the output value is limited to (0,1). The affected area The abnormal status indicator warning value; is the correlation factor; The affected area Regional status evaluation index; based on Perform settings, initial settings for The median value of the range; can be adjusted accordingly to the actual application; and Calculate the corresponding values ​​according to the actual application scenario. The calculation method is: actual application index value / normal state index value, and the obtained ratio is normalized, and the output value is limited to (0,1);

[0185] In this embodiment, when applied to the video stream in the boarding bridge of an aircraft, Indicates the noise parameters or vibration parameters generated when a boarding passenger throws objects from the warning area to the impact area, or the relevant parameters of the impact force generated in the impact area due to the thrown objects;

[0186] When the abnormal correlation degree between the abnormal status indicator and the warning area i reaches more than 35%, it means that the abnormal correlation degree between the abnormal status indicator and the warning area i is large;

[0187] When the abnormal correlation degree between the abnormal status indicator and the warning area i reaches below 35%, it means that the abnormal correlation degree between the abnormal status indicator and the warning area i is small. The above correlation degree of 35% is adjusted based on the experimental data of the expert group and is not limited to this value.

[0188] The above correlation degree is determined by analyzing the synchronization ratio between the abnormal state indicator and the slight movement when the detected target in the warning area i makes a slight movement;

[0189] When the abnormal status indicator has a high correlation with the abnormality of warning area i, The value range is set to ; This setting enables When a value changes, Large changes in values; large changes in values ​​indicate Under the settings, The change exceeds the set 30% of the time;

[0190] When the abnormal status indicator has a small correlation with the abnormality of warning area i, The value range is set to ; This setting enables When a value changes, Small changes in values; small changes in values ​​indicate Under the settings, The range of change is not set Less than 30% of the time;

[0191] Through standardization, and The value range of is (0,1);

[0192] Indicates when hour, The calculation formula is ;

[0193] Indicates when hour, The output value is 0.5; the evaluation results of the second abnormal behavior are as follows:

[0194] set up The valid value range of is (0,1); when The closer it is to 1, the more it affects the area. The more severe the abnormality;

[0195] when The closer it is to 0, the more it affects the area. The smaller the abnormality;

[0196] Based on experimental demonstration or expert group system analysis, set The classification threshold is , The value range is ; The specific value is determined by the expert group system through actual calculation needs and is not described in detail. Select 0.5;

[0197] when When The abnormality level is in the high impact range;

[0198] when When The abnormality level is in the low impact range.

[0199] Further explanation: Under different priority classification results, the corresponding level of alarm mechanism is triggered, including:

[0200] The comprehensive evaluation index output values ​​of all warning areas in the warning area list {1, 2, …, i, …, n} are filtered. The filtering strategy is to remove the warning areas with a comprehensive evaluation index output value of 0. Then, the comprehensive evaluation indexes of the filtered warning areas are prioritized according to the output value. The larger the output value, the higher the corresponding priority ranking. The filtered list set {1, 2, …, U} is obtained, where U is the total number of warning areas corresponding to the priority-sorted list set, and {1, 2, …, U} is included in {1, 2, …, n}.

[0201] For abnormal behaviors in the list set {1, 2, …, U}, the conditions for triggering the first-level alarm mechanism are any of the following:

[0202] In the warning area with severity level 1, if the abnormality level is in the high impact range or low impact range, the first level alarm mechanism will be triggered;

[0203] The severity level is level 2 warning area, and the abnormality level is in the high impact range, triggering the level 1 alarm mechanism;

[0204] For abnormal behaviors in the list set {1, 2, …, U}, the severity level is the warning area of ​​level 2, and the abnormality level is in the low impact range, triggering the second level alarm mechanism;

[0205] The response speed and alarm measures of the first-level alarm mechanism are higher than those of the second-level alarm mechanism;

[0206] The first level alarm mechanism is as follows:

[0207] Video push: The video stream corresponding to the warning area that triggers the first-level alarm is pushed to the monitoring center manager interface in real time. The relevant warning area is flashing or marked with a red frame in the interface to ensure that it attracts the attention of the operator.

[0208] Audible alarm: Triggers a high-priority alarm tone inside the system, using a siren.

[0209] Log records: record the timestamp of the triggering of the first-level abnormal behavior, the warning area number, the value of the comprehensive evaluation index, and the video stream screenshot / video clip after the alarm is triggered.

[0210] Smart Notification: Push high-priority warning messages to system administrators' mobile phones, management platforms, etc. The message content includes the time of the abnormality, the specific warning area number, and the cause or description of the abnormality.

[0211] Subsequent processing recommendations: Automatically record the duration of a Level 1 anomaly. If a Level 1 anomaly persists for longer than a set time threshold (set to 10 seconds in this example), a higher-level alert will be issued, including a full staff notification or more urgent action. Alternatively, AI-assisted diagnosis can be enabled to further analyze the video content of the Level 1 anomaly area to assess whether a dangerous event exists.

[0212] The second level alarm mechanism is as follows:

[0213] Video Marking: Mark the warning area of ​​the second-level alarm in the monitoring center interface, highlighting it with a yellow frame or other graded mark to alert the administrator. The video stream is automatically saved in the background but will not be pushed to the front desk immediately, waiting for operator confirmation.

[0214] Sound prompt: triggers a medium-priority sound prompt within the system, which in this embodiment is a beep, to avoid interfering with the processing of high-priority events.

[0215] Abnormal statistics: The system background records the warning area number, timestamp and comprehensive evaluation index value that triggers the second-level alarm.

[0216] Time monitoring: Continuously monitor changes in Level 2 abnormal behavior. If the comprehensive evaluation index value increases and exceeds 0.7 within a certain period of time, it will immediately upgrade to the first level alarm mechanism. In this embodiment, the certain period of time is 10 seconds; 0.7 is adjusted according to the expert group system;

[0217] Follow-up Action Recommendations: The system generates a summary of abnormal behavior at regular intervals, set at one minute, and pushes abnormal statistical information about the relevant Level 2 alarm areas to the monitoring center. If the Level 2 anomaly persists for longer than a set threshold (set at 30 seconds in this example), a manual review process is triggered, with a dedicated person determining whether an alarm escalation is necessary.

[0218] The key advantages and performance improvements of this embodiment are as follows:

[0219] Enhanced reliability in complex scenarios:

[0220] By introducing the stability index Taking into account the considerations of the system, when faced with large-area ambient light changes, video noise or screen flicker, the system can adaptively reduce its "oversensitivity" to micro-motion anomalies, thereby reducing false alarms;

[0221] Reduce false detection and missed detection rates

[0222] Compared with the traditional method that relies on only a single path, the stability index is coupled After the information is collected, both false detection and missed detection can be improved. In actual tests, the false detection or missed detection rate can be reduced by 5% to 10%.

[0223] Suitable for automatic analysis of massive video streams:

[0224] When monitoring multiple warning areas simultaneously, as long as abnormal behavior occurs in a certain channel and the background field stability is not in a violent fluctuation range, the system can quickly make a relatively accurate judgment and push it to the monitoring center;

[0225] Through visual recommendations and alarm mechanisms, security personnel can reduce the tedious "patrols" of all channels, thus improving work efficiency.

[0226] By focusing on both the target's micro-motion characteristics and background stability during the detection process, more stable abnormal behavior recognition is achieved in environments with complex lighting or changing backgrounds. Combined with the application of "framing warning areas", the system can quickly filter out suspicious images that truly require focus from massive real-time video streams and automatically recommend them to monitoring personnel, thereby effectively improving the efficiency and accuracy of overall security monitoring.

[0227] The above formulas are all dimensionless and numerical calculations. The formulas are obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters in the formulas are set by technicians in this field according to actual conditions.

[0228] The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed by hardware or software depends on the specific application and design constraints of the technical solution.

[0229] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, and may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment as needed.

[0230] The above is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. A real-time integrated multi-video stream multi-level analysis and monitoring system, characterized in that: Specifically include: Warning area demarcation module: used to predetermine at least one warning area in the surveillance video image, record the coordinate information or mask information of the warning area, and match each warning area with its associated impact area; The affected area refers to a specific range outside the warning area, which involves the safety of monitored equipment or personnel; Target detection and micro-motion feature extraction module: used to detect the target in the warning area and obtain key point information of the detected target in the warning area; Input the acquired key point information into a micro-motion analysis module, which includes a frequency domain analysis algorithm; By using a sliding time window and frequency domain analysis algorithm, the micro-motion of the detected target is analyzed to extract its main frequency and amplitude. Combine the extracted micro-motion main frequency and amplitude to generate the micro-motion characteristic index of the detected target; Background motion field stability index generation module: used to perform background modeling on the warning area and its surrounding areas to obtain a background model; Continuously calculate the likelihood value fluctuation amplitude between the current frame and the background model; By calculating the standard deviation of the likelihood value fluctuation amplitude, the background variation amplitude is measured to obtain the stability index of the background motion field; Micro-motion and background field index coupling evaluation module: used for performing nonlinear coupling analysis on the generated micro-motion characteristic index and the generated stability index; The coupling analysis uses Kalman filtering to construct a time series model to obtain a comprehensive evaluation index for evaluating the abnormal suspicion of the detected target, and the comprehensive evaluation index is used to provide a first abnormal behavior evaluation result for the warning area; Impact area status evaluation module: used to collect real-time status data of the impact area associated with each warning area, analyze and process these real-time status data, and generate a regional status evaluation index. The regional status evaluation index is used to provide a second abnormal behavior evaluation result for the impact area; Abnormal detection and alarm triggering module: Based on the first abnormal behavior evaluation result of the warning area, the abnormal behavior of the detected target in the warning area is prioritized; then, based on the second abnormal behavior evaluation result of the affected area, the corresponding level of alarm mechanism is triggered under different priority division results, and the monitoring video channel is pushed to the monitoring center or automatically highlighted in the interface.

2. The real-time integrated multi-video stream multi-stage analysis and monitoring system according to claim 1, characterized in that: Warning area demarcation module, specifically including: (1.1) Video input preprocessing: Access multiple video streams and obtain real-time data through video acquisition devices; Adopt codec algorithm to encode and decode video stream in real time to ensure synchronous processing; Perform split processing on the encoded video frames to ensure that each video stream can be analyzed independently; (1.2) Video stream background modeling; (1.3) Warning area selection: Pre-select the warning areas that need to be monitored in the video stream to generate a warning area list, marked as {1,2,…,i,…,n}, where i represents the index of the warning area i; n represents the total number of warning areas; (1.4) Coordinate and mask information recording: Determine the boundaries of each warning area i, extract its coordinate information and mask information for subsequent processing, and store these data in the spatial database; (1.5) Matching of warning area and impact area: Match the information of each warning area i with the predefined impact area, and use the impact area of ​​warning area i as express.

3. The real-time integrated multi-video stream multi-stage analysis and monitoring system according to claim 2, characterized in that: The target detection and micro-motion feature extraction module includes: The key point information is set to include the small displacement of the arms and legs of the detected target within a preset unit time; In the target detection phase of the warning area i, the skeleton point tracking algorithm is used to count the small displacements of the human arms and legs per unit time; Set an upper threshold for the small displacement of the arms and legs within a preset unit time, and mark micro-movements exceeding the corresponding upper threshold as "suspicious small movements"; The first evaluation index of warning area i is defined as , the calculation formula is as follows: in, is the small displacement of the skeleton key point in the warning area i, is the upper threshold of the skeleton key point in the warning area i; It is the first evaluation index of the skeleton key point in the warning area i; GG represents SB and TB, SB and TB represent the arm and leg respectively; like When the micro-movement is marked as "suspicious micro-movement"; The small displacements of the arms and legs within a preset unit time are placed in a sliding time window. Using frequency domain analysis algorithms such as Fourier transform or wavelet analysis, the frequency components and corresponding amplitudes of the "suspicious small movements" of the detected target are extracted. Generate micro-motion feature indicators for each detected target based on the extracted frequency components and amplitude information ; Micro-motion characteristic index The calculation formula is as follows: in, is the frequency component of "suspicious small movements" in warning area i, is the amplitude of the frequency component; is the maximum displacement in the time window T; are weighting factors for frequency, amplitude, and displacement, The values ​​of are all in the interval (0,1), and ; Pre-set the micro-motion characteristic indicators of the detected target The comparison threshold, if within any time window T, If the comparison threshold is exceeded, it is considered suspicious behavior. The characterization formula is: ;in is the comparison threshold of warning area i.

4. The real-time integrated multi-video stream multi-stage analysis and monitoring system according to claim 3, characterized in that: The background motion field stability index generation module specifically includes: Background modeling is constructed using an algorithm based on Gaussian mixture model; Make sure the warning area contains pixels, and each pixel has an independent background model; Each pixel in the warning area is marked as , Represents the pixel at the jth row and zth column, and according to The corresponding background model calculates the likelihood of the current frame t pixel value as ; In the current time window T, it is assumed that the time window T contains N1 frames, and these frames are represented as {1, 2, ..., t-1, t, ..., N1}, where t represents the index of the current frame, and the likelihood values ​​of all pixels in the warning area are integrated into the total likelihood value of the current frame. : is the average likelihood value of the entire warning area in the current t-th frame; M is the total number of pixel rows in the warning area; N is the total number of pixel columns in the warning area; The total likelihood value difference between two consecutive frames is taken as the likelihood value fluctuation amplitude, which is defined as: is the fluctuation amplitude of the likelihood value between the t-th frame and the t-1-th frame; In the time window T, the stability index of the background motion field is defined as ; and Defined as the fluctuation amplitude of the likelihood value The standard deviation of: in, is the average value of all likelihood value fluctuations within the time window T, The larger the value, the more dramatic the background change; Set the threshold of the stability index to ,when When , the background changes greatly.

5. The real-time integrated multi-video stream multi-stage analysis and monitoring system according to claim 4, characterized in that: The micro-motion and background field index coupling evaluation module specifically includes: The state transition equation of the Kalman filter is used to describe the evolution of the warning area i from the current t-th frame state to the next t+1-th frame state; Specifically, a nonlinear function is used to update the micro-motion feature index corresponding to the t+1 frame state, and the micro-motion feature index is marked as , and then introduce the stability index The influence is calculated as follows: is the state process noise, simulating environmental uncertainty; It is a nonlinear function that reflects the complex relationship between micro-movements and background changes; They represent the micro-motion characteristic index and stability index in the t-th frame state respectively; The Kalman filter is used to obtain To calculate the abnormal suspicion of the warning area, and obtain the following comprehensive evaluation index for the warning area i: set up The valid range of is [0,1); is the comprehensive evaluation index of warning area i; is the output relationship function, which maps the filter state to a comprehensive evaluation index; and It has the following selection features: When there is suspicious behavior within the time window T and the background changes greatly, the corresponding The closer the value is to 1; When there is no suspicious behavior within the time window T, The output value is 0.

6. The real-time integrated multi-video stream multi-stage analysis and monitoring system according to claim 5, characterized in that: The following is A way to choose, and then get The specific calculation formula is: The results of the first abnormal behavior evaluation are as follows: When there is suspicious behavior within the time window T and the background changes greatly, The output value is ; At this time, the micro-motion characteristic index and stability index The larger the value, the The closer the value is to 1, the more severe the abnormal behavior in warning area i is. When there is suspicious behavior within the time window T and the background change amplitude is small, The output value is , at this time the micro-motion characteristic index The larger the value, the The closer the value is to 1, the more severe the abnormal behavior in warning area i is. Among them, the severity of the first-level abnormal behavior is greater than the severity of the second-level abnormal behavior; When there is no suspicious behavior within the time window T, The output value is 0.

7. The real-time integrated multi-video stream multi-stage analysis and monitoring system according to claim 6, characterized in that: Impact area status assessment module, specifically including: Set the warning area i and the associated impact area The real-time status data includes abnormal status indicators and correlation factors, where the correlation factor is used to describe the degree of abnormal correlation between the abnormal status indicator and the warning area i; The abnormal status indicators and correlation factors are combined for analysis to generate the regional status evaluation index, which is defined as follows: in, The affected area Abnormal status indicators, The affected area The abnormal status indicator warning value; is the correlation factor; The affected area Regional status evaluation index; When the abnormal status indicator has a high correlation with the abnormality of warning area i, The value range is set to ; This setting enables When a value changes, Large changes in values; When the abnormality indicator has a small correlation with the abnormality of warning area i, The value range is set to ; This setting enables When a value changes, The numerical value changes slightly; the second abnormal behavior evaluation results include: set up The valid value range of is (0,1); when The closer it is to 1, the more it affects the area. The more severe the abnormality; when The closer it is to 0, the more it affects the area. The smaller the abnormality, the smaller the The classification threshold is , The value range is ; when When The abnormality level is in the high impact range; when When The abnormality level is in the low impact range.

8. The real-time integrated multi-video stream multi-stage analysis and monitoring system according to claim 7, characterized in that: Under different priority classification results, the corresponding level of alarm mechanism is triggered, including: The comprehensive evaluation index output values ​​of all warning areas in the warning area list {1, 2, …, i, …, n} are filtered. The filtering strategy is to remove the warning areas with a comprehensive evaluation index output value of 0. Then, the comprehensive evaluation indexes of the filtered warning areas are prioritized according to the output value. The larger the output value, the higher the corresponding priority ranking. The filtered list set {1, 2, …, U} is obtained, where U is the total number of warning areas corresponding to the priority-sorted list set, and {1, 2, …, U} is included in {1, 2, …, n}. For abnormal behaviors in the list set {1, 2, …, U}, the conditions for triggering the first-level alarm mechanism are any of the following: In the warning area with severity level 1, if the abnormality level is in the high impact range or low impact range, the first level alarm mechanism will be triggered; The severity level is level 2 warning area, and the abnormality level is in the high impact range, triggering the level 1 alarm mechanism; For abnormal behaviors in the list set {1, 2, …, U}, the severity level is the warning area of ​​level 2, and the abnormality level is in the low impact range, triggering the second level alarm mechanism; The response speed and alarm measure mechanism of the first-level alarm mechanism are higher than those of the second-level alarm mechanism.

Images