Monitoring system for judging video behavior based on deep learning

Abstract

The application relates to the technical field of video behavior monitoring, and discloses a monitoring system for judging video behavior based on deep learning. The system comprises a video stream acquisition unit, a time delay analysis unit, a behavior fluctuation analysis unit, an abnormal probability analysis unit, an error probability analysis unit and a judgment threshold adjustment unit. The system collects multi-monitoring point video stream data through the video stream acquisition unit; the time delay analysis unit determines a time delay value according to the time sequence correlation of two monitoring points; the behavior fluctuation analysis unit combines the short-time window deep feature distribution of the monitoring points and the corresponding feature distribution of the lower monitoring point after time delay adjustment to calculate a behavior fluctuation factor; the abnormal probability analysis unit comprehensively considers the correlation difference and the fluctuation factor of adjacent monitoring points to determine the abnormal probability of each feature dimension and extract the abnormal dimension; the error probability analysis unit obtains the behavior error probability according to the abnormal dimension distribution of all adjacent monitoring points; and the judgment threshold adjustment unit adjusts the threshold value according to the above to complete automatic judgment of the video behavior.

Description

Technical Field

[0001] This invention relates to the field of video behavior monitoring technology, specifically a monitoring system based on deep learning to determine video behavior. Background Technology

[0002] With the widespread application of surveillance technology in public safety, traffic management, and park supervision, multi-monitoring point collaborative operation has become the main mode for achieving large-scale, all-round behavior monitoring. Behavior judgment based on video data is the core link for effective supervision by surveillance systems. Currently, most surveillance systems still have many technical shortcomings when processing behavior judgment from multiple monitoring points, making it difficult to meet the application requirements of precision and intelligence.

[0003] In terms of time-series processing of multi-monitoring data, traditional systems, while capable of simultaneously acquiring video streams from multiple monitoring points, often analyze each point's data independently without considering time differences arising from factors such as device response speed, data transmission paths, and network environments. These time differences disrupt the temporal relationships between the video streams from different monitoring points, rendering subsequent feature comparisons based on temporal logic inaccurate. For example, when tracking the movement of the same target across monitoring points, the lack of time delay correction may cause the system to misjudge the target's actions at different times as concurrent, leading to chaotic behavior trajectory analysis and an inability to accurately reconstruct the target's behavioral logic.

[0004] At the level of behavioral fluctuation capture, existing technologies typically rely solely on the feature distribution of a single monitoring point's video stream within a short time window to determine behavioral changes. This approach fails to incorporate feature data from adjacent monitoring points or adjust for their time delays. The short-term feature distribution of a single monitoring point only reflects localized behavioral fragments, lacking correlation with features from adjacent monitoring points, making it difficult to comprehensively capture dynamic behavioral fluctuations. Furthermore, the lack of adjusted time delays leads to temporal mismatches in the feature distributions of adjacent monitoring points, further reducing the accuracy of behavioral fluctuation analysis. For example, when judging behavioral fluctuations related to changes in walking speed, relying solely on the features of a single monitoring point may fail to identify speed changes when crossing monitoring point boundaries, resulting in a one-sided judgment of behavioral fluctuations that fails to accurately reflect the true dynamics of the behavior.

[0005] In the anomaly probability calculation stage, traditional systems often focus on single feature differences in video stream data, such as pixel grayscale differences or target contour differences, ignoring the multi-dimensional correlations of features in the deep feature space and failing to comprehensively consider the impact of temporal differences and behavioral fluctuation factors on anomaly detection. This single-dimensional anomaly assessment method easily misses anomalies caused by temporal logic errors or abnormal behavioral fluctuations. For example, some behaviors may appear normal in a single feature dimension, but when combined with temporal differences and behavioral fluctuation factors, they exhibit obvious anomaly characteristics. Traditional systems, lacking multi-dimensional comprehensive analysis, cannot effectively identify such anomalies, resulting in insufficient accuracy in anomaly detection.

[0006] Current monitoring systems typically use fixed thresholds for behavior assessment, which remain unchanged after deployment. However, the frequency, characteristics, and error rates of behaviors vary significantly across different monitoring scenarios, making fixed thresholds ill-suited for adapting to these changes. Furthermore, traditional systems do not calculate the probability of behavior errors based on the anomaly distribution across multiple adjacent monitoring points; instead, they adjust thresholds solely based on the error rate of a single monitoring point, resulting in a lack of comprehensive error data for threshold adjustments. In densely populated shopping malls versus sparsely populated parking lots, fixed thresholds may lead to either excessively high false alarm or false negative rates, failing to meet the practical needs for accurate monitoring in different scenarios. These technical shortcomings collectively result in significant deficiencies in the accuracy, adaptability, and comprehensiveness of current monitoring systems for video behavior assessment, making it difficult to meet the regulatory requirements of complex scenarios. Summary of the Invention

[0007] The purpose of this invention is to provide a monitoring system based on deep learning to judge video behavior, so as to solve the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides a monitoring system for judging video behavior based on deep learning, the system comprising: The video stream acquisition unit is used to acquire video stream data from multiple monitoring points; The time delay analysis unit is used to determine the time delay value between the video stream data of two monitoring points based on the timing correlation between the video stream data of two monitoring points. The behavior fluctuation analysis unit is used to determine the behavior fluctuation factor of the monitoring point in each short window based on the depth feature distribution of the video stream data of the monitoring point in the short window and the depth feature distribution of the video stream data of the next monitoring point after time delay adjustment in the short window. The anomaly probability analysis unit is used to determine the anomaly probability of the video stream data of the monitoring point in each feature dimension based on the feature differences, temporal differences, and behavioral fluctuation factors of the video stream data contained in all short-time windows of adjacent monitoring points in the deep feature space; and to obtain the anomaly dimension based on the anomaly probability of the video stream data of the monitoring point in all feature dimensions. The error probability analysis unit is used to obtain the behavior error probability based on the distribution of the anomaly dimension at all adjacent monitoring points; The threshold adjustment unit is used to adjust the behavior judgment threshold according to the behavior error probability to complete the automatic judgment of video behavior.

[0009] Preferably, the time delay analysis unit determines the time delay value by including: The video stream data from the two monitoring points are preprocessed to extract the depth feature sequence of each frame; Two deep feature sequences are input into a long short-term memory network for temporal modeling, and the hidden state sequence is output. Calculate the cosine similarity of two hidden state sequences at different time offsets and generate similarity curves; By finding the global peak in the similarity curve, the time offset corresponding to the peak is determined as the time delay value.

[0010] Preferably, the behavior fluctuation analysis unit determines the behavior fluctuation factor by including: Extract multidimensional feature representations from the video stream data of each monitoring point within a short time window; Using the discriminator module in the generative adversarial network, calculate the distribution difference score between the multidimensional feature representations of the current monitoring point and the next monitoring point; The behavioral fluctuation factor is obtained by inverting the distribution difference score.

[0011] Preferably, the method for obtaining the short window is as follows: Motion intensity is detected for each video stream, and the window is triggered when the motion intensity exceeds the dynamic threshold. The window length is adaptively adjusted based on motion duration and inter-frame consistency to generate a short-time window of variable length.

[0012] Preferably, the anomaly probability analysis unit determines the anomaly probability of the video stream data at the monitoring point in each feature dimension, including: For any two adjacent monitoring points, the time delay window is obtained by applying a time delay value to each short window of the video stream data of the previous monitoring point. Extract all frame sequences that overlap with the time delay window from the video stream data of the next monitoring point to form a combined window; The encoder-decoder architecture is used to calculate the reconstruction error and potential representation distance of each short-time window and the combined window under the same feature dimension. The reconstruction error and the latent representation distance are fused into a fluctuation recovery coefficient; The ratio of the behavioral volatility factor to the volatility recovery coefficient is normalized and then compared with a constant parameter to obtain the difference. The average of the differences over all short-term windows is taken as the anomaly probability.

[0013] Preferably, determining the fluctuation recovery coefficient for each feature dimension within each short-time window includes: The feature change is learned through a neural network. This feature change is obtained by passing the ratio of the feature values ​​of the combined window and the short-time window through a non-linear activation function. The temporal variation is learned through a recurrent neural network. This temporal variation is obtained by using the phase information of the combined window and the short window through a temporal attention mechanism. The characteristic change and the time series change are weighted and fused to output the fluctuation recovery coefficient.

[0014] Preferably, the anomaly probability analysis unit obtains the anomaly dimensions based on the anomaly probabilities of the video stream data from the monitoring point across all feature dimensions, including: Density estimation is performed on the anomaly probability of each feature dimension to generate a probability distribution function; Select the quantile at the tail of the probability distribution function as the adaptive threshold; Feature dimensions whose anomaly probability exceeds an adaptive threshold are marked as anomaly dimensions.

[0015] Preferably, the error probability analysis unit obtains the behavioral error probability based on the distribution of anomaly dimensions at all adjacent monitoring points, including: Collect the set of anomaly dimensions from all monitoring points, use graph neural networks to perform community detection, and identify the frequently occurring anomaly dimensions as key feature dimensions. For each monitoring point, calculate the divergence index of its video stream data feature value sequence on the key feature dimension; The probability of behavioral error is obtained by minimax scaling of the divergence index of all monitoring points.

[0016] Preferably, the judgment threshold adjustment unit adjusts the behavior judgment threshold according to the behavior error probability, including: The probability of behavioral error at each monitoring point is input into a fully connected neural network, which outputs a threshold adjustment coefficient. The temporary threshold is obtained by multiplying the threshold adjustment factor by the historical judgment threshold. The temporary thresholds of all monitoring points are weighted and averaged to generate the actual judgment threshold. The video behavior is classified into two categories based on the actual judgment threshold.

[0017] Preferably, when extracting multidimensional feature representations in the behavior fluctuation analysis unit, a multi-task learning framework is used to simultaneously extract appearance features, motion features, and semantic features; the discriminator module in the adversarial generative network adopts a multilayer perceptron structure, and an attention mechanism is introduced when calculating the distribution difference score to focus on key feature dimensions; the inversion transform uses a logarithmic function to process the distribution difference score to obtain the behavior fluctuation factor.

[0018] Compared with the prior art, the beneficial effects of the present invention are: The video stream acquisition unit can acquire video stream data from multiple monitoring points, breaking the limitations of traditional monitoring systems that rely on data from a single monitoring point or, although acquiring data from multiple monitoring points, fail to effectively integrate it. The introduction of data from multiple monitoring points allows the system to capture behavioral information from different perspectives and locations, avoiding incomplete behavioral judgments caused by blind spots or missing data from a single monitoring point. Whether in a wide plaza or inside a complex building, data from multiple monitoring points provides rich information support for subsequent analysis, allowing behavioral judgments to be based on a more comprehensive information foundation and reducing judgment biases caused by incomplete data.

[0019] The time delay analysis unit determines the time delay value based on the temporal correlation between video stream data from two monitoring points, accurately solving the problem of traditional systems ignoring time differences between multiple monitoring points. In multi-monitoring scenarios, factors such as the response speed of data acquisition devices and the length of data transmission paths can cause time differences in video streams from different monitoring points. If these differences are not corrected, all subsequent time-series-based analyses will lose accuracy. This unit calculates the time delay value through temporal correlation, accurately quantifying and correcting these time differences. This ensures that the video stream data from different monitoring points maintains temporal consistency, providing a reliable temporal basis for subsequent behavioral fluctuation analysis and anomaly probability calculation, and avoiding deviations in behavioral logic analysis caused by temporal inconsistencies.

[0020] The behavior fluctuation analysis unit combines the depth feature distribution of the monitoring point's own short-term window with the corresponding feature distribution of the next monitoring point after time delay adjustment to determine the behavior fluctuation factor, enabling more accurate dynamic behavior capture capabilities. The depth feature distribution more comprehensively reflects the essential characteristics of video behavior, such as the target's action patterns and posture changes, while time delay adjustment ensures the temporal matching of features from adjacent monitoring points. The combination of these two factors allows the calculation of the behavior fluctuation factor to accurately reflect the dynamic trends of behavior within a short period. Whether it's a sudden change in a person's movement speed, a temporary shift in an object's state, or an abnormal adjustment in behavior patterns, this unit can accurately capture these changes through correlation analysis of feature distributions, avoiding misjudgments of behavior fluctuations caused by single features or temporal mismatches, making the system more sensitive to behavioral details.

[0021] The anomaly probability analysis unit integrates feature differences, temporal differences, and behavioral fluctuation factors of all short-time window video streams from adjacent monitoring points in the deep feature space to determine the anomaly probability of each feature dimension, and further derives the anomaly dimension, breaking through the limitation of the single-dimensionality of traditional anomaly judgment. Feature differences in the deep feature space can reflect anomalies in the essential characteristics of behavior, such as abnormal deformation of the target contour; temporal differences can identify anomalies caused by disordered temporal logic, such as disordered sequence of behavior; behavioral fluctuation factors can capture anomalies in dynamic changes of behavior, such as sudden increase in behavioral amplitude. The combination of these three factors allows the calculation of anomaly probability to comprehensively assess the degree of anomaly of behavior from different dimensions, avoiding the anomaly omission or misjudgment caused by single-dimensional analysis in traditional systems, making the judgment of anomaly dimensions more comprehensive and accurate, and providing accurate anomaly basis for subsequent behavioral error analysis.

[0022] The error probability analysis unit derives the behavioral error probability based on the distribution of anomalies at all adjacent monitoring points, providing a more comprehensive error assessment capability. In multi-monitoring collaborative monitoring scenarios, the anomalies at a single monitoring point may be accidental, such as local anomalies caused by temporary equipment failures. However, the distribution of anomalies at all adjacent monitoring points reflects the overall error trend of behavioral judgment across the entire monitoring network, including error differences in different areas and the degree of correlation between errors. By analyzing this distribution, the system can accurately grasp the source and overall level of errors in the behavioral judgment process, avoiding the one-sidedness of error analysis at a single monitoring point. This makes the calculation of behavioral error probability more closely aligned with the actual operation of the entire monitoring system, providing a comprehensive and reliable error reference for threshold adjustment.

[0023] The threshold adjustment unit adjusts the behavior judgment threshold based on the probability of behavior error, achieving dynamic threshold adaptation and solving the problem that traditional fixed thresholds cannot adapt to different scenarios. In different monitoring scenarios, the frequency, characteristics, and error levels of behaviors vary significantly. Fixed thresholds can easily lead to more false alarms in scenarios with high error rates, while potentially causing more missed alarms in scenarios with low error rates. This unit, through the linkage adjustment of behavior error probability and threshold, allows the threshold to change in real time according to the actual error situation. When the error is high, the threshold is appropriately increased to reduce false alarms; when the error is low, the threshold is reasonably decreased to avoid missed alarms. Whether in densely populated public places or sparsely populated remote areas, the system can adapt to scenario requirements through dynamic threshold adjustment, improving the system's practicality and reliability in diverse scenarios. Attached Figure Description

[0024] Figure 1 This is a schematic diagram illustrating the working principle of the video behavior judgment monitoring system based on deep learning as described in this invention. Figure 2 A flowchart for determining the time delay value for the time delay analysis unit; Figure 3 A flowchart for determining the anomaly probability for the anomaly probability analysis unit. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Please see Figure 1 This invention provides a monitoring system for judging video behavior based on deep learning. The system includes: a video stream acquisition unit, a time delay analysis unit, a behavior fluctuation analysis unit, an anomaly probability analysis unit, an error probability analysis unit, and a judgment threshold adjustment unit. The video stream acquisition unit is responsible for collecting real-time video stream data from multiple monitoring points, which serves as the basis for subsequent analysis. The time delay analysis unit processes video stream data from two monitoring points, calculating the time delay value by analyzing the temporal correlation. The behavior fluctuation analysis unit derives the behavior fluctuation factor based on the comparison of deep feature distributions within a short time window. The anomaly probability analysis unit combines feature differences, temporal differences, and the behavior fluctuation factor to calculate the anomaly probability of each feature dimension and identify the anomaly dimension. The error probability analysis unit aggregates the anomaly dimension distributions of all monitoring points to generate behavior error probabilities. The judgment threshold adjustment unit dynamically adjusts the behavior judgment threshold using the behavior error probabilities, achieving automated judgment of video behavior.

[0027] Example 1: See Figure 2The time delay analysis unit determines the time delay value by preprocessing the video stream data from two monitoring points. Preprocessing operations include frame sampling, color space conversion, image size normalization, and pixel value standardization. Frame sampling extracts keyframes from the original video stream at a fixed frequency. The color space is converted from the RGB model to the YUV model to separate luminance and chromaticity information. Image sizes are uniformly adjusted to preset width and height, and pixel values ​​are linearly scaled to a zero-mean, unit-variance distribution. Extracting the deep feature sequence of each frame relies on a deep convolutional neural network model pre-trained on a large image dataset. The model's input is the preprocessed single-frame image, and its output is a high-dimensional feature vector. This feature vector is generated by a global average pooling layer before the last fully connected layer of the model. The feature vector dimension is fixed at 1024 dimensions, with each frame corresponding to a 1024-dimensional feature vector, thus transforming the video stream data into a time-ordered deep feature sequence. The deep feature sequences from two monitoring points are input into a Long Short-Term Memory (LSTM) network for temporal modeling. The LTM network structure consists of an input layer, two stacked hidden layers, and an output layer. The number of nodes in the input layer matches the dimension of the deep feature vectors to 1024. Each hidden layer contains 512 LTM units, and the activation function for the LTM units is the hyperbolic tangent function. The output layer is a fully connected layer that maps the state of the last hidden layer to a hidden state vector of a specified dimension. The LTM network processes the deep feature sequences frame by frame using a forward propagation method. For each time step in the sequence, the LTM network receives the current feature vector and the hidden state and cell state from the previous time step. It updates its internal state through gating mechanisms including input gates, forget gates, and output gates, and outputs the current hidden state. The two deep feature sequences are processed separately through the same LTM network instance, generating two new hidden state sequences. The length of the hidden state sequences is consistent with the original deep feature sequences, and each hidden state vector encapsulates the sequence history information up to the current time step.

[0028] The cosine similarity between two hidden state sequences at different time offsets is calculated. The cosine similarity is calculated for two vectors, using the formula: the dot product of the two vectors divided by the product of their magnitudes. The result ranges from -1 to +1. For a fixed time offset, the hidden state sequences of the first and second monitoring points are slidably aligned along the time axis. For each possible overlap, the cosine similarity between the two corresponding hidden state vectors at that position is calculated. The arithmetic mean of the cosine similarity values ​​calculated for all overlap positions is the overall similarity score at that specific time offset. The search range of the time offset is set according to the physical layout of the monitoring system and the maximum expected latency, for example, from -500 frames to +500 frames. The entire search range is traversed step-by-step in frames. Each time offset corresponds to an overall similarity score, thus generating a similarity curve with the time offset as the horizontal axis and the similarity score as the vertical axis. The peak detection algorithm determines the time offset corresponding to the global peak in the similarity curve as the time delay value. It employs a local comparison-based method, scanning the entire similarity curve and identifying points whose values ​​are greater than all neighboring points and exceed a preset absolute threshold as candidate peak points. The point with the highest similarity score is selected from all candidate peak points as the global peak. The coordinate value of this global peak point on the time offset axis is the calculated time delay value. This time delay value represents the time lag or lead of the video stream data from the second monitoring point relative to the video stream data from the first monitoring point. The sign of the time delay value indicates the order: a positive sign indicates that the video stream data from the second monitoring point lags behind the first, while a negative sign indicates that it leads. The calculated time delay value is stored and used for time alignment of the video stream data in subsequent behavior fluctuation analysis and anomaly probability analysis units.

[0029] The extraction of deep feature sequences requires convolutional neural network models to possess powerful feature representation capabilities. The pre-training process uses millions of labeled images, optimizing network parameters by minimizing classification error, enabling the model to extract high-level features semantically relevant to the image content. Training the Long Short-Term Memory (LSTM) network is a separate process, using a large number of video sequence pairs with known temporal relationships as training samples. The training objective is to minimize the differences between hidden state sequences under correct temporal alignment, and the optimization algorithm employs the Adam optimizer based on gradient descent. The smoothness of the cosine similarity curve is affected by the quality of the hidden state sequences. To reduce random fluctuations, a moving average filter can be applied to the hidden state sequences before calculating the final similarity score. The robustness of peak detection is enhanced by setting a dynamic relative threshold, calculated based on the overall mean and standard deviation of the similarity curve, avoiding misclassification of minor fluctuations caused by noise as valid peaks. The entire workflow design of the time delay analysis unit ensures the accuracy and reliability of time delay value calculation. Preprocessing eliminates the impact of differences in video sources, deep feature extraction captures the essential information of frame content, long short-term memory network temporal modeling uncovers the dynamic correlations between video streams, cosine similarity comparison quantifies the matching degree between sequences, and peak detection precisely locates the optimal alignment point. This time delay value, as a fundamental parameter in the entire monitoring system analysis chain, directly affects the correctness of subsequent calculations of behavior fluctuation factors and anomaly probabilities. Therefore, the implementation of the time delay analysis unit needs to fully consider computational efficiency and noise resistance. In actual deployment, the time delay analysis unit can run periodically to update the time delay values ​​between monitoring points, adapting to changes in time relationships that may be caused by network transmission fluctuations or device clock drift.

[0030] Example 2: The operation of the behavior fluctuation analysis unit to determine the behavior fluctuation factor begins with dividing the video stream data of each monitoring point into short-term windows. The acquisition of short-term windows is based on performing motion intensity detection on the video stream data. Motion intensity detection is achieved by calculating the sum of the magnitudes of the optical flow vectors at the pixel level between consecutive video frames. When the calculated motion intensity value exceeds a dynamic threshold, the system immediately triggers a new short-term window to start recording. The dynamic threshold is not fixed, but is adaptively calculated based on the motion intensity distribution in recent historical video stream data, usually taking a linear combination of the average motion intensity and standard deviation over a past period. The window length is also not fixed. The window length is adaptively adjusted based on the duration of the detected motion event and the feature consistency between frames. The motion duration is determined by the number of frames where the motion intensity continuously exceeds the threshold. The inter-frame consistency is evaluated by comparing the cosine similarity of the depth features of adjacent frames. This mechanism can generate short-term windows of variable length, thereby ensuring that each window can completely cover a relatively independent behavior segment.

[0031] Within each defined short-time window, the system needs to extract multidimensional feature representations from the video stream data of the monitoring points. This extraction is performed synchronously using a multi-task learning framework. This framework shares a basic feature extraction backbone network, which is a deep convolutional neural network. From this shared backbone, the network structure branches into three independent sub-network branches, responsible for extracting appearance features, motion features, and semantic features, respectively. The appearance feature extraction branch captures static visual elements in the video frames, such as the texture, color, and shape information of objects, through further multi-layer convolutional operations. The motion feature extraction branch takes multiple consecutive frames as input and uses 3D convolutional kernels or separate optical flow calculation modules to capture the motion trajectory and speed changes of objects between frames. The semantic feature extraction branch combines scene recognition and object detection models to output high-level semantic information about scene categories, object identities, and the interactions between them. The output feature vectors of the three branches are concatenated to form a comprehensive multidimensional feature representation that fully describes the video content within the short-time window.

[0032] After obtaining the multidimensional feature representations of the current monitoring point and the next monitoring point within a short window, the behavior fluctuation analysis unit uses a discriminator module commonly used in generative adversarial networks (GANs) to calculate the distribution difference score between the two. The discriminator module in the GAN employs a multilayer perceptron structure, where the number of neurons in the input layer matches the dimension of the multidimensional feature representation. The hidden layers consist of several fully connected layers, each followed by a non-linear activation function such as LeakyReLU. The output layer is a single neuron, using the sigmoid activation function to produce a scalar value between 0 and 1. When calculating the distribution difference score, an attention mechanism is introduced within the discriminator module. This mechanism generates a weight vector, which is multiplied element-wise with the input multidimensional feature representation, thereby amplifying the influence of key feature dimensions on the final score while relatively suppressing the contribution of secondary feature dimensions. The attention weights are learned from the input features through a separate shallow neural network. The window feature representation of the current monitoring point is taken as the "real" sample, and the window feature representation of the next monitoring point after time delay adjustment is taken as the "generated" sample. These are input into the discriminator module respectively, and the difference in their output values ​​is the distribution difference score. The larger the score, the more significant the feature distribution difference between the two windows.

[0033] The behavioral volatility analysis unit performs an inversion transformation on the calculated distribution difference score to obtain the final behavioral volatility factor. The inversion transformation uses the natural logarithm function to process the distribution difference score. Specifically, the behavioral volatility factor is calculated using the formula: Behavioral Volatility Factor = -log(Distribution Difference Score + ... )Calculated, where It is a very small positive number used to prevent taking the logarithm of zero. This logarithmic function ensures that values ​​with larger original distribution difference scores correspond to smaller behavior fluctuation factors, while smaller distribution difference scores correspond to larger behavior fluctuation factors, realizing a shift from the concept of "difference" to "consistency of fluctuation." The larger the behavior fluctuation factor value, the more coherent and consistent the behavior patterns of the two monitoring points within the short-term window. The sensitivity of motion intensity detection directly affects the accuracy of short-term window segmentation. The adaptive update cycle of the dynamic threshold is set to be triggered after processing a certain number of video frames. The training of the multi-task learning framework requires a large-scale labeled dataset containing labels at three levels: appearance, motion, and semantics. The training process simultaneously updates the parameters of the shared backbone network and the three branch networks by optimizing a joint loss function. The parameters of the discriminator module in the adversarial generative network need to be pre-trained with a large number of positive and negative sample pairs during the preprocessing stage to enable it to accurately distinguish differences in feature distributions. The weight generation network in the attention mechanism is trained end-to-end with the discriminator module, enabling the network to autonomously discover which feature dimensions are more important for judging behavioral coherence. The logarithmic function in the inversion transform maps the distributional difference score from the original range to a continuous scale that is more in line with the probabilistic interpretation, with the behavioral fluctuation factor serving as a standardized indicator.

[0034] Example 3: See Figure 3 The anomaly probability analysis unit determines the anomaly probability of the video stream data of the monitoring point in each feature dimension based on a detailed comparison of the video stream data of adjacent monitoring points. The analysis process begins with windowing the video stream sequence. For any two monitoring points that are spatially adjacent in the monitoring network, each short-time window divided by the behavior fluctuation analysis unit in the video stream data of the preceding monitoring point is taken. The time delay value calculated by the time delay analysis unit is applied to this short-time window. The application operation is manifested by adding the time delay value to both the start and end timestamps of the window, thereby generating a time delay window that has been shifted on the time axis. The positioning of the time delay window aims to predict the time interval in which the behavior segment captured by the preceding monitoring point will appear in the video stream of the following monitoring point after network transmission and processing delays. All video frames that overlap with the aforementioned time delay window on the time axis are extracted from the video stream data of the following monitoring point. These frames are arranged in chronological order to form a combined window. The boundaries of the combined window are determined by the intersection of the delay window and the effective time range of the video stream data from the next monitoring point. The starting frame of the combined window is the first frame in the video stream from the next monitoring point whose timestamp is not less than the start timestamp of the delay window, and the ending frame of the combined window is the last frame in the video stream from the next monitoring point whose timestamp is not greater than the end timestamp of the delay window. The frame sequence of the combined window represents the observation data that is expected to correspond to the behavior described by the short-time window of the previous monitoring point.

[0035] Calculating the difference between short-time windows and combined windows at the feature level requires an encoder-decoder structure. The encoder in this structure is a deep neural network whose input is a sequence of numerical values ​​for all video frames within a window along a specific feature dimension. The encoder maps the input sequence to a low-dimensional latent space vector through a nonlinear transformation, capturing a compressed representation of the input sequence. The decoder is another deep neural network that takes the latent space vector output by the encoder as input. The decoder's goal is to reconstruct the original input sequence. The reconstruction error is calculated by comparing the reconstructed sequence output by the decoder with the original input sequence, using the mean squared error function. The latent representation distance is measured by the Euclidean distance between the latent space vectors corresponding to the short-time window and the combined window. Since the reconstruction error and latent representation distance are two quantities with different physical meanings, they need to be dimensionless to be integrated into a unified and clearly interpretable metric. The reconstruction error is divided by its maximum value calculated on a representative training dataset, and the latent representation distance is divided by its maximum value calculated on the same training dataset. This normalization operation converts both quantities into dimensionless values ​​in the range of zero to one. The normalized reconstruction error and the latent representation distance are fused into a fluctuation recovery coefficient by weighted summation. The formula for calculating the fluctuation recovery coefficient is as follows: Where: symbol Indicates the first On the feature dimension, for the first The fluctuation recovery coefficient is calculated using a short-time window. (Symbol) Indicates the first On the feature dimension, the first Reconstruction error corresponding to a short-time window. Symbol Indicates the first The maximum value of the reconstruction error obtained statistically from historical training data across each feature dimension. (Symbol) Indicates the first On the feature dimension, the first The Euclidean distance between the latent representations of a short-time window and the latent representations of the corresponding combined window. (Symbol) Indicates the first The maximum value of the latent representation distance obtained statistically from historical training data across each feature dimension. (Symbol) and These are weighting coefficients, satisfying... The conditions are used to adjust the relative importance of normalized reconstruction error and normalized latent representation distance in the fluctuation recovery coefficient. Both terms on the right-hand side of the formula are dimensionless ratios, and their weighted sum... It is also a dimensionless value, consistent with the dimension of the wave recovery coefficient on the left side of the equation.

[0036] After the fluctuation recovery coefficient is calculated, the anomaly probability analysis unit combines it with the behavior fluctuation factor provided by the behavior fluctuation analysis unit. For each feature dimension and each short-term window, the ratio of the behavior fluctuation factor to the fluctuation recovery coefficient is calculated. The behavior fluctuation factor is also a dimensionless quantity, so this ratio is also dimensionless. This ratio reflects the relative magnitude between the expected behavior fluctuation and the actual observed feature recovery difficulty. The ratio is normalized to the range of zero to one using the min-max scaling method. The normalization process is based on the ratios of all windows and all feature dimensions within the current analysis batch. The normalized ratio is compared with a preset constant parameter, which serves as a benchmark reference value. The absolute difference between the normalized ratio and the constant parameter is calculated. The arithmetic mean of the differences calculated for all short-term windows in the video stream data of the current monitoring point is then calculated. The result of the arithmetic mean is the anomaly probability of the video stream data of that monitoring point in the current feature dimension. The closer the anomaly probability value is to one, the higher the probability of an abnormal pattern occurring in that feature dimension. By iterating through all feature dimensions and repeating the above calculation process, a vector containing the anomaly probabilities of all feature dimensions is finally generated. This probability vector provides a direct input basis for the error probability analysis unit to identify anomaly dimensions.

[0037] Example 4: The anomaly probability analysis unit determines the fluctuation recovery coefficient for each feature dimension within each short-term window. This involves learning and fusing feature changes and temporal changes. The calculation of the fluctuation recovery coefficient relies on a neural network modeling the difference between the combined window and the short-term window. A specially designed feedforward neural network learns the feature changes, which characterize the relative difference between the combined window and the short-term window in static feature values. The input to the feedforward neural network is the ratio of feature values ​​of the combined window and the short-term window in the same feature dimension; this ratio is a scalar. The feedforward neural network contains two fully connected layers. The first fully connected layer maps the input ratio to a higher-dimensional hidden space, and the second fully connected layer maps the hidden representation back to a scalar output. A nonlinear activation function, such as the ReLU function, is used between the two fully connected layers to introduce nonlinear transformation capability. The output of the feedforward neural network is compressed by a sigmoid activation function, limiting the numerical range to between zero and one. This normalized output is the learned feature change.

[0038] A recurrent neural network (RNN) is used to learn temporal variability, which captures the phase difference between combined and short-term windows in dynamic temporal patterns. The RNN employs a gated recurrent unit (ROU) structure, which effectively processes sequential data and captures long-term dependencies. The calculation of temporal variability requires phase information from the input sequence, extracted from the feature value sequence via Fast Fourier Transform (FFT). For the feature value sequences of the short-term and combined windows, FFTs are performed separately to obtain their respective frequency domain representations. The phase angles of the main frequency components are extracted from these frequency domain representations as the phase information sequence. The gated RNU uses this phase information sequence as input, and the hidden state at the last time step of the gated RNU serves as a general representation of the entire sequence. A temporal attention mechanism is introduced into the output of the gated RNU, calculating the attention weights of the hidden states at each time step. These attention weights are learned through a small fully connected network, reflecting the varying importance of phase information at different time points to the overall temporal variability. The weighted sum of the hidden states is then passed through a fully connected layer and a sigmoid activation function, outputting a temporal variability ranging from zero to one.

[0039] The feature variation and temporal variation are weighted and fused to output the final fluctuation recovery coefficient. This weighted fusion is performed in a fusion layer that receives both feature variation and temporal variation as input. The feature variation and temporal variation are concatenated into a two-dimensional vector, which is then linearly transformed through a fully connected layer. This fully connected layer has two output neurons, corresponding to the fusion weights for the feature variation and temporal variation, respectively. These weights are normalized using a softmax function so that the sum of the two weights is one. The normalized weights are then multiplied by the original feature variation and temporal variation, respectively. The sum of these products yields the initial value of the fluctuation recovery coefficient. This initial value may exceed the range of zero to one; therefore, a sigmoid activation function is used to constrain the fluctuation recovery coefficient to the standard range of zero to one. The weights for the feature variation and temporal variation are not fixed; they are dynamically generated by the network based on the input data. This dynamic weighting mechanism allows the fusion process to adapt to different feature dimensions and window characteristics.

[0040] The anomaly probability analysis unit derives the anomaly dimension from the anomaly probabilities across all feature dimensions based on the video stream data from the monitoring points. This requires statistical learning methods, and the identification of anomaly dimensions is built upon the analysis of the anomaly probability distribution for each feature dimension. Density estimation is performed on the anomaly probability of each feature dimension to generate a probability distribution function. Kernel density estimation, a non-parametric method, is used and does not assume that the data follows a distribution. A smooth kernel function is chosen for kernel density estimation; a Gaussian kernel function is commonly used. The kernel function centers on each data point, and its bandwidth parameter controls the smoothness. The probability distribution function is generated by superimposing the kernel functions at each data point. The probability distribution function describes the probability density of different values ​​of the anomaly probability, and the area under the curve of the probability distribution function is 1. The shape of the probability distribution function reflects the central tendency and dispersion of the anomaly probability along that feature dimension.

[0041] Referring to Table 1, the quantile at the tail of the probability distribution function is selected as the adaptive threshold. A quantile is a point on the probability distribution function, and the probability value corresponding to this point is equal to the specified quantile level. For anomaly detection, a higher quantile level, such as the 95th percentile, is usually chosen, meaning that 95% of the anomaly probability values ​​are below this threshold. The adaptive threshold is determined based on the anomaly probability distribution of each feature dimension. Different feature dimensions will have different adaptive thresholds, which can adapt to the inherent probability distribution differences of different feature dimensions. Feature dimensions whose anomaly probability exceeds the adaptive threshold of their corresponding feature dimension are marked as anomaly dimensions. The marking operation generates a binary indicator vector, where each element corresponds to a feature dimension. A value of 1 indicates that the dimension is an anomaly dimension, and a value of zero indicates that the dimension is a normal dimension. The feature change learning network and the temporal change learning network need to be jointly trained, and the training data consists of a large number of video clips with known behavioral patterns. The fusion weight learning of the fluctuation recovery coefficient is part of the end-to-end training, and the training objective is to minimize the difference between the fluctuation recovery coefficient and the true behavioral continuity label. The choice of the bandwidth parameter for kernel density estimation affects the smoothness of the probability distribution function, and the bandwidth parameter is usually determined through cross-validation. The quantile level of the adaptive threshold is a system-level hyperparameter, and its setting needs to strike a balance between sensitivity and false alarm rate. The labeling results of the anomaly dimensions are a key input to the subsequent error probability analysis unit; the set of anomaly dimensions indicates the feature components in video behavior most likely to deviate from the normal pattern.

[0042] Table 1: Anomaly Probability and Adaptive Threshold of Feature Dimensions The table displays data for five feature dimensions. Feature dimension numbers distinguish different features, and the anomaly probability is a value calculated by the anomaly probability analysis unit. The 95th percentile of the kernel density estimate is an adaptive threshold for the anomaly probability distribution of that feature dimension. Whether a dimension is marked as an anomalous is a binary judgment based on whether the anomaly probability exceeds the adaptive threshold. The table shows that feature dimensions 2 and 4 have anomaly probabilities exceeding their respective adaptive thresholds and are therefore marked as anomalous. Feature dimensions 1, 3, and 5 have anomaly probabilities below their thresholds and are considered normal dimensions. This adaptive threshold-based marking method allows the system to flexibly adapt to the inherent statistical characteristics of different feature dimensions.

[0043] Example 5: The process of obtaining the behavioral error probability based on the distribution of abnormal dimensions at all adjacent monitoring points by the error probability analysis unit begins with the aggregation of global information. Assume a monitoring network contains four monitoring points: A, B, C, and D, arranged sequentially along a path. The abnormal probability analysis unit has analyzed the video stream data of each monitoring point and output an abnormal dimension set. For example, the abnormal dimension set of monitoring point A includes dimensions 2 and 7; the abnormal dimension set of monitoring point B includes dimensions 2, 5, and 11; the abnormal dimension set of monitoring point C includes dimensions 5, 7, and 13; and the abnormal dimension set of monitoring point D includes dimensions 2 and 13. Collecting the abnormal dimension sets of all monitoring points constitutes a global abnormal dimension occurrence record. A graph neural network is used for community detection to identify frequently occurring abnormal dimensions as key feature dimensions. The graph neural network is constructed with feature dimensions as nodes, and the adjacency relationships between monitoring points define the edges of the graph. Each feature dimension node is initialized with a feature vector containing statistical information such as the frequency and location of that dimension across all monitoring points. The message passing mechanism of the graph neural network allows nodes to exchange information, and connected nodes gradually acquire similar representations. After multi-layer graph convolution operations, node features are updated. Community detection algorithms, such as the Louvain method, are applied to the updated node features. These algorithms group nodes into several communities, with nodes within each community being closely connected. The community containing the most nodes or having the highest node centrality is identified; the feature dimensions within this community are considered high-frequency, interconnected key feature dimensions. In the example above, dimension 2 appears at monitoring points A, B, and D; dimension 5 appears at monitoring points B and C; dimension 7 appears at monitoring points A and C; and dimension 13 appears at monitoring points C and D. The graph neural network might identify dimensions 2, 5, 7, and 13 as a community and determine them as key feature dimensions for the monitoring network in the current time period.

[0044] For each monitoring point, the divergence index of its video stream data feature value sequence across key feature dimensions is calculated. Taking monitoring point A as an example, the video stream data of monitoring point A has a feature value sequence for each key feature dimension, and each sequence contains the change of the feature value in that dimension over a period of time. The divergence index measures the distribution difference between the actual feature value sequence of monitoring point A and an expected normal sequence, which can be derived from a distribution modeled from historical normal behavior data. The Jensen-Shannon divergence is chosen for the calculation of the divergence index. The Jensen-Shannon divergence is symmetric and takes values ​​between zero and one. For each key feature dimension, the Jensen-Shannon divergence between the actual sequence distribution and the expected sequence distribution is calculated separately. The divergence index of monitoring point A is the average of the Jensen-Shannon divergences calculated across all key feature dimensions. This average reflects the degree to which monitoring point A deviates from the normal pattern overall across key features dimensions. The divergence indices of monitoring points B, C, and D are calculated using the same method. The behavior error probability is obtained by performing min-max scaling on the divergence indices of all monitoring points. Min-max scaling is a linear transformation method that maps the original data to the interval between zero and one. A threshold adjustment unit adjusts the behavior judgment threshold based on the behavior error probability to achieve automatic classification of video behavior. The behavior judgment threshold adjustment is performed for each monitoring point. The behavior error probability of each monitoring point is input into a fully connected neural network, which has one input layer, two hidden layers, and one output layer. The input layer receives a scalar value, i.e., the behavior error probability. The first hidden layer contains multiple neurons using the ReLU activation function, the second hidden layer has fewer neurons, and the output layer has one neuron using a linear activation function, outputting the threshold adjustment coefficient. The parameters of the fully connected neural network are trained to learn the complex nonlinear relationship between the behavior error probability and the threshold adjustment amount. The threshold adjustment coefficient is a scaling factor. Multiplying the threshold adjustment coefficient by the historical judgment threshold yields the temporary judgment threshold for the current monitoring point. The historical judgment threshold is the threshold used in the previous cycle of the system or a base threshold. The temporary judgment threshold reflects the local judgment criterion based on the current behavior error probability.

[0045] A globally unified actual judgment threshold is generated by weighted averaging of temporary judgment thresholds for all monitoring points. The weights of the average can be assigned based on the importance of the monitoring points, which can be set according to the location of the monitoring points, the amount of historical data, or other prior knowledge. The actual judgment threshold serves as the final critical value for binary classification of video behavior. The comprehensive score obtained from the behavior analysis of the monitoring point video stream data is compared with the actual judgment threshold. If the comprehensive score is higher than the actual judgment threshold, it is judged as abnormal behavior; if the comprehensive score is lower than or equal to the actual judgment threshold, it is judged as normal behavior. This dynamic threshold adjustment mechanism allows the system to adapt to changes in overall behavior patterns, appropriately relaxing the judgment criteria when the global behavior error is large and tightening the criteria when the global behavior error is small, thus maintaining the stability of the judgment. The community detection results of the graph neural network depend on the initialization of the network structure and node features; the identification of key feature dimensions is a dynamic process. The divergence index is calculated by selecting the absolute value that affects the probability of behavior error, but the minimum and maximum scaling maintain a relative relationship. Training the fully connected neural network requires a large amount of labeled data to learn how the probability of behavior error affects the optimal judgment threshold. The weighted averaging to generate the actual judgment threshold reflects the idea of ​​centralized decision-making, balancing local anomalies and the global state. The final binary classification judgment is a crucial step in the monitoring system's output, and the actual judgment threshold is the core parameter for achieving automated judgment. The entire process starts with the distribution of anomalies, and through graph analysis, divergence calculation, probability scaling, and neural network adjustments, it ultimately reaches an actionable judgment threshold, completing the transformation from data features to decision instructions.

[0046] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A monitoring system for judging video behavior based on deep learning, characterized in that, The system includes: The video stream acquisition unit is used to acquire video stream data from multiple monitoring points; The time delay analysis unit is used to determine the time delay value between the video stream data of two monitoring points based on the timing correlation between the video stream data of two monitoring points. The behavior fluctuation analysis unit is used to determine the behavior fluctuation factor of the monitoring point in each short window based on the depth feature distribution of the video stream data of the monitoring point in the short window and the depth feature distribution of the video stream data of the next monitoring point after time delay adjustment in the short window. The anomaly probability analysis unit is used to determine the anomaly probability of the video stream data of the monitoring point in each feature dimension based on the feature differences, temporal differences, and behavioral fluctuation factors of the video stream data contained in all short-time windows of adjacent monitoring points in the deep feature space; and to obtain the anomaly dimension based on the anomaly probability of the video stream data of the monitoring point in all feature dimensions. The error probability analysis unit is used to obtain the behavior error probability based on the distribution of the anomaly dimension at all adjacent monitoring points; The threshold adjustment unit is used to adjust the behavior judgment threshold according to the behavior error probability to complete the automatic judgment of video behavior; The anomaly probability analysis unit determines the anomaly probability of the video stream data at the monitoring point in each feature dimension, including: For any two adjacent monitoring points, the time delay window is obtained by applying a time delay value to each short window of the video stream data of the previous monitoring point. Extract all frame sequences that overlap with the time delay window from the video stream data of the next monitoring point to form a combined window; The encoder-decoder architecture is used to calculate the reconstruction error and potential representation distance of each short-time window and the combined window under the same feature dimension. The reconstruction error and the latent representation distance are fused into a fluctuation recovery coefficient; The ratio of the behavioral volatility factor to the volatility recovery coefficient is normalized and then compared with a constant parameter to obtain the difference. The average of the differences over all short-term windows is taken as the anomaly probability. The normalized reconstruction error and the latent representation distance are fused together by a weighted sum to obtain the volatility recovery coefficient. The formula for calculating the volatility recovery coefficient is as follows: ； Among them, symbols Indicates the first On the feature dimension, for the first The fluctuation recovery coefficient calculated within a short-time window, sign... Indicates the first On the feature dimension, the first The reconstruction error corresponding to each short-time window, sign Indicates the first The maximum value of the reconstruction error obtained from statistical analysis of historical training data across each feature dimension, sign... Indicates the first On the feature dimension, the first The Euclidean distance between the latent representations of a short-time window and the latent representations of the corresponding combined window, denoted as Euclidean distance. Indicates the first The maximum value of the latent representation distance obtained statistically from historical training data across each feature dimension, and the symbol and These are weighting coefficients, satisfying... The conditions are used to adjust the relative importance of normalized reconstruction error and normalized latent representation distance in the fluctuation recovery coefficient.

2. The monitoring system for judging video behavior based on deep learning as described in claim 1, characterized in that, The time delay analysis unit determines the time delay value by including: The video stream data from the two monitoring points are preprocessed to extract the depth feature sequence of each frame; Two deep feature sequences are input into a long short-term memory network for temporal modeling, and the hidden state sequence is output. Calculate the cosine similarity of two hidden state sequences at different time offsets and generate similarity curves; By finding the global peak in the similarity curve, the time offset corresponding to the peak is determined as the time delay value.

3. The monitoring system for judging video behavior based on deep learning as described in claim 2, characterized in that, The behavioral fluctuation analysis unit determines the behavioral fluctuation factors, including: Extract multidimensional feature representations from the video stream data of each monitoring point within a short time window; Using the discriminator module in the generative adversarial network, calculate the distribution difference score between the multidimensional feature representations of the current monitoring point and the next monitoring point; The behavioral fluctuation factor is obtained by inverting the distribution difference score.

4. The monitoring system for judging video behavior based on deep learning as described in claim 3, characterized in that, The method for obtaining the short window is as follows: Motion intensity is detected for each video stream, and the window is triggered when the motion intensity exceeds the dynamic threshold. The window length is adaptively adjusted based on motion duration and inter-frame consistency to generate a short-time window of variable length.

5. The monitoring system for judging video behavior based on deep learning as described in claim 4, characterized in that, The determination of the fluctuation recovery coefficient for each feature dimension within each short-time window includes: The feature change is learned through a neural network. This feature change is obtained by passing the ratio of the feature values ​​of the combined window and the short-time window through a non-linear activation function. The temporal variation is learned through a recurrent neural network. This temporal variation is obtained by using the phase information of the combined window and the short window through a temporal attention mechanism. The characteristic change and the time series change are weighted and fused to output the fluctuation recovery coefficient.

6. The monitoring system for judging video behavior based on deep learning as described in claim 1, characterized in that, The anomaly probability analysis unit obtains the anomaly dimensions based on the anomaly probabilities of the video stream data from the monitoring points across all feature dimensions. Density estimation is performed on the anomaly probability of each feature dimension to generate a probability distribution function; Select the quantile at the tail of the probability distribution function as the adaptive threshold; Feature dimensions whose anomaly probability exceeds an adaptive threshold are marked as anomaly dimensions.

7. The monitoring system for judging video behavior based on deep learning as described in claim 1, characterized in that, The error probability analysis unit obtains the behavioral error probability based on the distribution of anomaly dimensions at all adjacent monitoring points, including: Collect the set of anomaly dimensions from all monitoring points, use graph neural networks to perform community detection, and identify the frequently occurring anomaly dimensions as key feature dimensions. For each monitoring point, calculate the divergence index of its video stream data feature value sequence on the key feature dimension; The probability of behavioral error is obtained by minimax scaling of the divergence index of all monitoring points.

8. The monitoring system for judging video behavior based on deep learning as described in claim 1, characterized in that, The judgment threshold adjustment unit adjusts the behavior judgment threshold according to the behavior error probability, including: The probability of behavioral error at each monitoring point is input into a fully connected neural network, which outputs a threshold adjustment coefficient. The temporary threshold is obtained by multiplying the threshold adjustment factor by the historical judgment threshold. The temporary thresholds of all monitoring points are weighted and averaged to generate the actual judgment threshold. The video behavior is classified into two categories based on the actual judgment threshold.

9. The monitoring system for judging video behavior based on deep learning as described in claim 3, characterized in that, When extracting multidimensional feature representations in the behavior fluctuation analysis unit, a multi-task learning framework is used to simultaneously extract appearance features, motion features, and semantic features; the discriminator module in the adversarial generative network adopts a multilayer perceptron structure, and an attention mechanism is introduced when calculating the distribution difference score to focus on key feature dimensions; The inversion transformation uses a logarithmic function to process the distribution difference score, resulting in a behavior fluctuation factor.

Images