AI-based security monitoring video analysis method and system

Abstract

The application provides an AI-based security monitoring video analysis method and system, which first acquires an initial video data set corresponding to a monitoring video stream to be analyzed, the initial video data set containing multiple video segment units that are continuously collected and have time stamp markers; then performs video feature extraction processing on the initial video data set to obtain time sequence behavior features of a target object and spatial structure features of a scene where the target object is located; then calls a pre-constructed abnormal behavior detection model to jointly detect the two kinds of features to generate an abnormal behavior detection result; determines an abnormal event type and spatiotemporal distribution feature information in a video picture according to the abnormal behavior detection result; and finally generates a security warning instruction containing event positioning coordinates and sends the security warning instruction to a target security control terminal to trigger a linkage response operation, thereby improving the accuracy of abnormal behavior detection, and realizing accurate positioning and rapid warning of abnormal events.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to an AI-based security surveillance video analysis method and system. Background Technology

[0002] In modern society, security monitoring systems play a vital role in maintaining public safety, preventing crime, and ensuring the security of important facilities. Traditional security monitoring mainly relies on manual observation of surveillance videos in real time or for post-event playback. This method not only consumes a lot of manpower, but also, because human attention is difficult to maintain at a high level for extended periods, it can easily lead to the omission of important security incidents and make it impossible to detect and handle abnormal situations in a timely and effective manner.

[0003] With the continuous development of video surveillance technology, the amount of surveillance video data is growing explosively, and traditional manual monitoring methods can no longer meet practical needs. To solve this problem, automatic video analysis methods based on computer vision and image processing technologies have emerged. However, most existing automatic video analysis methods only focus on single features of target objects in the video, such as extracting only the appearance or motion features of the target object. They lack comprehensive analysis of the target object's behavioral features and the spatial structure features of the scene, resulting in low accuracy in detecting abnormal behavior, inability to accurately identify complex abnormal event types, and inability to determine the spatiotemporal distribution characteristics of abnormal events in the video frame. Consequently, they fail to meet the needs of rapid and accurate early warning and location of abnormal events in actual security monitoring. Summary of the Invention

[0004] In view of the aforementioned problems, and in conjunction with the first aspect of the present invention, embodiments of the present invention provide an AI-based security surveillance video analysis method, the method comprising:

[0005] Obtain the initial video data set corresponding to the monitoring video stream to be analyzed. The initial video data set contains multiple video segment units with timestamps collected continuously.

[0006] The initial video dataset is subjected to video feature extraction processing to obtain the temporal behavior features of the target object in the video segment unit and the spatial structure features of the scene where the target object is located.

[0007] A pre-built abnormal behavior detection model is invoked to perform joint abnormal behavior detection on the temporal behavior features and the spatial structure features, thereby generating abnormal behavior detection results for the target object in the video segment unit;

[0008] Based on the abnormal behavior detection results, determine the types of abnormal events existing in the monitoring video stream to be analyzed and the spatiotemporal distribution characteristics of the abnormal events in the video frame;

[0009] Based on the abnormal event type and the spatiotemporal distribution feature information, a security warning instruction containing the event location coordinates is generated, and the security warning instruction is sent to the target security control terminal to trigger a linkage response operation.

[0010] For example, in one possible implementation of the first aspect, after sending the security warning command to the target security control terminal to trigger the linkage response operation, the method further includes:

[0011] Receive instruction execution status feedback data packet returned by the target security control terminal, wherein the instruction execution status feedback data packet includes device response timestamp and execution result status code;

[0012] Parse the device response delay time and operation success identifier in the instruction execution status feedback data packet;

[0013] The execution efficiency evaluation index of the security early warning command is calculated based on the device response delay time and operation success identifier;

[0014] When the execution performance evaluation index is lower than a preset threshold, a strategy optimization request message containing the abnormal event type and the execution performance evaluation index is generated;

[0015] The strategy optimization request message is sent to the cloud management platform to trigger the online update process of the security response strategy, and the updated set of device linkage rules is received from the cloud management platform.

[0016] For example, in one possible implementation of the first aspect, before obtaining the initial video data set corresponding to the surveillance video stream to be analyzed, the method further includes:

[0017] Receive raw video stream data uploaded in real time from multiple security monitoring devices, wherein the raw video stream data includes video files with different resolutions and encoding formats;

[0018] The original video stream data is subjected to resolution unification processing, and video frames of different resolutions are adjusted to a preset standard resolution size;

[0019] The original video stream data after unification is subjected to keyframe sampling to remove temporally and spatially redundant video frame units.

[0020] The original video stream data after redundancy removal is time-stamped and then the time-stamped video stream data is segmented and spliced ​​in chronological order to generate the initial video data set containing complete temporal relationships.

[0021] For example, in one possible implementation of the first aspect, the keyframe sampling process performed on the unified raw video stream data to remove temporally and spatially redundant video frame units includes:

[0022] Calculate the image content difference index between adjacent video frame units, which is determined based on the pixel grayscale value change rate and texture feature similarity.

[0023] Based on the image content difference index, a keyframe sampling decision model is constructed to determine the set of keyframe candidate positions in the original video stream data;

[0024] The keyframe candidate location set is subjected to time window smoothing filtering, and video frame units with significant content changes are extracted as the keyframe set based on the keyframe candidate location set after smoothing filtering.

[0025] Video frame units not included in the keyframe set are marked as redundant data and removed to complete the compression and optimization processing of the video stream data.

[0026] In another aspect, embodiments of the present invention also provide an AI-based security monitoring video analysis system, including a processor and a machine-readable storage medium connected to the processor. The machine-readable storage medium is used to store programs, instructions, or code, and the processor is used to execute the programs, instructions, or code in the machine-readable storage medium to implement the above-described method.

[0027] Based on the above, this embodiment of the invention acquires an initial video data set containing multiple video segments continuously acquired and time-stamped, performs comprehensive and in-depth feature extraction on the video segment units, and simultaneously acquires the temporal behavior features of the target object and the spatial structure features of the scene. It makes full use of the multifaceted information in the video data, calls a pre-built abnormal behavior detection model to jointly detect abnormal behavior based on the above two features, and can comprehensively consider the behavior and scene information of the target object, greatly improving the accuracy and reliability of abnormal behavior detection. Based on the abnormal behavior detection results, it determines the type of abnormal event and its spatiotemporal distribution features in the video frame, and generates a security early warning command containing the event location coordinates, realizing accurate positioning and rapid early warning of abnormal events, enabling the target security control terminal to trigger linkage response operations in a timely manner, effectively improving the intelligence level of the security monitoring system and its ability to respond to abnormal events. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the execution flow of the AI-based security monitoring video analysis method provided in the embodiments of the present invention.

[0029] Figure 2 This is a schematic diagram of exemplary hardware and software components of the AI-based security monitoring video analysis system provided in this embodiment of the invention. Detailed Implementation

[0030] The present invention will now be described in detail with reference to the accompanying drawings. Figure 1 This is a flowchart illustrating an AI-based security monitoring video analysis method according to an embodiment of the present invention. The following is a detailed description of the AI-based security monitoring video analysis method.

[0031] Step S110: Obtain the initial video data set corresponding to the monitoring video stream to be analyzed. The initial video data set contains multiple video segment units with timestamps that are continuously collected.

[0032] This embodiment aims to analyze security surveillance video. In practical applications, such as security monitoring systems in large parks, multiple surveillance cameras distributed in different locations continuously collect video stream data. The video stream data collected by these cameras is aggregated, and each video segment is assigned a timestamp to ensure that the data has continuous time-series information. For example, cameras at the park entrance record the entry and exit of people and vehicles; these video segments all have precise timestamps, facilitating subsequent tracing and analysis.

[0033] Before step S110, a series of preprocessing operations need to be performed on the raw video stream data to ensure the quality and consistency of the data, which will facilitate subsequent analysis and processing.

[0034] For example, step S210: Receive raw video stream data uploaded in real time by multiple security monitoring devices, wherein the raw video stream data includes video files with different resolutions and encoding formats.

[0035] In large-scale park security monitoring scenarios, the raw video streams uploaded by various security monitoring devices have different resolutions and encoding formats due to differences in brand, model, and settings. For example, high-definition monitoring devices on the main roads of the park may upload videos with higher resolution and specific encoding formats, while ordinary monitoring devices in some corners may use lower resolution and different encoding formats. These differences can cause difficulties in subsequent unified processing, thus requiring further processing.

[0036] Step S220: Perform resolution unification processing on the original video stream data, adjusting video frames of different resolutions to a preset standard resolution size.

[0037] To facilitate subsequent analysis and processing, video frames with different resolutions need to be uniformly adjusted to a preset standard resolution size. Let the resolution of the original video frame be R_ori, containing horizontal pixels W_ori and vertical pixels H_ori, and the preset standard resolution be R_std, containing horizontal pixels W_std and vertical pixels H_std. Bilinear interpolation can be used for resolution adjustment. For each pixel P_ori(x_ori, y_ori) in the original video frame, find the corresponding pixel P_std(x_std, y_std) in the adjusted standard resolution video frame. By calculating the relative positional relationships between pixels, and based on the principle of bilinear interpolation, obtain the color values ​​from the adjacent pixels of the original video frame and perform a weighted average to obtain the adjusted pixel's color value. In this way, all video frames are adjusted to a uniform standard resolution, ensuring data consistency.

[0038] Step S230: Perform keyframe sampling processing on the unified raw video stream data to remove temporally and spatially redundant video frame units.

[0039] The purpose of keyframe sampling is to remove temporal and spatial redundancy from video stream data, reducing the amount of data while retaining important information. The specific implementation process is as follows:

[0040] Step S231: Calculate the image content difference index between adjacent video frame units. The image content difference index is determined based on the pixel gray value change rate and texture feature similarity.

[0041] For two adjacent video frames F_i and F_i+1, they are first converted into grayscale images, resulting in grayscale frames G_i and G_i+1. The grayscale value of each pixel in each grayscale frame is calculated. Let the grayscale value of pixel P(x, y) in grayscale frame G_i be g_i(x, y), and the grayscale value of the corresponding pixel in grayscale frame G_i+1 be g_i+1(x, y). The rate of change of pixel grayscale values ​​can be measured by summing the absolute values ​​of the differences in grayscale values ​​between corresponding pixels in adjacent grayscale frames, i.e., the rate of change of pixel grayscale values ​​Δg = Σ|g_i+1(x, y) - g_i(x, y)|, where Σ represents the summation over all pixels.

[0042] For calculating texture feature similarity, the Local Binary Pattern (LBP) algorithm can be used. LBP features are calculated for grayscale frames G_i and G_i+1 respectively, resulting in LBP feature maps L_i and L_i+1. The similarity between these two LBP feature maps can then be calculated using the histogram intersection method. Let the histograms of L_i and L_i+1 be H_i and H_i+1, respectively. The texture feature similarity S_t can be obtained by calculating the ratio of the intersection area of ​​the two histograms to the total area, i.e., S_t = Σmin(H_i(k), H_i+1(k)) / Σ(H_i(k) + H_i+1(k)), where k represents the bin index of the histogram.

[0043] Finally, the pixel grayscale value change rate and texture feature similarity are combined to obtain the image content difference index D. A weighted summation method can be used, where the weight of the pixel grayscale value change rate is α, the weight of the texture feature similarity is β, and α + β = 1, then D = α * Δg + β * (1 - S_t).

[0044] Step S232: Construct a keyframe sampling decision model based on the image content difference index to determine the set of keyframe candidate positions in the original video stream data.

[0045] Based on the calculated image content difference index D, a keyframe sampling decision model can be constructed. A threshold T is set; if the image content difference index D between adjacent video frames is greater than the threshold T, then a significant content change is considered to have occurred between the two video frames, and the latter video frame can be considered a keyframe candidate. The entire video stream data is traversed to find all keyframe candidate positions that meet the conditions, forming a keyframe candidate position set C.

[0046] Step S233: Perform time window smoothing filtering on the key frame candidate position set, and extract video frame units with significant content changes as key frame set based on the key frame candidate position set after smoothing filtering.

[0047] To avoid overly dense or sparse keyframe candidate positions, a time window smoothing filter is applied to the keyframe candidate position set C. A time window size W is defined. For each position c_i in the keyframe candidate position set C, with c_i as the center, the number of keyframe candidate positions within the time window [max(1, c_i-W / 2), min(N, c_i+W / 2)] (where N is the total number of video frames) is counted. If the number exceeds a set threshold, only the keyframe candidate position with the highest content difference index within the window is retained. After the smoothing filter, the processed keyframe candidate position set C' is obtained. The corresponding video frame units are extracted from the processed keyframe candidate position set C' to form the keyframe set K.

[0048] Step S234: Mark video frame units not included in the keyframe set as redundant data and remove them to complete the compression and optimization processing of the video stream data.

[0049] After obtaining the keyframe set K, video frame units in the original video stream data that are not included in the keyframe set K are marked as redundant data. By deleting these redundant data, compression optimization of the video stream data is achieved, reducing data storage space and the computational load of subsequent processing.

[0050] Step S240: Perform timestamp synchronization processing on the original video stream data after redundancy removal, and then perform segment splicing processing on the video stream data after timestamp synchronization processing according to time order to generate the initial video data set containing complete temporal relationships.

[0051] After removing redundant data, the video stream data needs to be timestamped. Since the clocks of different monitoring devices may have some error, the timestamps of the video stream data may be inconsistent. A method based on the Network Time Protocol (NTP) can be used to synchronize the clocks of all monitoring devices to a unified time base. Then, the timestamps of each video frame unit are calibrated to ensure they have accurate time sequence.

[0052] The timestamped video stream data is segmented and spliced ​​sequentially. Adjacent video frames are connected based on their timestamps to form video segment units with a complete temporal relationship. These video segment units constitute the initial video data set, which includes multiple timestamped video segment units acquired consecutively.

[0053] Step S120: Perform video feature extraction processing on the initial video data set to obtain the temporal behavior features of the target object in the video segment unit and the spatial structure features of the scene where the target object is located.

[0054] After obtaining the initial video dataset, video feature extraction processing is required to acquire the temporal behavioral features of the target object and the spatial structural features of the scene in which the target object is located. These features are crucial for subsequent abnormal behavior detection and abnormal event analysis. The specific implementation process is as follows:

[0055] Step S121: Perform frame segmentation on the initial video data set to obtain multiple video frame units with a continuous time sequence relationship.

[0056] The initial video dataset consists of multiple video segment units, each of which can be further decomposed into multiple video frame units. By performing frame segmentation on the initial video dataset, the video segments are broken down into a series of video frame units with a continuous time sequence relationship. For example, in park surveillance video, each video segment is broken down into individual frames to facilitate subsequent analysis of each frame.

[0057] Step S122: Perform dynamic information capture processing on the video frame unit to generate motion trajectory change features of the target object between adjacent video frame units as temporal behavior features. The temporal behavior features include motion trajectory direction consistency parameters and periodic action continuity indicators.

[0058] Dynamic information capture and processing are performed on video frame units to generate temporal behavioral features of the target object. The specific process is as follows:

[0059] Step S1221: Extract the pixel change region between adjacent video frame units and calculate the optical flow vector distribution characteristics of the pixel change region.

[0060] For two adjacent video frames F_i and F_i+1, the regions where pixel values ​​have changed are identified by comparing their pixel values. A frame difference method can be used, calculating the absolute value of the difference between corresponding pixels in the two video frames. If the absolute value of the difference is greater than a certain threshold, the region containing that pixel is considered to have changed. For regions with pixel changes, optical flow is used to calculate optical flow vectors. Optical flow is based on the motion information of pixels in the image sequence. By solving the constant brightness constraint equation, the optical flow vector of each pixel is obtained. The optical flow vector contains the pixel's motion direction and velocity information. The distribution of all optical flow vectors within the regions of pixel change is statistically analyzed to obtain the optical flow vector distribution characteristics, such as the direction distribution and velocity distribution of the optical flow vectors.

[0061] Step S1222: Determine the offset angle of the motion direction and the change in motion speed of the target object between adjacent video frame units based on the optical flow vector distribution characteristics.

[0062] Based on the characteristics of optical flow vector distribution, the offset angle of the target object's motion direction and the change in motion velocity between adjacent video frame units can be determined. For each target object, a set of representative optical flow vectors are selected within the pixel variation region. The average direction and average velocity of these optical flow vectors are calculated, serving as the target object's motion direction and velocity, respectively. The motion directions of the target objects in adjacent video frames are compared, and the angle between them is calculated to obtain the motion direction offset angle. Simultaneously, the difference in motion velocity of the target objects in adjacent video frames is calculated to obtain the change in motion velocity.

[0063] Step S1223: Construct a motion trajectory continuity analysis model for the target object based on the motion direction offset angle and the change in motion speed.

[0064] A continuity analysis model for the motion trajectory of a target object is constructed using the offset angle of the motion direction and the change in motion velocity. A Hidden Markov Model (HMM) can be used to describe the motion trajectory of the target object. The offset angle of the motion direction and the change in motion velocity are used as observations, and the motion state of the target object is used as the hidden state. By training the HMM model, the distribution of observations of the target object under different motion states is learned. In practical applications, based on the current offset angle of the motion direction and the change in motion velocity, the HMM model is used to predict the next motion state of the target object, thereby analyzing the continuity of the target object's motion trajectory.

[0065] Step S1224: Calculate the motion trajectory direction consistency parameters of the target object in continuous video frame units using the motion trajectory continuity analysis model.

[0066] A motion trajectory continuity analysis model is used to calculate the motion trajectory direction consistency parameter of a target object in consecutive video frame units. The motion direction of the target object is recorded in consecutive video frames. By comparing the motion directions of the target object in adjacent video frames, the angle between them is calculated. A smaller angle indicates that the motion directions of the target object are relatively consistent; a larger angle indicates that the motion direction of the target object has changed significantly. The proportion of motion direction angles in consecutive video frames that are less than a certain threshold is used as the motion trajectory direction consistency parameter. The larger this parameter, the more consistent the motion trajectory direction of the target object.

[0067] Step S1225: Perform periodic motion pattern recognition processing on the motion trajectory of the target object, and extract the motion repetition frequency parameter of the target object within a preset time window as a periodic motion continuity index.

[0068] The process involves recognizing periodic action patterns in the motion trajectory of a target object. Template matching can be used, pre-defining common periodic action templates such as walking, running, and waving. Within a preset time window, the target object's motion trajectory is matched against these templates, identifying the template with the highest match. The number of repetitions of this action template within the time window is counted and used as the action repetition frequency parameter. This parameter reflects the continuity of the target object's periodic actions; a higher parameter value indicates more continuous periodic actions.

[0069] Step S123: Perform spatial region segmentation processing on the video frame unit to generate spatial structure features of the scene where the target object is located. The spatial structure features include the contour boundary integrity parameters of the segmented regions and the relative positional correlation between regions.

[0070] Spatial region segmentation is performed on video frame units to generate spatial structural features of the scene containing the target object. The specific process is as follows:

[0071] Step S1231: Perform edge enhancement processing on the video frame unit to strengthen the contour boundary information of the target object in the video frame unit.

[0072] Edge enhancement processing is applied to video frame units to strengthen the contour boundary information of target objects. The Sobel operator can be used to convolve video frames, calculating the gradients in the horizontal and vertical directions. Pixels with larger gradient values ​​typically correspond to image edges. By thresholding the gradient image, pixels with gradient values ​​greater than the threshold are marked as edge pixels, thus highlighting the contour boundaries of the target object.

[0073] Step S1232: Based on the enhanced contour boundary information, perform region growing segmentation on the video frame unit to obtain multiple spatial sub-region units.

[0074] Based on the enhanced contour boundary information, region growing and segmentation are performed on video frame units. A seed point is selected as the starting point, and pixels adjacent to and similar to the seed point are merged into the same region according to pixel similarity criteria. Similarity criteria can be based on pixel features such as color, grayscale value, and texture. The region is continuously expanded until there are no more adjacent pixels that satisfy the similarity criteria, forming a spatial sub-region unit. This process is repeated until all pixels are assigned to a spatial sub-region unit, resulting in multiple spatial sub-region units.

[0075] Step S1233: Extract the texture density distribution information within each spatial sub-region unit, and calculate the standard deviation of the texture density distribution information in the spatial dimension as a discreteness parameter.

[0076] For each spatial sub-region, extract its internal texture density distribution information. The gray-level co-occurrence matrix (GLCM) can be used to describe texture features. Calculate the GLCM for each pixel within the spatial sub-region and extract texture feature parameters such as contrast, correlation, and energy. Use these texture feature parameters as texture density distribution information. Calculate the standard deviation of the texture density distribution information in the spatial dimension as a parameter of dispersion. A larger standard deviation indicates a more dispersed texture density distribution; a smaller standard deviation indicates a more concentrated texture density distribution.

[0077] Step S1234: Determine the contour boundary integrity parameter of the spatial sub-region unit based on the discreteness parameter, wherein the contour boundary integrity parameter is negatively correlated with the discreteness parameter.

[0078] The contour boundary integrity parameter of a spatial sub-region unit is determined based on the dispersion parameter. Since the texture density distribution within a spatial sub-region unit with a complete contour boundary is relatively concentrated, its dispersion parameter is small; conversely, the texture density distribution within a spatial sub-region unit with an incomplete contour boundary is relatively dispersed, resulting in a larger dispersion parameter. Therefore, the contour boundary integrity parameter and the dispersion parameter are negatively correlated. The contour boundary integrity parameter can be calculated using a linear function. Let the dispersion parameter be σ and the contour boundary integrity parameter be C, then C = km * σ, where k and m are constants, and m > 0.

[0079] Step S1235: Analyze the relative positional relationships between different spatial sub-region units and generate a descriptor for the relative positional correlation between regions that includes a distance distribution matrix and orientation angle parameters.

[0080] Analyze the relative positional relationships between different spatial sub-region units. Calculate the centroid coordinates of any two spatial sub-region units and then calculate the distance between them based on these coordinates. Compile the distances between all spatial sub-region units into a distance distribution matrix. Simultaneously, calculate the angle between the line connecting the centroids of any two spatial sub-region units and the horizontal axis to obtain the direction angle parameter. Combine the distance distribution matrix and the direction angle parameter to form a descriptor for the relative positional correlation between regions. This descriptor reflects the relative positional relationships between different spatial sub-region units.

[0081] Step S124: Input the temporal behavior features and the spatial structure features into the feature alignment module for correlation analysis processing to obtain a target feature set with a unified dimensional representation.

[0082] Temporal behavioral features and spatial structural features are input into the feature alignment module for correlation analysis. Since temporal behavioral features and spatial structural features may have different dimensions and feature representations, feature alignment is necessary. Principal Component Analysis (PCA) can be used to reduce the dimensionality of both temporal behavioral features and spatial structural features, mapping them to the same low-dimensional space. Then, the dimensionality-reduced temporal behavioral features and spatial structural features are concatenated to obtain a target feature set with a unified dimensional representation.

[0083] Step S125: Based on the contribution calculation results of features of different dimensions in the target feature set, dynamically adjust the fusion weight parameters of the temporal behavior features and the spatial structure features in joint abnormal behavior detection.

[0084] Based on the contribution calculation results of features of different dimensions in the target feature set, the fusion weight parameters of temporal behavior features and spatial structure features in joint anomaly detection are dynamically adjusted. A random forest algorithm can be used to calculate the contribution of features of different dimensions. A random forest model is trained, taking the target feature set as input and anomaly behavior labels as output. During training, the importance score of each feature dimension in the decision tree is recorded. Based on these scores, the contribution of features of different dimensions is calculated. Then, the fusion weight parameters of temporal behavior features and spatial structure features are dynamically adjusted according to the contribution. For example, if the contribution of temporal behavior features is high, the fusion weight of temporal behavior features is increased; if the contribution of spatial structure features is high, the fusion weight of spatial structure features is increased.

[0085] Step S130: Call the pre-built abnormal behavior detection model to perform joint abnormal behavior detection on the temporal behavior features and the spatial structure features, and generate abnormal behavior detection results of the target object in the video segment unit.

[0086] After obtaining the temporal and spatial structural features of the target object, a pre-built abnormal behavior detection model is invoked to jointly detect abnormal behavior in these features, generating abnormal behavior detection results for the target object in the video segment unit. The specific process is as follows:

[0087] Step S131: Input the temporal behavior features and the spatial structure features into the feature fusion layer of the abnormal behavior detection model, and perform correlation modeling processing in combination with the fusion weight parameters to generate a fusion feature vector with spatiotemporal consistency constraints.

[0088] Temporal behavioral features and spatial structural features are input into the feature fusion layer of the abnormal behavior detection model. In the feature fusion layer, the temporal behavioral features and spatial structural features are correlated and modeled using the dynamically adjusted fusion weight parameters. A weighted concatenation method can be used to concatenate the temporal behavioral features and spatial structural features according to their weights, generating a fused feature vector with spatiotemporal consistency constraints. During the concatenation process, it is crucial to ensure that the dimensions of the temporal behavioral features and spatial structural features match to avoid inconsistencies in scale. If their dimensions are inconsistent, the features can be adjusted in dimension first, for example, through interpolation or dimensionality reduction, to ensure they can be correctly concatenated. The resulting fused feature vector integrates the temporal behavioral and spatial structural information of the target object, providing a more comprehensive feature representation for subsequent abnormal behavior detection.

[0089] Step S132: The multi-scale perception module of the abnormal behavior detection model performs collaborative extraction processing of local detail features and global context information on the fused feature vector to obtain a multi-scale perception feature set.

[0090] The fused feature vectors are input into the multi-scale perception module of the abnormal behavior detection model. The main function of this multi-scale perception module is to collaboratively extract local detail features and global contextual information. The specific operation is as follows:

[0091] Step S1321: Input the fused feature vector into the feature segmentation layer of the multi-scale perception module, and perform uniform division of the feature channels of the fused feature vector according to the preset channel segmentation ratio to generate a dual-path parallel feature stream containing local feature branches and global feature branches.

[0092] In the feature segmentation layer, the feature channels of the fused feature vector are divided according to a preset channel segmentation ratio. Assuming the number of channels in the fused feature vector is M, and the preset channel segmentation ratio is a and 1-a (where a is a scaling factor), the channels of the fused feature vector are divided into two parts: one part, with a*M channels, serves as the local feature branch, and the other part, with (1-a)*M channels, serves as the global feature branch, thus generating a dual-path parallel feature stream. The purpose of this division is to process local and global information independently to better capture features at different scales.

[0093] Step S1322: In the local feature branch, dilated convolution kernels with different dilation rates are used to perform multi-level spatial receptive field expansion processing on the local feature branch to extract local detail feature maps at different spatial scales. The number of channels in the local detail feature map is aligned with the number of channels in the global feature branch.

[0094] In the local feature branch, dilated convolutional kernels with different dilation rates are used to perform convolution operations. The dilation rate determines the spacing between elements in the convolutional kernel; different dilation rates allow the convolutional kernel to have different spatial receptive fields. By using multiple dilated convolutional kernels with different dilation rates to perform multi-level convolutions, local detail features at different spatial scales can be extracted. For example, dilated convolutional kernels with smaller dilation rates can capture subtle local features, while dilated convolutional kernels with larger dilation rates can capture features from a wider area. During the extraction process, it is important to ensure that the number of channels in the generated local detail feature map is consistent with the number of channels in the global feature branch; this is necessary for effective feature fusion in the subsequent steps.

[0095] Step S1323: In the global feature branch, adaptive average pooling is performed on the global feature branch to generate a context feature vector with global statistical information, and the spatial dimension of the context feature vector is reshaped to obtain a global context feature map that matches the size of the local detail feature map.

[0096] In the global feature branch, adaptive average pooling is first applied. Adaptive average pooling automatically adjusts the pooling window size based on the size of the input feature map, thus performing average pooling on each channel of the input feature map to generate a context feature vector containing global statistical information. This context feature vector contains the overall information of the global feature branch. Then, the spatial dimension of this context feature vector is reshaped by interpolation or other dimensional transformation methods to adjust it into a global context feature map with the same size as the local detail feature map, so that it can be fused with the local detail feature map later.

[0097] Step S1324: Input the local detail feature map and the global context feature map into the feature fusion submodule, dynamically generate the fusion weights of each channel based on the channel attention mechanism, and perform cross-scale information fusion processing on the local detail feature map and the global context feature map through weighted fusion operation to generate a multi-level fusion feature map set.

[0098] The local detail feature map and the global context feature map are input into the feature fusion submodule. In this submodule, a channel attention mechanism is used to dynamically generate the fusion weights for each channel. The channel attention mechanism calculates the importance weight of each channel based on the channel features of the input feature map. Specifically, global average pooling is performed on the input feature map, compressing the features of each channel into a scalar. These scalars are then processed through a fully connected network to obtain the weight coefficients for each channel. Based on these weight coefficients, a weighted fusion operation is performed on the local detail feature map and the global context feature map. Weighted fusion can be achieved by adding the elements of corresponding channels according to their weights, thereby achieving cross-scale information fusion and generating a multi-level fused feature map set.

[0099] Step S1325: Perform channel normalization processing on the multi-level fusion feature mapping set to eliminate the dimensional differences between features of different scales, and then splice the normalized feature mapping along the channel dimension to generate a multi-scale perceptual feature set with a unified dimensional representation.

[0100] Channel normalization is performed on the multi-level fused feature map set. Since features at different scales may have different dimensions and numerical ranges, channel normalization adjusts the feature values ​​of each channel to the same range, eliminating dimensional differences. Normalization can be achieved by calculating the mean and standard deviation of each channel, then subtracting the mean from each element within the channel and dividing by the standard deviation. After normalization, these feature maps are concatenated along the channel dimension. This involves sequentially connecting the channels of each feature map to generate a multi-scale perceptual feature set with a unified dimensional representation. This multi-scale perceptual feature set integrates local detail features and global contextual information at different scales, providing richer feature information for subsequent abnormal behavior detection.

[0101] Step S133: Use the time attention mechanism module of the abnormal behavior detection model to perform time-dependent modeling on the multi-scale perceptual feature set to generate an attention weight distribution map in the time dimension.

[0102] The multi-scale perceptual feature set contains feature information of the target object at different time points, but the importance of features at different time points for abnormal behavior detection may vary. Therefore, the temporal attention mechanism module of the abnormal behavior detection model is used to model the temporal dependencies of the multi-scale perceptual feature set. The temporal attention mechanism module analyzes the feature changes of the multi-scale perceptual feature set in the time dimension and calculates the importance weight of each feature at each time step for the overall anomaly judgment. Specifically, the multi-scale perceptual feature set is processed in the time dimension, and a neural network is used to learn the temporal dependencies between features. This neural network outputs the attention weight for each time step based on the input multi-scale perceptual feature set. These attention weights are then combined into a distribution map, i.e., the attention weight distribution map in the time dimension. This attention weight distribution map reflects the importance of features at different time points in abnormal behavior detection.

[0103] Step S134: Perform dynamic weighted aggregation processing on the multi-scale perceptual feature set according to the time attention weight distribution map to obtain a time-enhanced feature vector that strengthens the temporal correlation.

[0104] Based on the attention weight distribution map along the time dimension, a dynamic weighted aggregation process is performed on the multi-scale perceptual feature set. For each feature at each time step in the multi-scale perceptual feature set, the features are weighted according to their corresponding attention weights. That is, the feature vector of each time step is multiplied by its corresponding attention weight coefficient, and then the weighted feature vectors of all time steps are aggregated. The aggregation method can be addition or concatenation; here, concatenation is used, where the weighted feature vectors are concatenated sequentially according to time order to obtain a temporally enhanced feature vector that strengthens temporal correlation. This temporally enhanced feature vector highlights the feature information at important time points in abnormal behavior detection, strengthens the temporal correlation between features, and helps to detect abnormal behavior more accurately.

[0105] Step S135: Call the classification output layer of the abnormal behavior detection model to perform abnormal probability prediction processing on the temporal enhanced feature vector, and generate an abnormal behavior detection result containing an abnormal probability distribution curve, wherein the abnormal probability distribution curve is used to represent the confidence score corresponding to different abnormal event types.

[0106] The temporal augmentation feature vector is input into the classification output layer of the abnormal behavior detection model. The classification output layer is a neural network layer for classification, capable of predicting the probability of different abnormal event types based on the input temporal augmentation feature vector. The classification output layer maps the temporal augmentation feature vector to the probability space of different abnormal event types through a series of neurons and activation functions. Each abnormal event type corresponds to a probability value, and these probability values ​​form an anomaly probability distribution curve. This anomaly probability distribution curve visually represents the confidence score corresponding to different abnormal event types, i.e., the likelihood of each abnormal event type occurring. By analyzing this anomaly probability distribution curve, it is possible to determine whether the target object in a video segment exhibits abnormal behavior and the possible types of abnormal events.

[0107] Step S140: Determine the types of abnormal events existing in the monitoring video stream to be analyzed and the spatiotemporal distribution characteristics of the abnormal events in the video frame based on the abnormal behavior detection results.

[0108] Based on the abnormal behavior detection results, the types of abnormal events present in the surveillance video stream to be analyzed and the spatiotemporal distribution characteristics of these abnormal events in the video frame are determined. The specific steps are as follows:

[0109] Step S141: Analyze the abnormal probability distribution curve in the abnormal behavior detection result, and detect the abnormal probability peak points that exceed the preset threshold and their corresponding timestamp information.

[0110] The abnormal probability distribution curve in the abnormal behavior detection results is analyzed. A preset threshold is used to determine whether an abnormal event has occurred. The abnormal probability distribution curve is traversed to find the peak points that exceed the preset threshold. These peak points represent moments when the abnormal event is more likely to occur. At the same time, the timestamp information corresponding to each peak point is recorded, and this timestamp information indicates the time point when the abnormal event may have occurred.

[0111] Step S142: Based on the timestamp information corresponding to the peak point of the abnormal probability, and combined with the continuous change trend of the abnormal probability of the preceding and following video segment units, determine the start time point and end time point of the abnormal event, and calculate the difference between the start time point and the end time point as the duration length parameter.

[0112] Based on the timestamp information corresponding to the peak anomaly probability point, and combined with the continuous trend of anomaly probability changes in preceding and following video segment units, the start and end times of the anomaly event are determined. Starting from the peak point, the time point when the anomaly probability began to rise and exceeded a certain small threshold is traced backwards and taken as the start time point of the anomaly event; the time point when the anomaly probability began to fall and fell below a certain small threshold is traced backwards and taken as the end time point of the anomaly event. The difference between the start and end times is calculated to obtain the duration parameter of the anomaly event. This duration parameter helps to understand the duration of the anomaly event and is of significant value in assessing the severity and scope of its impact.

[0113] Step S143: Extract the spatial structure features of the video segment unit corresponding to the peak point of the abnormal probability, and determine the initial occurrence area coordinates and diffusion direction parameters of the abnormal event in the video frame.

[0114] In this step, to determine the initial occurrence area coordinates and propagation direction parameters of the abnormal event in the video frame, it is necessary to carefully extract and analyze the spatial structural features of the video segment unit corresponding to the peak probability point. After obtaining the peak probability point, the corresponding video segment unit is identified. The spatial structural features of this video segment unit are obtained in the previous stage by performing spatial region segmentation on the video frame unit, and include information such as the integrity parameters of the contour boundary of the segmented region and the relative positional correlation between regions.

[0115] Determining the initial location coordinates begins with the results of region segmentation based on spatial structural features. Since region segmentation divides the video frame into multiple spatial sub-regions, it's necessary to filter out sub-regions related to the abnormal event. This can be done based on the characteristics of abnormal behavior, such as abnormal crowding or abnormal object movement, to find the region within the spatial sub-region that best matches these characteristics. For example, if abnormal crowding is detected in a certain spatial sub-region with a significantly higher density of people than in other regions, and the behavior patterns of people in that region differ significantly from normal, then that region is likely the initial location of the abnormal event. By recording the boundary coordinates of this region and performing coordinate transformations, the accurate initial location coordinates of that region in the video frame can be obtained.

[0116] The determination of the diffusion direction parameter depends on the changes in the abnormal regions within adjacent video segment units. Within adjacent video segment units, the position and shape changes of the abnormal regions are continuously monitored. The centroid coordinates of each abnormal region are calculated. These centroid coordinates can be obtained by weighted averaging of the coordinates of all pixels within the region; the weights can be determined based on the pixel's grayscale value or other features. The changes in the centroid coordinates of the abnormal regions within adjacent video segment units are compared to calculate the direction of centroid movement. For example, if the centroid coordinates of the abnormal region in the current video segment unit are (x1, y1), and the centroid coordinates of the abnormal region in the next adjacent video segment unit are (x2, y2), then the direction of centroid movement can be represented by the vector (x2-x1, y2-y1), and the direction of this vector is the diffusion direction of the abnormal event. This direction information is quantified to obtain the diffusion direction parameter, which can be represented, for example, as an angle or a direction vector.

[0117] Step S144: Construct a trajectory path model of the abnormal event in a continuous video segment unit based on the initial occurrence area coordinates, the diffusion direction parameter, and the duration length parameter.

[0118] After obtaining the initial location coordinates, diffusion direction parameters, and duration parameters, a trajectory path model of the abnormal event within consecutive video segment units is constructed. It is assumed that the movement of the abnormal event follows certain physical laws, and that the diffusion direction and velocity are relatively stable over a short period.

[0119] Starting from the initial location coordinates, the direction of movement of the anomalous event is determined based on the diffusion direction parameter. This parameter can be a direction vector or angular information. At each time step (corresponding to a video segment unit), the new location of the anomalous event is predicted based on the diffusion direction and a certain velocity assumption. The velocity can be determined by combining the duration parameter and the changes in the anomalous region. For example, if the area or position of the anomalous region changes significantly within a known duration, the anomalous event is spreading rapidly; conversely, the diffusion rate is slow. The velocity can be estimated by statistically analyzing the rate of change of the anomalous region's area or the distance the centroid moves in adjacent video segment units.

[0120] Let the initial occurrence area coordinates be P0, the diffusion direction vector be D, the velocity be v, and the time step be t (t starts from 0, corresponding to the initial moment). Then, at the t-th time step, the predicted position Pt of the abnormal event can be calculated using the formula Pt = P0 + v * t * D (this calculation is based on the concept of vector operations; in actual implementation, coordinate system transformation and quantization must be considered). By calculating the predicted position at each time step sequentially and connecting these positions, a trajectory path model of the abnormal event in continuous video segment units is formed. During the construction process, constraints on the movement of the abnormal event can also be considered, such as preventing the movement of the abnormal event from exceeding the boundaries of the video frame or being affected by obstacles in the scene. For obstacle handling, obstacle areas can be pre-marked in the video frame. When the predicted position falls within the obstacle area, the movement direction is adjusted, for example, by using reflection or bypassing to update the movement trajectory.

[0121] Step S145: Generate spatiotemporal distribution feature information containing regional coverage and motion trend prediction information through the movement trajectory path model.

[0122] Step S1451: Perform density clustering analysis on the motion trajectory coordinate sequence output by the motion trajectory path model to identify the boundaries of dense regions where trajectory points are distributed, and calculate the coverage radius parameter based on the geometric center coordinates of the dense region boundaries to generate a region coverage descriptor with center coordinates and coverage radius as elements.

[0123] Density clustering analysis is performed on the motion trajectory coordinate sequence output by the motion trajectory path model to identify densely distributed regions of trajectory points. A density-based clustering algorithm, such as DBSCAN, is employed. Its core idea is to divide the data points into clusters based on their density. First, two key parameters are set: the neighborhood radius ε and the minimum number of points MinPts. For each point P in the motion trajectory coordinate sequence, a neighborhood is drawn with P as the center and ε as the radius. If the number of points in this neighborhood is greater than or equal to MinPts, then point P is called a core point. Then, starting from a core point, all points within its neighborhood are added to the same cluster, and this process is continued until no further expansion is possible. In this way, the motion trajectory coordinate sequence is divided into multiple clusters.

[0124] The boundaries of each cluster are identified, i.e., the boundaries of densely distributed trajectory points. These boundaries can be determined by traversing the points within each cluster and finding the outermost points. The geometric center coordinates of the boundary of each dense region are calculated; these coordinates can be obtained by averaging the coordinates of all points on the boundary. Next, the coverage radius parameter is calculated. For each dense region, the distances from points on the boundary to the geometric center coordinates are calculated, and the maximum value among these distances is taken as the coverage radius parameter. The geometric center coordinates and the coverage radius parameter are combined to generate a region coverage descriptor with the center coordinates and coverage radius as elements. This region coverage descriptor can visually represent the coverage area of ​​an anomaly event in the video footage.

[0125] Step S1452: Extract the set of direction vectors of the motion trajectory coordinate sequence under continuous timestamps, calculate the rate of change of the angle between the direction vectors of adjacent timestamps and the gradient of the change of the vector magnitude based on the time interval of each frame, and construct a motion trend direction stability evaluation function based on the frame rate.

[0126] Extract the set of direction vectors at consecutive timestamps from the trajectory coordinate sequence. For trajectory points Pi and Pi+1 corresponding to two adjacent timestamps, the direction vector Vi can be calculated by subtracting Pi from Pi+1 (this is a vector subtraction operation). Combine the direction vectors of all adjacent timestamps into a set of direction vectors.

[0127] Based on the time interval of each frame, calculate the rate of change of the angle between the direction vectors of adjacent timestamps and the gradient of the vector magnitude change. For two adjacent direction vectors Vi and Vi+1, calculate the angle θi between them. The cosine of the angle can be calculated using the dot product formula of vectors, thus obtaining the angle θi. The rate of change of the angle can be obtained by dividing the difference between adjacent angles by the time interval, i.e., Δθi=(θi+1-θi) / Δt, where Δt is the time interval between adjacent timestamps. For the gradient of the vector magnitude change, calculate the magnitudes |Vi| and |Vi+1| of adjacent direction vectors, and the gradient of the vector magnitude change is Δ|V|i=(|Vi+1|-|Vi|) / Δt.

[0128] A motion trend direction stability evaluation function is constructed using frame rate as the time base. Frame rate represents the number of frames per second in the video, reflecting the degree of time dispersion. An evaluation function is designed using the rate of change of the included angle and the gradient of the vector magnitude change as input parameters. For example, a weighted summation method can be used. Let the weight of the rate of change of the included angle be α, and the weight of the gradient of the vector magnitude change be β, and α + β = 1. Then the motion trend direction stability evaluation function S = α * Σ|Δθi| + β * Σ|Δ|V|i|, where Σ represents the summation of parameters over all time steps. This evaluation function can assess the stability of motion trends during abnormal events; the smaller the value of the evaluation function, the more stable the motion trend.

[0129] Step S1453: Perform sliding window weighted averaging on the set of direction vectors based on the direction stability evaluation function to generate a trend prediction direction sequence after direction smoothing, and calculate the cumulative slope change of the trend prediction direction sequence within a preset time window.

[0130] A sliding window weighted average is applied to the set of direction vectors based on a directional stability evaluation function. A sliding window of size W is set and slid across the set of direction vectors. For each direction vector within the sliding window, a weight is assigned based on the output of the directional stability evaluation function. Direction vectors with high stability (i.e., those with small rate of change of angle and small gradient of vector magnitude change) have higher weights, while those with low stability have lower weights. A weighted average is then applied to the direction vectors within the window to obtain a smoothed direction vector. This process of moving the window sequentially yields a series of smoothed direction vectors, forming a sequence of trend prediction directions after directional smoothing.

[0131] The cumulative slope change of the trend prediction direction sequence within a preset time window is calculated. For each direction vector in the trend prediction direction sequence, it can be projected onto a coordinate axis (such as the x-axis or y-axis) to obtain a one-dimensional numerical sequence. The slope of this numerical sequence within the preset time window is calculated; the slope can be obtained by dividing the difference between two adjacent values ​​by the time interval. The changes in all slope values ​​within the preset time window are accumulated to obtain the cumulative slope change. This cumulative slope change reflects the degree of change in the trend of the abnormal event; the larger the cumulative amount, the more drastic the change in the trend.

[0132] Step S1454: Construct a dynamic region expansion model based on the coverage radius parameter and the cumulative slope change of the trend prediction direction sequence, and predict the expansion direction and expansion rate parameters of the region coverage in subsequent video segment units.

[0133] A dynamic region expansion model is constructed based on the coverage radius parameter and the cumulative slope change of the trend prediction direction sequence. The coverage radius parameter reflects the size of the current anomalous event's coverage area, while the cumulative slope change of the trend prediction direction sequence reflects the degree of change in the anomalous event's movement trend.

[0134] When constructing the dynamic region expansion model, it is assumed that the expansion direction of anomalies is related to the average direction of the trend prediction direction sequence, and the expansion rate is related to the coverage radius parameter and the cumulative slope change. First, the average direction of the trend prediction direction sequence is calculated as the basis for the expansion direction of anomalies. Then, the expansion direction is adjusted based on the cumulative slope change. If the cumulative slope change is large, it indicates a drastic change in the movement trend, and the expansion direction may deviate significantly; conversely, the expansion direction is relatively stable.

[0135] To determine the expansion rate parameter, the combined effects of the coverage radius parameter and the cumulative slope change are considered. A function can be designed that takes the coverage radius parameter and the cumulative slope change as input and outputs the expansion rate. For example, let the coverage radius parameter be R, the cumulative slope change be C, and the expansion rate V = f(R, C), where f is a predefined function that can be adjusted according to actual conditions. This dynamic region expansion model can then predict the expansion direction and expansion rate parameters of the region coverage area in subsequent video segment units.

[0136] Step S1455: Input the expansion direction and expansion rate parameters into the trajectory prediction function to generate a probability distribution map containing the trajectory path within the future time window, and extract the coordinates of the largest connected region with a confidence level exceeding a preset threshold from the probability distribution map as the boundary of the predicted motion path.

[0137] The expansion direction and expansion rate parameters are input into the trajectory prediction function. This function is constructed based on the motion patterns and historical trajectories of the anomalous event. It predicts the possible trajectory of the anomalous event within a future time window, taking into account the current location of the event and the expansion direction and expansion rate parameters. During the prediction process, considering the potential uncertainty in the movement of the anomalous event, a probability distribution is used to represent the trajectory. For example, a Gaussian distribution can be used to simulate the probability distribution of the anomalous event's position at each time step; the expansion direction and expansion rate parameters affect the mean and variance of the Gaussian distribution.

[0138] A probability distribution map containing trajectory paths within a future time window is generated using a trajectory prediction function. This probability distribution map represents the probability of an abnormal event occurring at different locations in the future. A preset threshold is set to filter out areas with high confidence. The probability distribution map is traversed to find areas with confidence exceeding the preset threshold. Connectivity analysis is performed on these areas to identify the largest connected region. The coordinates of this largest connected region are extracted as the boundary of the predicted motion path. This predicted motion path boundary can help determine the possible development path of the abnormal event, providing a reference for subsequent early warning and handling.

[0139] Step S1456: Generate spatiotemporal distribution feature information containing a spatial coverage polygon coordinate sequence and a trend direction change vector based on the area coverage descriptor and the predicted motion path boundary. The spatial coverage polygon coordinate sequence is used to mark the real-time coverage area of ​​the abnormal event in the video frame, and the trend direction change vector is used to describe the predicted offset angle and velocity change rate of the abnormal event's motion direction.

[0140] Spatiotemporal distribution feature information is generated based on the region coverage descriptor and the predicted motion path boundary. The region coverage descriptor contains the center coordinates and coverage radius of the current anomalous event's coverage area, while the predicted motion path boundary represents the possible future development path of the anomalous event.

[0141] The area coverage descriptor is converted into a spatial coverage polygon coordinate sequence. A circle is drawn with the center coordinate as the center and the coverage radius as the radius. The circle is then discretized to obtain a series of boundary point coordinates. These boundary point coordinates are connected in a predetermined order to form the spatial coverage polygon coordinate sequence. This spatial coverage polygon coordinate sequence can accurately mark the real-time coverage area of ​​abnormal events in the video frame.

[0142] Based on the predicted motion path boundary and the current motion direction, a trend direction change vector is calculated. This vector contains the predicted offset angle and velocity change rate of the anomalous event's motion direction. The predicted offset angle can be obtained by comparing the current motion direction with the average direction of the predicted motion path, and the velocity change rate can be obtained by comparing the current expansion rate with the predicted expansion rate. The predicted offset angle and velocity change rate are combined to form the trend direction change vector. This vector describes the changes in the anomalous event's motion direction, providing crucial spatiotemporal information for the monitoring and handling of anomalous events.

[0143] Step S150: Generate a security warning instruction containing the event location coordinates based on the abnormal event type and the spatiotemporal distribution feature information, and send the security warning instruction to the target security control terminal to trigger a linkage response operation.

[0144] Based on the abnormal event type and spatiotemporal distribution characteristics, a security early warning command containing the event location coordinates is generated, and the command is sent to the target security control terminal to trigger a linkage response operation. The specific steps are as follows:

[0145] Step S151: Parse the preset early warning rule base corresponding to the abnormal event type, and extract the early warning level identifier and response strategy code associated with the abnormal event type.

[0146] The system parses the preset early warning rule base corresponding to different abnormal event types. This base stores early warning levels and response strategies for various abnormal event types. Based on the abnormal event type, it extracts the associated early warning level identifier and response strategy code from the rule base. The early warning level identifier indicates the severity of the abnormal event, and the response strategy code instructs the target security control terminal on the specific response measures to be taken.

[0147] Step S152: Based on the spatial coverage polygon coordinate sequence in the spatiotemporal distribution feature information, calculate the centroid coordinates and bounding rectangle parameters of the coverage area of ​​the abnormal event within the current video frame unit.

[0148] Based on the spatial coverage polygon coordinate sequence from the spatiotemporal distribution feature information, the centroid coordinates of the coverage area of ​​the abnormal event within the current video frame unit are calculated. The centroid coordinates can be obtained by weighted averaging of the coordinates of all vertices of the spatial coverage polygon; the weights can be determined based on the importance or area contribution of each vertex. Simultaneously, the parameters of the bounding rectangle of the coverage area are calculated, including the coordinates of the top-left corner, width, and height of the rectangle. These parameters help to accurately locate the position of the abnormal event in the video frame.

[0149] Step S153: Extract the directional offset angle and velocity change rate from the trend direction change vector in the spatiotemporal distribution feature information, and construct a dynamic diffusion direction prediction function based on the linear regression model of the historical trajectory by combining the centroid coordinates of the covered area.

[0150] The directional offset angle and velocity change rate are extracted from the trend direction change vector in the spatiotemporal distribution feature information. Combined with the centroid coordinates of the covered area, a dynamic diffusion direction prediction function is constructed based on a linear regression model of historical trajectories. The linear regression model fits a linear equation based on the historical movement trajectory of the anomalous event to predict its future diffusion direction. The directional offset angle and velocity change rate are used as input parameters to update the model's prediction results. This dynamic diffusion direction prediction function allows for a more accurate prediction of the development direction of anomalous events.

[0151] Step S154: Input the bounding rectangle parameters of the boundary into the pre-trained coordinate mapping model to generate a standardized set of positioning coordinates that are consistent with the geographic coordinate system of the monitoring scene.

[0152] The bounding rectangle parameters are input into a pre-trained coordinate mapping model. This model is a trained system that converts coordinates in video footage into standardized location coordinates consistent with the geographic coordinate system of the monitored scene. The pre-trained model is trained on a large amount of labeled data containing both video frame coordinates and their corresponding geographic coordinates. By inputting the bounding rectangle parameters into this model, it converts the video frame coordinates into geographic coordinates based on its internal mapping relationships, generating a standardized set of location coordinates. These standardized coordinates accurately represent the location of abnormal events within the actual monitored scene, facilitating subsequent security measures.

[0153] Step S155: Perform time-domain interpolation on the standardized positioning coordinate set according to the dynamic diffusion direction prediction function to generate a predicted coordinate sequence of the abnormal event diffusion path within a future preset time period.

[0154] A dynamic diffusion direction prediction function is used to perform temporal interpolation on a standardized set of location coordinates. Temporal interpolation is used to predict the diffusion path of anomalies within a predetermined time period. The dynamic diffusion direction prediction function predicts the direction and distance of movement of the anomaly at each time step based on its current location and movement trend. For each coordinate point in the standardized set of location coordinates, the dynamic diffusion direction prediction function calculates its positional change at each time step within the predetermined time period. By interpolating along the time dimension, a predicted coordinate sequence of the anomaly's diffusion path within the predetermined time period is generated. This predicted coordinate sequence can help predict the trajectory of anomalies, providing a basis for the early deployment of security measures.

[0155] Step S156: Integrate the warning level identifier, response strategy code, and predicted coordinate sequence to generate a security warning instruction containing an event location coordinate chain with timestamp alignment. The dimension of each coordinate node in the event location coordinate chain is consistent with the dimension of the standardized location coordinate set.

[0156] Integrate the warning level identifier, response strategy code, and predicted coordinate sequence to generate a security warning instruction. The warning level identifier indicates the severity of the abnormal event, the response strategy code indicates the specific measures that the target security control terminal should take, and the predicted coordinate sequence provides information on the location and spread path of the abnormal event. To make this information correspond in time, timestamp alignment is required. For each coordinate point in the predicted coordinate sequence, add the corresponding timestamp information to form an event location coordinate chain. Ensure that the dimension of each coordinate node in the event location coordinate chain is consistent with the dimension of the standardized location coordinate set, so as to ensure the consistency and accuracy of the coordinate information. Integrate the warning level identifier, response strategy code, and the event location coordinate chain with timestamp alignment to generate a security warning instruction containing detailed information.

[0157] Step S157: Send the security warning instruction to the target security control terminal to trigger a linkage response operation.

[0158] After generating the security warning instruction, send it to the target security control terminal. The target security control terminal can be various security devices, such as alarms, surveillance cameras, access control systems, etc. Through the network communication protocol, transmit the security warning instruction to the target security control terminal. After receiving the security warning instruction, the target security control terminal will trigger the corresponding linkage response operation according to the warning level identifier and response strategy code in the security warning instruction. For example, if the warning level is high and the response strategy code indicates starting the alarm and strengthening the access control, then the target security control terminal will immediately start the alarm to sound an alarm and restrict the entry and exit of personnel and vehicles to respond to the abnormal event.

[0159] For example, Step S158: Receive the instruction execution status feedback data packet returned by the target security control terminal. The instruction execution status feedback data packet contains the device response timestamp and the execution result status code.

[0160] After executing a linkage response operation, the target security control terminal returns a command execution status feedback data packet. This data packet contains a device response timestamp and an execution result status code. The device response timestamp records the time when the target security control terminal received the command and began executing the operation, while the execution result status code indicates the operation's status, such as success or failure. The system receives the command execution status feedback data packet returned by the target security control terminal via network communication to evaluate the effectiveness of the security warning commands.

[0161] Step S158: Parse the device response delay time and operation success identifier in the instruction execution status feedback data packet.

[0162] The instruction execution status feedback data packet is parsed to extract the device response delay time and the operation success identifier. The device response delay time refers to the time interval from sending the security warning instruction to the target security control terminal starting to execute the operation, calculated by the difference between the device response timestamp and the instruction sending timestamp. The operation success identifier is a representation of the execution result status code, used to determine whether the operation on the target security control terminal was successful. If the execution result status code indicates success, the operation success identifier is a success flag; if the execution result status code indicates failure, the operation success identifier is a failure flag.

[0163] Step S190: Calculate the execution performance evaluation index of the security early warning command based on the device response delay time and operation success identifier.

[0164] The execution performance evaluation index for security warning commands is calculated based on device response latency and operation success identifier. This index comprehensively reflects the effectiveness of the security warning commands. A comprehensive evaluation function can be used to calculate this index; for example, device response latency and operation success identifier can be used as input parameters, weighted according to different weights. A higher weight is given to a success identifier, and a lower weight is given to a failure identifier. Furthermore, a shorter device response latency contributes more to the execution performance evaluation index. In this way, a comprehensive execution performance evaluation index is calculated to assess the effectiveness of security warning commands.

[0165] Step S159: When the execution performance evaluation index is lower than the preset threshold, generate a strategy optimization request message containing the abnormal event type and the execution performance evaluation index.

[0166] A preset threshold is set to determine whether the execution effect of security warning commands meets the target. When the execution performance evaluation index is lower than the preset threshold, it indicates that the current security response strategy may have problems and needs optimization. A strategy optimization request message is generated, containing the abnormal event type and the execution performance evaluation index. The abnormal event type helps to identify the specific scenario of the problem, while the execution performance evaluation index provides a quantitative evaluation result. This information is integrated into the strategy optimization request message to request strategy optimization from the cloud management platform.

[0167] Step S1510: Send the policy optimization request message to the cloud management platform to trigger the online update process of the security response policy, and receive the updated set of device linkage rules issued by the cloud management platform.

[0168] The strategy optimization request message is sent to the cloud management platform. The cloud management platform is a centralized platform for managing and optimizing security response strategies. Upon receiving the strategy optimization request message, the cloud management platform analyzes and optimizes the current security response strategy based on the type of abnormal event and execution performance evaluation indicators. Using machine learning algorithms or expert experience, it adjusts the device linkage rules, generating an updated set of device linkage rules. The cloud management platform then distributes the updated set of device linkage rules to the relevant target security control terminals. Upon receiving the updated set of device linkage rules, the target security control terminals update their own rule configurations to improve the execution effectiveness of security warning commands.

[0169] Throughout the entire security surveillance video analysis process, data acquisition, processing, and transmission may involve privacy-sensitive data, such as facial features and behavioral patterns. To protect the security of this sensitive data, various privacy protection and leak prevention technologies are employed. During the data acquisition phase, strict access control and security configuration are implemented for the acquisition devices, ensuring that only authorized personnel can access and operate them. The raw video data is encrypted using a symmetric encryption algorithm, and the encryption key is securely stored and managed. During data transmission, secure network communication protocols, such as SSL / TLS, are used to encrypt the data and prevent it from being stolen or tampered with during transmission. In the data processing phase, privacy-sensitive data is anonymized, such as blurring facial features and anonymizing behavioral patterns. Simultaneously, strict access control is implemented for data access during processing, allowing only authorized personnel and programs to access and process sensitive data. These technologies ensure the security of privacy-sensitive data and avoid the risk of privacy leaks.

[0170] The construction and training of the abnormal behavior detection model is a crucial step in the entire security surveillance video analysis method. The abnormal behavior detection model mainly consists of a feature fusion layer, a multi-scale perception module, a temporal attention mechanism module, and a classification output layer. The feature fusion layer fuses temporal behavioral features and spatial structural features to generate a fused feature vector with spatiotemporal consistency constraints. The multi-scale perception module collaboratively extracts local detail features and global contextual information to obtain a multi-scale perception feature set. The temporal attention mechanism module performs temporal dependency modeling on the multi-scale perception feature set, generating an attention weight distribution map along the temporal dimension. The classification output layer performs anomaly probability prediction on the temporally enhanced feature vector to generate the abnormal behavior detection result.

[0171] During the model training phase, a large amount of labeled data is first collected, including video data of normal and abnormal behaviors. This data is then labeled with information such as the type of abnormal event, its start and end times, and the region where it occurred. The labeled data is then divided into training, validation, and test sets. The training set is used to train the model, the validation set is used to tune the model's hyperparameters, and the test set is used to evaluate the model's performance.

[0172] The abnormal behavior detection model is trained using a training set. During training, stochastic gradient descent is employed to optimize the model's parameters. Video data from the training set is input into the model, and after feature extraction and processing, the abnormal behavior detection results are obtained. The model's output is compared with the labeled information, and the loss function is calculated. The cross-entropy loss function can be used to measure the difference between the abnormal probability distribution output by the model and the true abnormal labels. Through backpropagation, the model's parameters are updated based on the gradient of the loss function, gradually bringing the model's output closer to the true labels.

[0173] During training, the model's hyperparameters, such as learning rate, batch size, and number of training epochs, are continuously adjusted to improve model performance. The model is then validated using a validation set, and its performance metrics, such as accuracy, recall, and F1 score, are observed. Based on the validation results, the hyperparameters are adjusted until the model's performance on the validation set reaches its optimal level.

[0174] Finally, the trained model is evaluated using a test set. Video data from the test set is input into the model to obtain abnormal behavior detection results, and the model's performance metrics on the test set are calculated. If the model's performance metrics meet the requirements, the model is considered successfully trained and can be applied to actual security surveillance video analysis.

[0175] Figure 2The illustration shows exemplary hardware and software components of an AI-based security surveillance video analysis system 100 that can implement the ideas of this application, according to some embodiments of this application. For example, a processor 120 can be used in the AI-based security surveillance video analysis system 100 and to perform the functions described in this application.

[0176] The AI-based security surveillance video analysis system 100 can be a general-purpose server or a special-purpose server; both can be used to implement the AI-based security surveillance video analysis method of this application. Although only one server is shown in this application, for convenience, the functions described in this application can be implemented in a distributed manner on multiple similar platforms to balance the load.

[0177] For example, the AI-based security surveillance video analysis system 100 may include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and various forms of storage media 140, such as a disk, ROM, or RAM, or any combination thereof. Exemplarily, the AI-based security surveillance video analysis system 100 may also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The methods of this application can be implemented according to these program instructions. The AI-based security surveillance video analysis system 100 also includes an I / O interface 150 between the computer and other input / output devices.

[0178] For ease of explanation, only one processor is described in the AI-based security surveillance video analysis system 100. However, it should be noted that the AI-based security surveillance video analysis system 100 of this application may also include multiple processors. Therefore, the steps performed by one processor as described in this application may also be performed jointly or individually by multiple processors. For example, if the processor of the AI-based security surveillance video analysis system 100 performs steps A and B, it should be understood that steps A and B may also be performed jointly by two different processors or individually by one processor. For example, the first processor performs step A, the second processor performs step B, or the first processor and the second processor jointly perform steps A and B.

[0179] Furthermore, this embodiment of the invention also provides a readable storage medium, wherein computer-executable instructions are preset in the readable storage medium, and when the processor executes the computer-executable instructions, the above-mentioned AI-based security monitoring video analysis method is implemented.

[0180] It should be noted that, in order to simplify the description of the present invention and thus help to understand one or more embodiments of the invention, multiple features may sometimes be grouped into one embodiment, drawing or description thereof in the foregoing description of the embodiments of the present invention.

Claims

1. An AI-based security surveillance video analysis method, characterized in that, The method includes: Obtain the initial video data set corresponding to the monitoring video stream to be analyzed. The initial video data set contains multiple video segment units with timestamps collected continuously. The initial video dataset is subjected to video feature extraction processing to obtain the temporal behavior features of the target object in the video segment unit and the spatial structure features of the scene where the target object is located. A pre-built abnormal behavior detection model is invoked to perform joint abnormal behavior detection on the temporal behavior features and the spatial structure features, thereby generating abnormal behavior detection results for the target object in the video segment unit; Based on the abnormal behavior detection results, determine the types of abnormal events existing in the monitoring video stream to be analyzed and the spatiotemporal distribution characteristics of the abnormal events in the video frame; Based on the abnormal event type and the spatiotemporal distribution feature information, a security early warning instruction containing the event location coordinates is generated, and the security early warning instruction is sent to the target security control terminal to trigger a linkage response operation; The step of calling a pre-built abnormal behavior detection model to jointly detect abnormal behavior based on the temporal behavior features and the spatial structure features, and generating abnormal behavior detection results for the target object in the video segment unit, includes: The temporal behavior features and spatial structure features are input into the feature fusion layer of the abnormal behavior detection model, and correlation modeling is performed in combination with fusion weight parameters to generate a fusion feature vector with spatiotemporal consistency constraints. The multi-scale perception module of the abnormal behavior detection model performs collaborative extraction of local detail features and global context information on the fused feature vector to obtain a multi-scale perception feature set. The temporal attention mechanism module of the abnormal behavior detection model is used to perform temporal dependency modeling on the multi-scale perceptual feature set to generate a temporal attention weight distribution map in the time dimension. Based on the time attention weight distribution map, the multi-scale perceptual feature set is dynamically weighted and aggregated to obtain a time-enhanced feature vector that strengthens the temporal correlation. The classification output layer of the abnormal behavior detection model is invoked to perform abnormal probability prediction processing on the temporal enhanced feature vector, generating an abnormal behavior detection result containing an abnormal probability distribution curve, wherein the abnormal probability distribution curve is used to represent the confidence score corresponding to different abnormal event types.

2. The AI-based security surveillance video analysis method according to claim 1, characterized in that, The step of performing video feature extraction processing on the initial video data set to obtain the temporal behavior features of the target object in the video segment unit and the spatial structure features of the scene where the target object is located includes: The initial video data set is divided into frames to obtain multiple video frame units with a continuous time sequence relationship; The video frame unit is subjected to dynamic information capture processing to generate the motion trajectory change characteristics of the target object between adjacent video frame units as temporal behavior characteristics. The temporal behavior characteristics include motion trajectory direction consistency parameters and periodic action continuity indicators. The video frame unit is subjected to spatial region segmentation processing to generate spatial structural features of the scene where the target object is located. The spatial structural features include the contour boundary integrity parameters of the segmented region and the relative positional correlation between regions. The temporal behavior features and the spatial structure features are input into the feature alignment module for correlation analysis to obtain a set of target features with a unified dimensional representation. Based on the contribution calculation results of features of different dimensions in the target feature set, the fusion weight parameters of the temporal behavior features and the spatial structure features in the joint abnormal behavior detection are dynamically adjusted.

3. The AI-based security surveillance video analysis method according to claim 2, characterized in that, The step of performing dynamic information capture processing on the video frame units to generate motion trajectory change features of the target object between adjacent video frame units as temporal behavior features includes: Extract the pixel change region between adjacent video frame units and calculate the optical flow vector distribution characteristics of the pixel change region; Based on the optical flow vector distribution characteristics, determine the target object's motion direction offset angle and motion speed change between adjacent video frame units; A continuity analysis model of the target object's motion trajectory is constructed based on the motion direction offset angle and the change in motion speed. The motion trajectory direction consistency parameters of the target object in continuous video frame units are calculated using the motion trajectory continuity analysis model. The motion trajectory of the target object is processed by periodic motion pattern recognition, and the motion repetition frequency parameter of the target object within a preset time window is extracted as a periodic motion continuity index.

4. The AI-based security surveillance video analysis method according to claim 2, characterized in that, The step of performing spatial region segmentation on the video frame unit to generate spatial structural features of the scene where the target object is located includes: Edge enhancement processing is performed on the video frame unit to strengthen the contour boundary information of the target object in the video frame unit; Based on the enhanced contour boundary information, the video frame unit is subjected to region growing and segmentation processing to obtain multiple spatial sub-region units. Extract the texture density distribution information within each spatial sub-region unit, and calculate the standard deviation of the texture density distribution information in the spatial dimension as a discreteness parameter; The contour boundary integrity parameter of the spatial sub-region unit is determined based on the discreteness parameter, and the contour boundary integrity parameter is negatively correlated with the discreteness parameter. Analyze the relative positional relationships between different spatial sub-regions and generate a descriptor for the relative positional correlation between regions, which includes a distance distribution matrix and orientation angle parameters.

5. The AI-based security surveillance video analysis method according to claim 1, characterized in that, The multi-scale perception module of the abnormal behavior detection model performs collaborative extraction processing of local detail features and global context information on the fused feature vector to obtain a multi-scale perception feature set, including: The fused feature vector is input into the feature segmentation layer of the multi-scale perception module. The feature channels of the fused feature vector are uniformly divided according to the preset channel segmentation ratio to generate a dual-path parallel feature stream containing local feature branches and global feature branches. In the local feature branch, dilated convolution kernels with different dilation rates are used to perform multi-level spatial receptive field expansion processing on the local feature branch to extract local detail feature maps at different spatial scales. The number of channels in the local detail feature map is aligned with the number of channels in the global feature branch. In the global feature branch, adaptive average pooling is performed on the global feature branch to generate a context feature vector with global statistical information, and the spatial dimension of the context feature vector is reshaped to obtain a global context feature map that matches the size of the local detail feature map. The local detail feature map and the global context feature map are input into the feature fusion submodule. The fusion weights of each channel are dynamically generated based on the channel attention mechanism. The local detail feature map and the global context feature map are subjected to cross-scale information fusion processing through weighted fusion operation to generate a multi-level fusion feature map set. The multi-level fusion feature mapping set is subjected to channel normalization to eliminate the dimensional differences between features at different scales. The normalized feature mappings are then spliced ​​along the channel dimension to generate a multi-scale perceptual feature set with a unified dimensional representation.

6. The AI-based security surveillance video analysis method according to claim 1, characterized in that, The step of determining the types of abnormal events present in the monitored video stream to be analyzed and the spatiotemporal distribution characteristics of the abnormal events in the video frame based on the abnormal behavior detection results includes: Analyze the abnormal probability distribution curve in the abnormal behavior detection results, and detect the abnormal probability peak points that exceed the preset threshold and their corresponding timestamp information; Based on the timestamp information corresponding to the peak point of the abnormal probability, and combined with the continuous change trend of the abnormal probability of the preceding and following video segment units, the start time point and end time point of the abnormal event are determined, and the difference between the start time point and the end time point is calculated as the duration parameter. Extract the spatial structure features of the video segment unit corresponding to the peak point of the abnormal probability, and determine the initial occurrence area coordinates and diffusion direction parameters of the abnormal event in the video frame; Based on the initial occurrence area coordinates, the diffusion direction parameter, and the duration parameter, a movement trajectory path model of the abnormal event in a continuous video segment unit is constructed. The movement trajectory path model generates spatiotemporal distribution feature information containing regional coverage and movement trend prediction information.

7. The AI-based security surveillance video analysis method according to claim 6, characterized in that, The generation of spatiotemporal distribution feature information containing regional coverage and motion trend prediction information through the movement trajectory path model includes: Density clustering analysis is performed on the mobile trajectory coordinate sequence output by the mobile trajectory path model to identify the boundaries of dense regions where trajectory points are distributed. The coverage radius parameter is calculated based on the geometric center coordinates of the dense region boundaries to generate a region coverage descriptor with center coordinates and coverage radius as elements. Extract the set of direction vectors of the motion trajectory coordinate sequence under consecutive timestamps, calculate the rate of change of the angle between the direction vectors of adjacent timestamps and the gradient of the change of the vector magnitude based on the time interval of each frame, and construct a motion trend direction stability evaluation function based on the frame rate as the time reference. Based on the directional stability evaluation function, the set of directional vectors is subjected to sliding window weighted averaging to generate a directional smoothed trend prediction directional sequence, and the cumulative slope change of the trend prediction directional sequence within a preset time window is calculated. A dynamic region expansion model is constructed based on the coverage radius parameter and the cumulative slope change of the trend prediction direction sequence to predict the expansion direction and expansion rate parameters of the region coverage in subsequent video segment units; The expansion direction and expansion rate parameters are input into the trajectory prediction function to generate a probability distribution map containing the trajectory path within the future time window, and the coordinates of the largest connected region with a confidence level exceeding a preset threshold in the probability distribution map are extracted as the boundary of the predicted motion path. Based on the area coverage descriptor and the predicted motion path boundary, spatiotemporal distribution feature information containing a spatial coverage polygon coordinate sequence and a trend direction change vector is generated. The spatial coverage polygon coordinate sequence is used to mark the real-time coverage area of ​​the abnormal event in the video frame, and the trend direction change vector is used to describe the predicted offset angle and velocity change rate of the abnormal event's motion direction.

8. The AI-based security surveillance video analysis method according to claim 1, characterized in that, The generation of a security warning instruction containing event location coordinates based on the abnormal event type and the spatiotemporal distribution feature information includes: The preset early warning rule base corresponding to the abnormal event type is parsed, and the early warning level identifier and response strategy code associated with the abnormal event type are extracted; Based on the spatial coverage polygon coordinate sequence in the spatiotemporal distribution feature information, calculate the centroid coordinates and bounding rectangle parameters of the coverage area of ​​the abnormal event within the current video frame unit; The directional offset angle and velocity change rate in the trend direction change vector of the spatiotemporal distribution feature information are extracted, and combined with the centroid coordinates of the covered area, a dynamic diffusion direction prediction function is constructed based on the linear regression model of historical trajectories. The parameters of the bounding rectangle are input into a pre-trained coordinate mapping model to generate a standardized set of positioning coordinates that are consistent with the geographic coordinate system of the monitoring scene. The standardized location coordinate set is interpolated in the time domain according to the dynamic diffusion direction prediction function to generate a predicted coordinate sequence of the abnormal event diffusion path within a future preset time period. By integrating the warning level identifier, response strategy code, and predicted coordinate sequence, a security warning instruction containing an event location coordinate chain with timestamp alignment is generated, wherein the dimension of each coordinate node in the event location coordinate chain is consistent with the dimension of the standardized location coordinate set.

9. An AI-based security surveillance video analysis system, characterized in that, The device includes a processor and a memory, the memory being connected to the processor. The memory is used to store programs, instructions, or code, and the processor is used to execute the programs, instructions, or code in the memory to implement the AI-based security monitoring video analysis method according to any one of claims 1-8.

Images