An AI deep learning-based monitoring video target abnormal behavior detection method

Abstract

The application discloses a kind of based on AI deep learning's monitoring video target abnormal behavior detection method, it is related to intelligent video monitoring and computer vision field.The based on AI deep learning's monitoring video target abnormal behavior detection method, by continuous acquisition monitoring video stream data, synchronous key point space decomposition and scene semantic decomposition, target key point heat map and global semantic feature map are extracted in parallel;Respectively via coordinate coding and Gaussian weighting sampling, construct target node set and scene condition node set, and by heterogeneity association and cross-frame convergence, construct monitoring heteromeric space-time graph, the abnormal behavior recognition network is input to monitoring heteromeric space-time graph by the present application, by graph merging and space-time double bias superposition attention coding, output abnormal behavior detection result, to be able to excavate human posture time series change and environmental space linkage's implicit association features, strengthen the feature distinction between high similarity behaviors, to improve the discrimination precision degree of abnormal identification.

Description

Technical Field

[0001] This invention relates to the fields of intelligent video surveillance and computer vision, specifically to a method for detecting abnormal behavior of targets in surveillance videos based on AI deep learning. Background Technology

[0002] In indoor monitoring scenarios such as home-based elderly care, medical institution wards, and intelligent security systems, automatically identifying abnormal behaviors of target individuals through surveillance videos is an important means of ensuring the safety of groups such as elderly people living alone, patients, and young children. Among these, falls are one of the abnormal behaviors that monitoring systems need to prioritize detecting due to their suddenness and high degree of harm. During a fall, the target individual usually exhibits rapid changes in body posture, a significant drop in center of gravity, and contact with the ground within a short period of time. These visual characteristics constitute the main basis for existing fall detection methods.

[0003] The limitations of existing technologies include at least the following problems: In the process of detecting abnormal behavior in indoor monitoring videos, existing technologies generally rely solely on single-frame human skeleton angle thresholds or fixed temporal features to complete behavior discrimination. They can only capture extreme behaviors with large differences in appearance and contours, and it is difficult to accurately distinguish between self-adjusting movements with highly similar shapes and passive sudden falls in two-dimensional monitoring images. Relying solely on manually preset physical rules and fixed attention receptive fields for feature modeling, it is easy to misjudge self-lying on one's side as a high-risk fall alarm and miss genuine minor falls. At the same time, it lacks the ability to collaboratively model heterogeneous nodes in a spatiotemporal manner, and has poor adaptability to complex monitoring conditions such as indoor lighting fluctuations, partial human body occlusion, and slight background disturbances. The overall false alarm and missed alarm probability remains high, making it difficult to meet the safety and prevention needs in scenarios such as elderly care and hospital ward care, and thus making it difficult to reliably guarantee the close-fitting safety monitoring needs of people indoors. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method for detecting abnormal behavior of targets in surveillance videos based on AI deep learning. This method solves the problems of inaccurate differentiation between autonomous and falling behaviors with similar shapes, high false alarm rates, and low monitoring reliability in existing technologies.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a method for detecting abnormal behavior of targets in surveillance videos based on AI deep learning, comprising the following steps: continuously acquiring surveillance video stream data, including several frames of surveillance images; simultaneously performing key point spatial decomposition and scene semantic decomposition; and extracting target key point heatmaps and global semantic feature maps in parallel; constructing target node sets and scene condition node sets for the target key point heatmaps and global semantic feature maps respectively through coordinate encoding and Gaussian weighted sampling; constructing a surveillance heterogeneous spatiotemporal graph for the target node set and scene condition node set through heterogeneous association and cross-frame convergence; inputting the surveillance heterogeneous spatiotemporal graph into an abnormal behavior recognition network pre-trained by a graph enhancement layer, multiple Transformer layers, and a recognition classifier; and outputting the abnormal behavior detection result through attention encoding with graph merging and spatiotemporal dual bias superposition.

[0006] Furthermore, the specific steps for extracting the heatmap of key target points are as follows:

[0007] A hierarchical Gaussian blur and downsampling process is performed using a preset Gaussian kernel sequence to construct an image pyramid, and edge detection is performed to obtain a scale-level edge map. A blank matrix is ​​created, and all pixels in the scale-level edge map are traversed to calculate discrete curvature values. These discrete curvature values ​​are then filled into the blank matrix to obtain a scale-level curvature response map. The maximum value of each pixel in the scale-level curvature response map is taken for cross-scale fusion to obtain a multi-scale curvature fusion map. Non-maximum suppression is applied to the multi-scale curvature fusion map, and the maps are sorted in descending order of curvature response values ​​to obtain a set of target key points. Based on the target key point set, a target key point heatmap is generated through spatial Gaussian diffusion.

[0008] Further, the specific steps for extracting the global semantic feature map are as follows: perform linear iterative clustering segmentation with a preset number of superpixels to obtain a superpixel segmentation map; for each superpixel block in the superpixel segmentation map, calculate the first-order color moment, the second-order color moment, and the texture contrast, and construct the original feature vector of the corresponding superpixel block; construct a feature matrix with the original feature vectors of all superpixel blocks, and perform principal component analysis for dimensionality reduction to generate a global semantic feature map.

[0009] Further, the specific steps for constructing the target node set and the scene condition node set are as follows: After thresholding and non-maximum suppression of the target key point heatmap, a target key point coordinate set is obtained; for each target key point coordinate in the target key point coordinate set, multi-frequency sine and cosine encoding is performed to obtain a coordinate encoding vector, and target part labels are assigned according to the preset spatial partitioning rules to construct a target node set, which includes several target nodes and their corresponding coordinate encoding vectors; the global semantic feature map is weighted and sampled using multiple preset spatial Gaussian kernel functions to obtain scene condition node feature vectors, and a scene condition node set is constructed, which includes several scene condition nodes.

[0010] Furthermore, the specific steps for constructing the heterogeneous spatiotemporal graph of the monitoring system are as follows: A node set is formed by combining the target node set and the scene condition node set; based on the target part markings of each target node, a preset target part adjacency relationship table is queried to identify adjacent node pairs, and spatial edges are established between adjacent node pairs; temporal edges are established between target nodes with the same target part markings in adjacent frames; fully connected control edges are established one by one between scene condition nodes and target nodes; the node set, spatial edges, temporal edges, and fully connected control edges are combined to form the heterogeneous spatiotemporal graph of the monitoring system.

[0011] Furthermore, the specific steps for outputting the abnormal behavior detection results are as follows: In the graph enhancement layer, the set of freely interacting hidden nodes is initialized and added to the monitored heterogeneous spatiotemporal graph to obtain the enhanced heterogeneous spatiotemporal graph; In the first Transformer layer, for each pair of nodes with edge connections, the current node and its neighboring nodes are determined, the original attention score is calculated, and after spatiotemporal double bias superposition modulation, the bias modulation coding feature is obtained; The bias modulation coding feature is input to subsequent Transformer layers for attention coding operations; In the recognition classifier, the final bias modulation coding features of all nodes output by the last Transformer layer are received, and after spatiotemporal interaction filtering, the abnormal behavior detection results are output.

[0012] Furthermore, the specific steps of spatiotemporal dual-bias superposition modulation are as follows: preset the maximum time interval threshold, initialize the time bias parameters, and construct a learnable time bias table; based on the current node and its neighboring nodes, index the corresponding time bias values ​​from the learnable time bias table, and superimpose them onto the original attention score to obtain the attention score modulated by time bias; extract the final position encoding vectors corresponding to the current node and its neighboring nodes from the pre-constructed learnable position encoding matrix, calculate the position bias values, and superimpose them onto the attention score modulated by time bias to obtain the attention score modulated by dual bias.

[0013] Furthermore, the pre-construction steps of the learnable location coding matrix are as follows: randomly initialize location coding vectors for each node set; divide the enhanced heterogeneous spatiotemporal graph into multiple spatial location regions according to the preset region division rules, and randomly initialize spatial location coding vectors; determine the spatial location coding vector of each node in the enhanced heterogeneous spatiotemporal graph, and perform learnable heterogeneous fusion to obtain the final location coding vector, so as to form a learnable location coding matrix.

[0014] Furthermore, learnable attention pooling is applied to the final bias modulation and coding features. A cross-attention operation is then performed between the learnable classification-filtered query vector and the final bias modulation and coding features to generate a spatiotemporal bias feature vector. The spatiotemporal bias feature vector is then classified and mapped to output a multidimensional behavior determination probability vector, where each dimension of the multidimensional behavior determination probability vector corresponds to a probability value of a preset behavior category. The preset behavior category corresponding to the dimension with the highest probability value in the multidimensional behavior determination probability vector is taken as the behavior determination result.

[0015] Furthermore, the pre-training steps of the abnormal behavior recognition network are as follows: Collect a dataset of labeled surveillance video samples; input the surveillance video sample dataset into the abnormal behavior recognition network to be trained, and obtain the predicted behavior judgment result through forward propagation; calculate the behavior classification loss using the cross-entropy loss function, and based on this, combine the spatiotemporal dual bias regularization term and spatiotemporal interaction stability constraint term extracted from the abnormal behavior recognition network to be trained to form the total loss function; update all trainable parameters in the abnormal behavior recognition network through backpropagation until the total loss function converges, thus obtaining the abnormal behavior recognition network.

[0016] The present invention has the following beneficial effects:

[0017] (1) This AI deep learning-based surveillance video target abnormal behavior detection method uses image pyramids, edge detection and discrete curvature calculation to screen key inflection points of the human body and accurately generate key point heat maps. At the same time, it uses superpixel clustering, feature statistics and dimensionality reduction to obtain global semantic feature maps. Subsequently, it uses coordinate mapping and cross attention to build target nodes and scene condition nodes respectively. It can simultaneously refer to changes in human posture and the fit and support relationship between the person and the ground and bed. It can comprehensively distinguish between real risk behaviors and normal body position changes from the two levels of action evolution and environmental linkage. This distinguishes between actions such as lying down to rest and accidental falls that are similar in shape and height, avoids frequent invalid alarms, meets the needs of 24-hour stable monitoring in nursing homes, wards and home monitoring, and effectively reduces the pressure of manual review.

[0018] (2) The AI ​​deep learning-based method for detecting abnormal behavior of surveillance video targets establishes spatial adjacency relationships in the same frame based on human body part markings, builds cross-frame temporal associations using the same part markings, and then establishes global control connections between scene condition nodes and all target nodes, thus fully connecting the human body's own structural associations, temporal action changes, and the constraint associations between the human body and the surrounding scene. At the same time, free interactive hidden nodes are introduced into the graph structure in the graph enhancement layer, so as to autonomously explore the hidden associations between the human body and the environment, thereby avoiding ignoring the continuous posture evolution process of the human body. When encountering slight background disturbances, it is easy to miss potential dangerous actions such as slow imbalance and slight tilting, thereby improving the completeness of abnormal behavior capture.

[0019] (3) The AI ​​deep learning-based surveillance video target abnormal behavior detection method constructs a learnable time bias table through time intervals and generates learnable position codes. In the attention calculation, time bias and position bias are superimposed simultaneously to adapt to the action context and spatial association logic of different time scales. Then, the query vector is filtered by classification to perform spatiotemporal interaction filtering and output accurate behavior judgment results. During the training stage, the classification loss, spatiotemporal bias regularization term and spatiotemporal interaction stability constraint term are combined to adjust the parameters, so that the network can continuously adapt to the characteristics of various indoor monitoring samples. This gives it a more refined ability to distinguish easily confused boundary samples such as slow bending over to pick up objects and real slight falls, thereby improving the reliability of abnormal identification.

[0020] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0021] Figure 1 This is a flowchart of a method for detecting abnormal behavior of targets in surveillance videos based on AI deep learning, according to the present invention.

[0022] Figure 2 This is a flowchart illustrating the specific steps involved in extracting a heatmap of key points of a target in a surveillance video target abnormal behavior detection method based on AI deep learning, as described in this invention.

[0023] Figure 3 This is a flowchart illustrating the specific steps involved in constructing the target node set and the scene condition node set in a method for detecting abnormal behavior of surveillance video targets based on AI deep learning, as described in this invention. Detailed Implementation

[0024] Please see Figure 1 This invention provides a technical solution: a method for detecting abnormal behavior of targets in surveillance videos based on AI deep learning, comprising the following steps: continuously acquiring surveillance video stream data, including several frames of surveillance images; simultaneously performing key point spatial decomposition and scene semantic decomposition; and extracting target key point heatmaps and global semantic feature maps for each frame of surveillance images in parallel; constructing target node sets and scene condition node sets for the target key point heatmaps and global semantic feature maps respectively through coordinate encoding and Gaussian weighted sampling; constructing a surveillance heterogeneous spatiotemporal graph for the target node sets and scene condition node sets through heterogeneous association and cross-frame convergence; inputting the surveillance heterogeneous spatiotemporal graph into an abnormal behavior recognition network pre-trained by a graph enhancement layer, multiple Transformer layers, and a recognition classifier; and outputting abnormal behavior detection results through attention encoding with graph merging and spatiotemporal dual bias superposition.

[0025] Specifically, such as Figure 2 As shown, the specific steps for extracting the heatmap of target key points in each frame of the monitoring image are as follows:

[0026] For each frame of the monitoring image, Gaussian blurring and downsampling are performed stepwise using a preset Gaussian kernel sequence to construct an image pyramid containing multiple scale layers. Canny edge detection is then performed on each scale layer of the image pyramid to obtain a scale layer edge map corresponding to each scale layer. Specifically:

[0027] The current frame monitoring image is denoted as Preset Gaussian kernel sequence and downsampling factor s = 1, 2, 3, ..., S, where S is the preset total number of Gaussian kernels. Perform the following operations in sequence: Apply Gaussian blur to the image, then use a factor Perform bilinear downsampling to obtain the first... Images at various scales, all of which form an image pyramid, are used in one implementation where the base value of the Gaussian kernel sequence is... The value is 1.0, multiplied by a factor of 1.5 at each level, and then downsampled. The value is 0.5.

[0028] For each scale layer image, the Canny edge detection algorithm is used to process it. The first-order partial derivatives of each pixel in the horizontal and vertical directions are calculated to obtain the gradient magnitude map and gradient direction map. Non-maximum suppression is performed on the gradient magnitude map: for each pixel, the gradient magnitude of the pixel is compared with the gradient magnitude of the two adjacent pixels in the positive and negative directions. If the gradient magnitude of the current pixel is not the maximum value among the three, the magnitude of the pixel is set to zero. The suppression window radius is 1 pixel.

[0029] Perform dual threshold filtering: preset low threshold and high threshold In one implementation, values ​​of 50 and 150 (for an 8-bit grayscale image) are taken sequentially, and values ​​higher than [the specified value] are used. Pixels with values ​​between strong and weak edges are marked as strong edge points, those with values ​​between these two are marked as weak edge points, and the rest are set to zero. Finally, edge concatenation is performed: all strong edge points are traversed, and weak edge points within their 8-neighborhoods are re-marked as strong edge points and expanded. Unconnected weak edge points are discarded, resulting in a binary edge map for this scale layer. In this process, edge pixels are set to 1, and non-edge pixels are set to 0. This process is run in parallel across all scales to obtain a set of edge maps that corresponds one-to-one with each scale. ;

[0030] Create a blank matrix of the same size as the edge map of each scale layer. Iterate through all edge pixels in the edge map of each scale layer, calculate the discrete curvature value of each edge pixel, and fill the discrete curvature value into the corresponding pixel coordinate position in the blank matrix. After filling, a scale layer curvature response map corresponding to each scale layer is obtained, which is as follows:

[0031] For scale layer Edge map Create a matrix with the same number of rows and columns. Initialize all elements to zero, then iterate through them. For all pixels at (x, y) with a value of 1, take the pixels before and after that pixel on the edge chain. 1 pixel;

[0032] In one implementation, The value is set to 2, and the discrete curvature value at that point is calculated using the three-point circular arc method. Let the current point be Along the edge chain spacing The two adjacent points are and The curvature is then calculated as the reciprocal of the radius of the circumcircle of the triangle formed by these three points:

[0033] ;

[0034] Among them, in the molecule The scalar absolute value of the cross product of two-dimensional vectors reflects the directed area of ​​the triangle formed by the three points; the denominator is the product of the lengths of the three sides, calculated as follows. Fill in At the corresponding position (x, y), after traversal is complete... That is, the scale layer Curvature response plot;

[0035] The curvature response maps of all scale layers are scaled to the original resolution of the current frame monitoring image. The maximum value of each pixel in the scaled curvature response maps is then used for cross-scale fusion to obtain a multi-scale curvature fusion map, specifically:

[0036] For each scale layer curvature response map It is scaled to the original image using bilinear interpolation. Given the width W and height H, we get , build a with For a fused image F of the same size, calculate the following for each pixel position (x, y): ;

[0037] That is, take the maximum curvature response value at this location in all scale layers to obtain the multi-scale curvature fusion map F;

[0038] Non-maximum suppression is performed on the multi-scale curvature fusion map. The pixels retained after suppression are sorted in descending order of curvature response value (i.e., from largest to smallest) to obtain the target key point set (i.e., the pixel coordinates of the top K pixels are taken as the target key point coordinate set of the current frame). Specifically:

[0039] right Local nonmaximum suppression: For each pixel at (x, y), in its Compare response values ​​within the neighborhood (i.e., the suppression window radius is 1 pixel). If the response value of a pixel is greater than that of all its neighboring pixels, then the original response value of that pixel is retained; otherwise, the response value of that pixel is set to zero, resulting in a suppression map. ;

[0040] collect All non-zero pixel coordinates and their corresponding response values ​​are sorted in descending order of response value. The first K pixel coordinates are selected. In one implementation, K is set to 17, corresponding to the number of major human joints, and are used as the set of target keypoint coordinates for the current frame. The corresponding curvature response value is... ;

[0041] Based on the target key point set, a target key point heatmap for each frame of the monitoring image is generated through spatial Gaussian diffusion. Specifically, the Gaussian kernel center is used for each target key point coordinate in the target key point set, the Gaussian kernel amplitude is used for the curvature response value corresponding to that target key point coordinate, and the Gaussian kernel radius is the product of a preset base radius and the curvature response value. A single-point Gaussian response map for that target key point coordinate is then generated. All K single-point Gaussian response maps are superimposed to obtain the target key point heatmap for the current frame, specifically as follows:

[0042] Preset base radius In one implementation, The value is 3 pixels, for the k-th keypoint Its corresponding curvature response value is ;

[0043] Centered on this key point, with a preset base radius With curvature response value The product of these is used as the Gaussian kernel radius, expressed as the curvature response value. As the Gaussian amplitude, generate a single-point Gaussian response plot. The single-point Gaussian response maps corresponding to all K key points are superimposed pixel by pixel at the same pixel position to finally obtain the target key point heatmap for the current frame.

[0044] In this implementation scheme, the monitoring images are decomposed step by step through a multi-scale image pyramid. Edge detection is used to accurately capture the contour edge information of human limbs. Then, discrete curvature calculation is used to accurately extract key points at the bending points of human joints, thereby avoiding interference from indoor lighting changes, wall textures, and clothing folds. By integrating image detail information from different levels through cross-scale curvature fusion, it avoids missing remote limbs and weak key points under partial occlusion at a single scale. With the addition of a neighborhood screening mechanism, high-quality and effective key points are selected to ensure that the points conform to the arrangement of the human physiological skeleton. Furthermore, a key point heat map is generated based on an adaptive Gaussian diffusion method to conform to the actual spatial distribution of limbs, thus adapting to the complex indoor monitoring scenarios.

[0045] Specifically, the steps for extracting the global semantic feature map of each monitoring image frame are as follows:

[0046] For each frame of the monitored image, linear iterative clustering is performed to segment the image using a preset number of superpixels. During the clustering process, the similarity is measured by the weighted sum of color distance and spatial distance between pixels, and the pixels are aggregated into superpixel blocks to obtain a superpixel segmentation map, which is as follows:

[0047] For the current frame of the monitored image, convert it from the RGB color space to the CIELAB color space. Preset the desired number of superpixels. Calculate the initial grid step size Initial cluster centers are uniformly sampled in the image, and each center is moved to its nearest neighbor. The location with the minimum gradient in the neighborhood;

[0048] And perform iterative clustering: for each pixel i, in its Within the neighborhood, the comprehensive distance to each cluster center j is calculated. This distance is composed of the Euclidean distance in the color space and the Euclidean distance in the spatial coordinates with a specific weight. In one implementation, the compactness weight is 20. Pixels are assigned to the cluster center that minimizes the comprehensive distance, and the cluster center positions are iteratively updated.

[0049] In one implementation, the maximum number of iterations is set to 10, and the cluster center drift convergence threshold is set to 0.5 pixels. The iteration stops when the cluster center position drift is less than the convergence threshold between two consecutive iterations, or when the maximum number of iterations is reached, ultimately yielding a superpixel segmentation map. Each superpixel block It contains a group of spatially adjacent pixels that are similar in color;

[0050] For each superpixel block in the superpixel segmentation map, calculate the first-order color moment, second-order color moment, and texture contrast of all pixels within it, and construct the original feature vector of the corresponding superpixel block (that is, concatenate the first-order color moment, second-order color moment, and texture contrast to form the original feature vector of the superpixel block), specifically as follows:

[0051] For each superpixel block Extract the first-order color moments of all pixels within it in the CIELAB color space. (i.e., mean) and second moment (i.e., variance), texture contrast Defined as the ratio of the standard deviation of pixel brightness to the mean brightness within the superpixel block, the first moment of color, the second moment of color, and the texture contrast are concatenated into an original feature vector with a dimension of 7.

[0052] A feature matrix is ​​constructed using the original feature vectors of all superpixel blocks, and principal component analysis is performed to reduce its dimensionality. The feature vectors of each superpixel block after dimensionality reduction are arranged according to their spatial positions to generate a global semantic feature map for each frame of the monitoring image, specifically as follows:

[0053] All The original feature vectors of each superpixel block are stacked row-wise to form a feature matrix. ,calculate The covariance matrix is ​​calculated and its eigenvalues ​​are decomposed. The first eigenvalues ​​are then taken. The eigenvectors corresponding to the largest eigenvalues ​​constitute the projection matrix. ;

[0054] In one implementation, The value is 64. Calculate the feature matrix after dimensionality reduction. This yields the low-dimensional feature vector for each superpixel block. Construct an image with the same height and width as the original image, but with the same number of channels. Global semantic feature map For each superpixel block, a feature vector is assigned to all pixel locations within its coverage area. If there are pixels not covered by any superpixels, then assign them all-zero vectors. This results in a global semantic feature map for the current frame, where each spatial location corresponds to a... A 3D feature vector, which carries the color statistics and texture information of the local region.

[0055] In this implementation scheme, the monitoring images are aggregated and divided into regions through refined superpixel clustering. By fully combining color association and spatial adjacency association, pixel regularization and grouping are completed, so that the segmented image regions fit the natural scene boundaries of the indoor bed, floor and walls. Color distribution features and texture surface features are uniformly extracted for each region, and the inherent environmental attribute information of different functional areas is completely preserved. Then, redundant and scattered information is removed by feature dimensionality reduction, key environmental semantic content is retained and regularized to generate a global semantic feature map. This can completely restore the real environment layout around the human body, accurately distinguish the area supported by the human body from the open area, and thus stably assist in judging whether the human body is in contact with a reliable support surface and collaboratively distinguish similar human behaviors.

[0056] Specifically, such as Figure 3 As shown, the specific steps for constructing the target node set and the scene condition node set are as follows:

[0057] After thresholding and non-maximum suppression of the heatmap of target key points in the current frame, the set of target key point coordinates for the current frame is obtained, specifically as follows:

[0058] Take the target key point heatmap of the current frame Calculate its global maximum value Set response threshold ;

[0059] In one implementation, A value of 0.1 is used to filter out... For the remaining pixels, perform local non-maximum suppression using a 3×3 neighborhood window (with a suppression window radius of 1 pixel). Extract the coordinates of the local maximum points and select the top K points in descending order of response value (K is the same as before, for example, 17) to obtain the set of target keypoint coordinates. ;

[0060] For each target keypoint coordinate in the target keypoint coordinate set, its x-coordinate and y-coordinate values ​​are respectively subjected to multi-frequency sine and cosine encoding. The encoded vectors of the x-coordinate and y-coordinate values ​​are concatenated to obtain the coordinate encoding vector corresponding to the target keypoint coordinate. According to a preset spatial partitioning rule, a target part marker is assigned to each target keypoint coordinate. The assigned target part marker serves as the node identity identifier for that target node. All target keypoints construct the target node set of the current frame, which includes several target nodes and their corresponding coordinate encoding vectors, specifically as follows:

[0061] The preset encoding dimension D (which is an even number, D=128 in one implementation) is used to determine the coordinates of the k-th keypoint. First, normalize the coordinates to Interval: Define frequency sequence ,in, , respectively and Calculate the sine and cosine values ​​at each frequency, and concatenate the two resulting encoded vectors by channel to form a D-dimensional coordinate encoded vector. ;

[0062] Simultaneously, a location label is assigned to the keypoint according to a preset spatial division rule. Specifically, based on the minimum and maximum ordinates of all keypoints in the current frame, the ordinate range is divided into upper, middle, and lower regions. The middle and lower regions are further subdivided according to the horizontal coordinates, forming several grids. Each grid is pre-assigned a location label, such as head, left shoulder, or right knee. The keypoint obtains its corresponding target location label based on the grid region its coordinates fall into. This marker serves as the node identity identifier for the target node, does not participate in numerical calculations, and is only used to guide the establishment of spatial edges;

[0063] All target nodes constitute the target node set of the current frame. Each node carries node characteristics. (i.e., coordinate encoding vector) and identity identifier (i.e., target area marking);

[0064] As another implementation example, the extracted set of target keypoint coordinates can be input into a pre-trained human pose estimation network (such as OpenPose, HRNet, MediaPipe Pose, etc.), which directly outputs the body part label corresponding to each keypoint, such as left shoulder, right knee, etc.

[0065] Multiple pre-defined spatial Gaussian kernel functions are used to perform weighted sampling on the global semantic feature map to obtain scene condition node feature vectors. A scene condition node set for the current frame is then constructed, comprising several scene condition nodes. Specifically, multiple spatial Gaussian kernel functions with different center positions are used to perform weighted sampling on all pixel coordinate positions of the global semantic feature map. Each spatial Gaussian kernel function sums the feature vectors of each pixel coordinate position according to spatial distance using its own center coordinates, resulting in a scene condition node feature vector. This vector is used as the node feature of that scene condition node. Multiple spatial Gaussian kernel functions yield a corresponding number of scene condition nodes, constituting the scene condition node set assigned to the current frame.

[0066] Preset number of scene condition nodes R=1,2,… In one implementation, The value is set to 6, setting 6 Gaussian kernel functions with different sampling centers. The center coordinates (normalized) can be (0.2, 0.2), (0.8, 0.2), (0.2, 0.8), (0.8, 0.8), (0.5, 0.3), and (0.7, 0.5), respectively. The Gaussian kernel bandwidth is... The value is uniformly set to 0.25;

[0067] For the R-th kernel function, for each pixel position (x, y) in the global semantic feature map G, Gaussian weights are calculated based on its spatial distance from the kernel center. These weights are then used to adjust the feature vector of the pixel position. We perform a weighted average to obtain the node features of the Rth scene condition node. Repeat this process to generate These scene condition nodes constitute the scene condition node set of the current frame. .

[0068] It is important to emphasize that although the initial features of scene condition nodes come from the color statistics and texture information of superpixels, these features will interact deeply with the spatiotemporal features of human key points through an attention mechanism during the training of the abnormal behavior recognition network. The network can automatically learn that when a large number of human key points fall into or stay in a specific scene area for a long time (such as the low-texture area corresponding to the bed surface), it tends to be judged as actively lying down; while when human key points suddenly appear in areas that are not usually where the human body stays for a long time (such as the ground area) and are accompanied by rapid movement, it tends to be judged as falling.

[0069] Therefore, the scene condition nodes do not provide simple color and texture labels, but rather spatial region context anchors that can be used by the network and are related to human behavior decisions. Through large-scale data training, the network can implicitly establish association rules between key points and scene regions, thereby effectively distinguishing behaviors that are similar in appearance but have different environmental interactions.

[0070] The specific steps for constructing a heterogeneous spatiotemporal graph of monitoring are as follows:

[0071] A node set is formed by combining the target node set and the scene condition node set of all frames. Each target node carries its node features (i.e., coordinate encoding vector) and node identity identifier, and each scene condition node carries its node features (i.e., scene condition node feature vector), specifically as follows:

[0072] Suppose the video segment to be detected has a total of T frames, then the th frame... Target node set of the frame and scene condition node set By converging along the time dimension, a complete set of nodes for all frames is obtained, and each target node in the node set is bound to a coordinate encoding vector. and a part mark Each scene condition node is bound to a feature vector. ;

[0073] For target nodes within the same frame, based on the target part markers of each target node, a pre-defined target part adjacency table is queried to identify adjacent node pairs (target node pairs corresponding to node identifiers with direct adjacency relationships recorded in the table are determined as adjacent node pairs), and an unweighted spatial edge is established between each adjacent node pair, specifically as follows:

[0074] Pre-define an adjacency table of target parts that reflects physical connections. This table defines which parts have direct physical connections, such as (head, left shoulder), (left shoulder, torso), (torso, left hip), (left hip, left knee), (left knee, left foot), etc. For any two target nodes within the same frame, their identifiers are extracted. and ,like or If the nodes exist in the adjacency table, a spatial edge is created between them, labeled as a spatial edge, with a fixed weight of 1. This operation is performed independently across all frames, forming a set of spatial edges. ;

[0075] Traverse all frames, and establish an unweighted temporal edge between target nodes with the same target location marker in adjacent frames. All temporal edges constitute cross-frame temporal connections, specifically as follows:

[0076] For adjacent frames and Iterate through all possible part markers. If frame and frame The same mark exists in all of them. For the target node, the two nodes are directly connected to establish a temporal edge, labeled as a temporal edge, with a fixed weight of 1. This connection does not depend on the spatial distance or appearance similarity of the nodes, but only on a stable identity identifier, thus ensuring that a correct cross-frame correspondence can be established even when the target moves quickly. The temporal edges between all adjacent frame pairs constitute a set. ;

[0077] For each frame, between the scene condition node and the target node of that frame, unweighted fully connected control edges are established one by one, specifically as follows:

[0078] For frames The frame Each scenario condition node and All target nodes are connected in pairs, establishing... An unweighted edge is labeled as a control edge, which allows global information about the scene conditions to be completely transmitted to every target node in the frame. All control edges form a set. ;

[0079] By combining the node set, spatial edges, temporal edges, and fully connected control edges, a monitoring heterogeneous spatiotemporal graph is constructed. Specifically, the node set and edge set constructed above are merged to form a heterogeneous graph data structure. This refers to a heterogeneous spatiotemporal graph for monitoring, in which the edge types are heterogeneous, respectively encoding the structural relationships, temporal motion relationships, and scene semantic relationships of the target object.

[0080] In this implementation plan, structured target nodes are constructed by combining key human body point information. Spatial coding is used to enhance the positional association representation of human joint points. Dedicated body part markings are used to uniformly regulate the human body topological association logic, allowing each human body node to take into account both positional features and physiological attribution information. Simultaneously, global semantic features are used to complete the sampling and construction of global scene condition nodes, fully conforming to the actual environmental distribution of the indoor monitoring area and accurately restoring the scene association information of different areas. On this basis, the two types of nodes are integrated and uniformly collected into a complete node set. Spatial edges within the same frame are constructed according to human physiological connection rules, and cross-frame temporal edges are constructed based on body part identification matching. Then, global control edges between human body nodes and scene nodes are connected to build a complete monitoring heterogeneous spatiotemporal graph. This allows for the simultaneous integration of human body structure association, continuous posture change association, and human-machine environment interaction association, thereby improving the overall detection and operation effect.

[0081] Specifically, the steps for outputting the abnormal behavior detection results are as follows:

[0082] In the graph enhancement layer, several trainable interaction vectors are initialized as corresponding free interaction hidden nodes, forming a set of free interaction hidden nodes, which are then added to the node set of the monitored heterogeneous spatiotemporal graph. Fully connected physical inference edges are established between each free interaction hidden node and all target nodes in its own frame, and between each free interaction hidden node and all scene condition nodes in its own frame, respectively, to obtain the enhanced heterogeneous spatiotemporal graph, specifically as follows:

[0083] For each frame of the video clip ,initialization A trainable D-dimensional vector is used as a freely interacting hidden node;

[0084] In one implementation, the number of hidden nodes per frame The value is set to 4, the vector dimension D is 128, the parameters are initialized uniformly using Xavier, and the hidden nodes of all frames constitute a set of freely interacting hidden nodes. These vectors contain no pre-defined physical semantics and are learned entirely by data.

[0085] Will Add nodes to the original heterogeneous spatiotemporal graph. For each freely interacting hidden node, perform two edge operations: establish fully connected inference edges with all target nodes in the frame, and establish fully connected inference edges with all scene condition nodes in the frame. All edges are labeled as inference edges, with a fixed weight of 1. All newly added nodes and edges are merged with the original graph to form an enhanced heterogeneous spatiotemporal graph. ;

[0086] In the first Transformer layer, query vectors, key vectors, and value vectors are generated for each node in the enhanced heterogeneous spatiotemporal graph through linear projection. For each pair of nodes with edge connections, the current node and its neighboring nodes are determined (the node initiating attention calculation is taken as the current node, and the node being followed is taken as the neighboring node). The dot product of the query vector of the current node and the key vector of the neighboring node is calculated. The resulting dot product is used as the original attention score. After spatiotemporal double bias superposition modulation, the value vectors are weighted and summed after Softmax normalization to obtain the bias modulation coding features of the current node in the first Transformer layer, which are as follows:

[0087] This layer has three learnable projection matrices. For enhancing heterogeneous spatiotemporal graphs any node in Its input feature vector is denoted as (This corresponds to the original node features obtained in the above steps, such as the target node's built-in coordinate encoded feature vector), and the query vector is obtained through projection. key vector value vector .

[0088] For any pair of nodes in the graph that have edge connections... The party that initiates the attention calculation is taken as the current node. The party being followed is designated as an adjacent node. Calculate the raw attention score ;

[0089] raw attention score By applying spatiotemporal double bias superposition modulation, the modulated attention score is obtained. , will node Modulated score of all adjacent nodes Attention coefficients are obtained by normalization using the Softmax function. Use these coefficients to represent the value vectors of adjacent nodes. We perform a weighted summation to obtain the output of node z at this layer, and name it the bias modulation coding feature. ;

[0090] The bias modulation-coding features of the first Transformer layer are input into subsequent Transformer layers. The attention encoding operation is repeated in each subsequent Transformer layer to obtain the bias modulation-coding features of each node in each Transformer layer, specifically:

[0091] Each subsequent layer The bias modulation and coding features of the previous layer output are processed in the same way. Each layer uses an independent projection matrix and bias parameters, repeatedly performing linear projection, double bias superposition modulation, and value vector aggregation operations. Assuming there are L layers, after L iterations, each node obtains its final bias modulation coding features. ;

[0092] In the recognition classifier, the final bias modulation and coding features of all nodes output from the last Transformer layer are received, and after spatiotemporal interaction filtering, the abnormal behavior detection results are output.

[0093] The specific steps of spatiotemporal dual-bias superposition modulation are as follows:

[0094] A preset maximum time interval threshold is defined. For each integer time interval between zero and the maximum time interval threshold, a learnable time bias parameter is initialized. A learnable time bias table is constructed using all time bias parameters, specifically as follows:

[0095] Preset maximum time interval threshold It covers the possible distance between any two frames within a video clip, for For each integer value in the dataset, initialize a learnable scalar parameter. All parameters are initially set to zero, and together they form a one-dimensional learnable time bias table. ;

[0096] Based on the temporal interval between the frame indices of the current node and its neighboring nodes, the corresponding target temporal bias parameter is obtained from the learnable temporal bias table as the temporal bias value, and then superimposed on the original attention score to obtain the attention score modulated by the temporal bias, as follows:

[0097] Get the current node Frame index and adjacent nodes Frame index Calculate the absolute time interval Retrieve the index from the learnable time bias table. parameters As a time bias value, the attention score Adding this to the time bias value yields the attention score modulated by the time bias. ;

[0098] The final position encoding vectors corresponding to the current node and its neighboring nodes are extracted from the pre-constructed learnable position encoding matrix. The inner product of the two position encoding vectors is calculated as the position bias value, which is then superimposed on the attention score after temporal bias modulation to obtain the attention score after double bias modulation. Specifically:

[0099] Extract the current node from the learnable position encoding matrix. and adjacent nodes Each corresponding final position encoding vector and Calculate the dot product of these two vectors; this inner product value is the positional bias value. Add this positional bias value to the attention score modulated by the time bias to obtain the final attention score after double bias modulation. .

[0100] The pre-construction steps of the learnable positional encoding matrix are as follows:

[0101] For the target node set, scene condition node set, and free interaction hidden node set in the enhanced heterogeneous spatiotemporal graph, a learnable D-dimensional parameter matrix is ​​randomly initialized for each node set as the node type position encoding matrix for each group of nodes. Each row of the matrix corresponds to a D-dimensional position encoding vector of a node, specifically as follows:

[0102] Create independent learnable parameter matrices for the three different types of node sets. Each matrix has the same number of rows as the maximum number of nodes in the enhanced heterogeneous spatiotemporal graph (e.g., a fixed value determined based on training sample statistics), and the number of columns is D (e.g., 128). At the start of training, the elements in the matrix are initialized by random sampling from a normal distribution with a mean of 0 and a standard deviation of 0.02.

[0103] In practical use, when the current number of actual nodes in the node set is less than the preset maximum total number of nodes, only the first row of actual nodes in the matrix is ​​used, and the remaining rows are ignored through a masking mechanism; when the number of actual nodes exceeds the preset maximum total number of nodes, the nodes are truncated or processed in batches. Each row of the matrix is ​​the node type position encoding vector of the corresponding node (or its pre-assigned position), which is used to learn the spatial interaction patterns unique to this type of node.

[0104] According to the preset region division rules, the enhanced heterogeneous spatiotemporal graph is divided into multiple spatial location regions, and a learnable D-dimensional vector is randomly initialized for each spatial location region as the spatial location encoding vector for that region, specifically as follows:

[0105] The original image space is uniformly divided into In one implementation, the grid... The value is set to 7, resulting in 49 spatial location regions. A learnable D-dimensional vector is initialized for each region. Similarly, a normal distribution with a mean of 0 and a standard deviation of 0.02 is used for random initialization. These vectors are used to capture the absolute and relative spatial location information of nodes in the image.

[0106] The location encoding vector and spatial location encoding vector of each node in the enhanced heterogeneous spatiotemporal graph are determined (based on the node set to which the node belongs, the D-dimensional location encoding vector corresponding to the node is extracted from the corresponding node location encoding matrix; in actual use, the corresponding row is taken according to the order of the node in the node set; based on the pixel coordinates of the node in the image, the spatial location region in which it falls is determined, and the spatial location encoding vector of the spatial location region is extracted), and learnable heterogeneous fusion is performed to obtain the final location encoding vector of the node. (The final location encoding vectors of all nodes) constitute a learnable location encoding matrix, which is as follows:

[0107] For each node in the graph :

[0108] Based on the node set type to which the node belongs, the corresponding row is extracted from the corresponding node set position encoding matrix and used as its node type position encoding vector. ;

[0109] Determine the coordinates of the node (if it is a target node) in the image space and find the grid region it falls into. Extract the spatial location encoding vector of this region. ;

[0110] Introduce two learnable scalar fusion weights and Heterogeneously weighted fusion is performed on the node to obtain its final position encoding vector:

[0111] ;

[0112] The final positional encoding vectors of all nodes constitute a complete learnable positional encoding matrix.

[0113] It should be noted that the positional bias provided by the learnable positional encoding matrix and the temporal bias provided by the learnable temporal bias table are updated synchronously through backpropagation during the training of the abnormal behavior recognition network. The node type encoding weight and spatial location encoding weight are also updated synchronously during training, enabling the network to adaptively learn the fusion ratio of node type information and spatial location information in different scenarios.

[0114] In this implementation scheme, free-interactive hidden nodes are added to the existing heterogeneous spatiotemporal graph, and fully connected physical inference edges are built to enrich the deep interactive association dimension of the graph structure. Multi-layer Transformer structure is used to progressively encode various node features, and multi-dimensional interaction vectors are generated by combining projection mechanism. Attention score is calculated in an orderly manner based on node topological association. At the same time, a temporal dimension masking mechanism and a spatial position fusion encoding mechanism are introduced to complete dual bias modulation, which adaptively fits the relationship between the speed and rhythm of human movements and the actual spatial distribution, and fits the human movement change law in real monitoring scenarios. Finally, the position encoding representation is optimized by relying on multi-weight heterogeneous fusion method. Throughout the process, the model autonomously iterates and updates various bias parameters and fusion weights, thereby deeply mining the implicit association features of human posture temporal changes and environmental spatial linkage, strengthening the feature discrimination ability between highly similar behaviors, and improving the discrimination accuracy of the classification and recognition process.

[0115] Specifically, the steps for spatiotemporal interactive filtering are as follows:

[0116] Learnable attention pooling is applied to the final bias modulation-coding features of all nodes. A cross-attention operation is then performed between a set of learnable classification-selection query vectors and the final bias modulation-coding features of all nodes to generate a spatiotemporal bias feature vector, specifically:

[0117] Before entering the final classification, a spatiotemporal interactive filtering process is performed to initialize a set of learnable classification filtering query vectors. Where C can be set to a fixed value (e.g., C=4), these query vectors are used as the Query, and the final bias modulation coding features of all nodes in the entire enhanced heterogeneous spatiotemporal graph are used. Simultaneously, both the key and value are used to perform a cross-attention calculation, which outputs C feature vectors after information filtering and aggregation, called spatiotemporal bias feature vectors. (like Then, after averaging or splicing, it is mapped to D dimensions through a linear layer.

[0118] The spatiotemporal bias feature vector is classified and mapped to output a multidimensional behavior determination probability vector. Each dimension of the multidimensional behavior determination probability vector corresponds to a probability value of a preset behavior category. The preset behavior categories include at least three categories: high-risk fall, autonomous safe posture, and normal activity. Specifically:

[0119] Construct a learnable classification mapping layer, which contains a weight matrix. and bias (Assuming three types of behavior are defined), calculate A three-dimensional behavior decision probability vector is obtained through the Softmax function. ,in, Corresponding to the probability of a high-risk fall, The probability corresponding to an autonomous and safe posture. The probability corresponding to normal activity;

[0120] The preset behavior category corresponding to the dimension with the highest probability value in the multidimensional behavior judgment probability vector is taken as the behavior judgment result of the target object.

[0121] The pre-training steps for the abnormal behavior recognition network are as follows:

[0122] A dataset of labeled surveillance video samples was collected. Each sample was labeled with a category of normal or abnormal behavior. Abnormal behavior categories included at least high-risk falls and autonomous safety postures, specifically:

[0123] By deploying surveillance cameras in indoor monitoring scenarios, a large number of video clips containing various activities of the target object are collected. Each clip is fixed with T frames. Professionals accurately annotate each clip and assign a category label to each sample, such as high-risk fall, autonomous safe posture, normal activity, etc., to build a supervised training dataset.

[0124] The abnormal behavior recognition network is trained by inputting the surveillance video sample dataset into it and obtaining the predicted behavior judgment result for each sample through forward propagation. Specifically:

[0125] For each sample in the training set, its T consecutive monitoring images are sequentially input into the network to be trained. The network performs forward computation according to the above steps, and finally outputs a predicted multidimensional behavior determination probability vector for each sample. ;

[0126] The behavior classification loss between the predicted behavior judgment result and the corresponding category label is calculated using the cross-entropy loss function. Based on this, the spatiotemporal dual bias regularization term and spatiotemporal interaction stability constraint term extracted from the abnormal behavior recognition network to be trained are combined to form the total loss function (wherein, the spatiotemporal dual bias regularization term is obtained by applying the same distribution consistency constraint to each time bias parameter in the learnable time bias table, and applying the different distribution discriminative constraint to each position encoding vector in the above-mentioned learnable position encoding matrix).

[0127] The spatiotemporal interaction stability constraint term is obtained by extracting the attention scores between the free interactive hidden nodes in the last Transformer layer and each target node in the same frame, after double bias modulation and Softmax normalization, and calculating the difference loss of the attention scores between adjacent frames. Specifically:

[0128] Total loss function during training It consists of three parts:

[0129] ;

[0130] in To balance hyperparameters, in one implementation, The value is 0.1. The value is 0.05;

[0131] Classification loss: Standard cross-entropy loss is used to measure the difference between the predicted probability and the true label;

[0132] Spatiotemporal double bias regularization term: derived from temporal bias consistency loss Distinguishing loss from positional bias Composed of:

[0133] ;

[0134] In one implementation, All values ​​are 0.5;

[0135] Temporal Bias Consistency Loss: Used to constrain that the model should adopt similar temporal attention patterns within the same behavior category. For each behavior category, the temporal bias parameter vectors of all samples in that batch are taken, their intra-class means are calculated, and the squared Euclidean distance between the temporal bias vector of each sample and the mean of that class is calculated and summed to obtain the result. ;

[0136] Position bias discrimination loss: used to ensure that the spatial interaction patterns between nodes are discriminative under different behavior categories, taking the category... Final position encoding vector of all samples The average value is used as the class center of the class. Calculate the squared Euclidean distance between the class centers of each pair of distinct classes, and sum the negatives:

[0137] ;

[0138] This loss drives the class centers of different categories away from each other;

[0139] Spatiotemporal interaction stability constraint: This term ensures that the attention of freely interacting hidden nodes to target parts is smooth in time, implicitly learning physical constraints. From the last Transformer layer, the attention weights between each freely interacting hidden node and each target node in the same frame are extracted. Let the attention weight of the freely interacting hidden node to the target node in frame t be denoted as follows: this weight has undergone double bias modulation and Softmax normalization. Simultaneously, the attention weight of the target node to the freely interacting hidden node in the same frame is also extracted. These two directions constitute a bidirectional association.

[0140] To calculate the degree of change in the bidirectional attention distribution between adjacent frames, one implementation method is to use KL divergence to measure the difference between the two probability distributions. For each pair of adjacent frames t and t+1, the KL divergence of forward attention (from hidden node to target node) and reverse attention (from target node to hidden node) are calculated respectively, and the results for all hidden nodes and target nodes are summed to obtain... ;

[0141] The abnormal behavior recognition network is obtained by updating all trainable parameters in the network through backpropagation until the total loss function converges. Specifically, this involves using a gradient descent-based optimizer (such as Adam) to adjust the total loss function. Perform backpropagation to calculate and update the gradients of all trainable parameters. All trainable parameters include at least: the linear projection matrix of each Transformer layer, the learnable temporal bias table, the learnable positional encoding matrix and its fusion weights, the classification filtering query vector, the weights and biases of the classification mapping layer, and the vectors of all free-interacting hidden nodes. Iterate training until the total loss converges on the validation set to obtain the abnormal behavior recognition network.

[0142] In one implementation, the hyperparameters and convergence determination mechanism of the training process are configured as follows:

[0143] The initial learning rate is set to 1×10. −4 The batch size is set to 16 video segments, and a learning rate decay strategy is adopted: if the total loss on the validation set does not decrease for 5 consecutive training epochs, the learning rate is reduced to one-tenth of the original. At the same time, an early stopping method is used as the convergence criterion: a patience value (e.g., 15) is set. If the validation set loss does not reach a new low for 15 consecutive epochs, training is stopped, and the model parameters with the lowest validation set loss are used as the final pre-trained abnormal behavior recognition network.

[0144] For example, in a training dataset consisting of 500 surveillance video samples, 80% are used for training and 20% for validation. During training, the validation set loss reaches its minimum value of 0.152 in the 23rd round. In the subsequent 15 rounds (rounds 24 to 38), this minimum loss value is not exceeded. Therefore, the early stopping mechanism is triggered before the start of the 39th round of training, and the training is terminated. At this time, the model parameters saved after the 23rd round of training are used as the final network parameters.

[0145] In this implementation scheme, dedicated attention pooling combined with cross-attention operations is used to complete the full-domain node feature selection. It autonomously focuses on effective spatiotemporal features that are highly correlated with changes in human behavior, filters out irrelevant background and redundant information in the image, and then outputs multi-dimensional behavior judgment results through classification mapping to quickly complete the classification of target human behavior categories. In addition, during the network training phase, a comprehensive loss function is jointly constructed by combining multiple dedicated constraints, simultaneously standardizing the temporal attention distribution pattern, optimizing the spatial node interaction and correlation representation, and smoothly constraining the fluctuation range of implicit interaction correlations between adjacent frames. Relying on the gradient descent optimization mechanism, all core parameters of the network are iteratively updated across the entire domain. Combined with reasonable training and control rules, the model convergence judgment is completed, which effectively enhances the model's adaptability to behavioral features in various indoor monitoring scenarios, improves the model's generalization and adaptation level, optimizes the differentiation effect between highly similar body postures and sudden abnormal falls from the source, reduces misjudgments in actual monitoring scenarios, and improves the overall monitoring work efficiency.

[0146] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0147] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for detecting abnormal behavior of targets in surveillance videos based on AI deep learning, characterized in that, Includes the following steps: The system continuously collects monitoring video stream data, which includes several frames of monitoring images, and simultaneously performs key point spatial decomposition and scene semantic decomposition, and extracts target key point heat map and global semantic feature map in parallel. The target node set and the scene condition node set are constructed by using coordinate encoding and Gaussian weighted sampling for the target key point heatmap and the global semantic feature map, respectively. For the target node set and the scene condition node set, a monitoring heterogeneous spatiotemporal graph is constructed through heterogeneous association and cross-frame aggregation; The heterogeneous spatiotemporal graph of the monitoring is input into an abnormal behavior recognition network pre-trained by a graph enhancement layer, multiple Transformer layers, and a recognition classifier. After attention encoding with graph merging and spatiotemporal dual bias, the abnormal behavior detection result is output.

2. The method for detecting abnormal behavior of targets in surveillance videos based on AI deep learning according to claim 1, characterized in that, The specific steps for extracting the heatmap of key target points are as follows: A progressive Gaussian blur and downsampling process is performed using a preset Gaussian kernel sequence to construct an image pyramid, and edge detection is performed to obtain a scale-level edge map. Create a blank matrix, traverse all pixels in the scale layer edge map, calculate the discrete curvature value, fill the discrete curvature value into the blank matrix, and obtain the scale layer curvature response map. The maximum value of each pixel in the curvature response map of the scale layer is taken and fused across scales to obtain a multi-scale curvature fusion map; Non-maximum suppression is applied to the multi-scale curvature fusion map, and the points are sorted in descending order of curvature response values ​​to obtain the target key point set; A heatmap of target key points is generated based on the target key point set through spatial Gaussian diffusion.

3. The method for detecting abnormal behavior of targets in surveillance videos based on AI deep learning according to claim 1, characterized in that, The specific steps for extracting the global semantic feature map are as follows: Linear iterative clustering is performed with a preset number of superpixels to obtain a superpixel segmentation map; For each superpixel block in the superpixel segmentation map, calculate the first color moment, the second color moment, and the texture contrast, and construct the original feature vector of the corresponding superpixel block. The feature matrix is ​​constructed using the original feature vectors of all superpixel blocks, and principal component analysis is performed to reduce the dimensionality, generating a global semantic feature map.

4. The method for detecting abnormal behavior of targets in surveillance videos based on AI deep learning according to claim 1, characterized in that, The specific steps for constructing the target node set and the scene condition node set are as follows: After threshold filtering and non-maximum suppression, the set of target key point coordinates is obtained from the heat map of target key points; For each target key point coordinate in the target key point coordinate set, multi-frequency sine and cosine encoding is performed to obtain a coordinate encoding vector. According to the preset spatial partitioning rules, target part markers are assigned to construct a target node set, which includes several target nodes and their corresponding coordinate encoding vectors. The global semantic feature map is weighted and sampled using multiple pre-defined spatial Gaussian kernel functions to obtain scene condition node feature vectors, and a scene condition node set is constructed, which includes several scene condition nodes.

5. The method for detecting abnormal behavior of targets in surveillance videos based on AI deep learning according to claim 4, characterized in that, The specific steps for constructing a heterogeneous spatiotemporal graph of monitoring are as follows: A node set is composed of the target node set and the scene condition node set; Based on the target part markings of each target node, the preset target part adjacency relationship table is queried to identify adjacent node pairs and establish spatial edges between adjacent node pairs. Establish temporal edges for target nodes with the same target location marker between adjacent frames; Establish fully connected control edges between the scenario condition nodes and the target nodes one by one; By combining the node set, spatial edges, temporal edges, and fully connected control edges, a monitoring heterogeneous spatiotemporal graph is constructed.

6. The method for detecting abnormal behavior of targets in surveillance videos based on AI deep learning according to claim 1, characterized in that, The specific steps for outputting the abnormal behavior detection results are as follows: In the graph enhancement layer, the set of freely interacting hidden nodes is initialized and added to the monitored heterogeneous spatiotemporal graph to obtain the enhanced heterogeneous spatiotemporal graph; In the first Transformer layer, for each pair of nodes with edge connections, the current node and its neighboring nodes are determined, the original attention score is calculated, and after spatiotemporal double bias superposition modulation, the bias modulation coding feature is obtained. The bias modulation coding features are input into subsequent Transformer layers for attention coding operations. In the recognition classifier, the final bias modulation and coding features of all nodes output from the last Transformer layer are received, and after spatiotemporal interaction filtering, the abnormal behavior detection results are output.

7. The method for detecting abnormal behavior of targets in surveillance videos based on AI deep learning according to claim 6, characterized in that, The specific steps of spatiotemporal dual-bias superposition modulation are as follows: Preset the maximum time interval threshold and initialize the time bias parameters to build a learnable time bias table; Based on the current node and its neighboring nodes, the corresponding time bias value is obtained by indexing from the learnable time bias table and superimposed on the original attention score to obtain the attention score modulated by time bias. Extract the final position encoding vectors corresponding to the current node and its neighboring nodes from the pre-constructed learnable position encoding matrix, calculate the position bias value, and superimpose it onto the attention score after time bias modulation to obtain the attention score after double bias modulation.

8. The method for detecting abnormal behavior of targets in surveillance videos based on AI deep learning according to claim 7, characterized in that, The pre-construction steps of the learnable positional encoding matrix are as follows: Randomly initialize the position encoding vector for each node set; According to the preset regional division rules, the enhanced heterogeneous spatiotemporal map is divided into multiple spatial location regions, and the spatial location encoding vector is randomly initialized. The spatial location encoding vector of each node in the enhanced heterogeneous spatiotemporal graph is determined and a learnable heterogeneous fusion is performed to obtain the final location encoding vector, which constitutes a learnable location encoding matrix.

9. The method for detecting abnormal behavior of targets in surveillance videos based on AI deep learning according to claim 6, characterized in that, The specific steps for spatiotemporal interactive filtering are as follows: Learnable attention pooling is applied to the final bias modulation and coding features. The query vector is selected by learnable classification and cross-attention operation is performed with the final bias modulation and coding features to generate a spatiotemporal bias feature vector. The spatiotemporal bias feature vector is classified and mapped to output a multidimensional behavior determination probability vector, and each dimension of the multidimensional behavior determination probability vector corresponds to the probability value of a preset behavior category. The preset behavior category corresponding to the dimension with the highest probability value in the multidimensional behavior determination probability vector is taken as the behavior determination result.

10. The method for detecting abnormal behavior of targets in surveillance videos based on AI deep learning according to claim 1, characterized in that, The pre-training steps for the abnormal behavior recognition network are as follows: Collect a dataset of labeled surveillance video samples; The abnormal behavior recognition network to be trained is input into the monitoring video sample dataset and the predicted behavior judgment result is obtained through forward propagation. The behavior classification loss is calculated using the cross-entropy loss function, and based on this, the spatiotemporal dual bias regularization term and spatiotemporal interaction stability constraint term extracted from the abnormal behavior recognition network to be trained are combined to form the total loss function. The abnormal behavior recognition network is obtained by updating all trainable parameters in the abnormal behavior recognition network through backpropagation until the total loss function converges.

Images