Security guard behavior feature data extraction method and system based on deep learning

Abstract

The application relates to the technical field of behavior recognition, and proposes a security personnel behavior feature data extraction method and system based on deep learning, which comprises the following steps: target detection is performed on a monitoring video stream to obtain a continuous image sequence; heat map regression is performed on the continuous image sequence to obtain posture time sequence features; feature aggregation is performed on the posture time sequence features based on a preset convolutional neural network to obtain short-time action primitives; a behavior semantic graph is constructed by taking the short-time action primitives as nodes, adjacent relationships as edges, and time intervals as weights; a main line manifold is separated from the behavior semantic graph to obtain a behavior feature chain; dynamic Bayesian inference is performed on the behavior feature chain to obtain a behavior intention label; and joint feature coding is performed on the behavior feature chain and the behavior intention label to obtain behavior feature data; the application solves the problems of behavior feature fragmentation and weak intention reasoning capability in the prior art, and improves the structured expression capability of behavior features and the adaptability to high-level applications.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and system for extracting behavioral feature data of security personnel based on deep learning. Background Technology

[0002] In existing technologies, security personnel behavior feature extraction often employs end-to-end deep learning models to classify video clips into actions, outputting pre-defined action labels such as patrolling and standing. This approach highly couples feature extraction with classification tasks, lacking refined modeling of the internal structure of behavior. It struggles to explicitly extract semantically interpretable short-term action primitives from video streams and fails to construct temporal adjacency relationships and evolutionary paths between actions. Consequently, behavioral features appear as isolated sequences of action labels rather than structured behavioral semantic graphs. The intermediate feature representations are highly coupled with specific tasks, making them unsuitable for direct use in high-level applications such as security report generation and behavioral situation assessment.

[0003] In terms of behavioral intent reasoning, existing temporal models mostly infer intent by classifying action sequences, generally lacking explicit extraction mechanisms for the main behavioral path. This makes it difficult to accurately extract key behavioral chains from redundant actions, and they neglect joint representation learning of behavioral features and high-level intent, resulting in insufficient semantic understanding of complex behavioral patterns. When abnormal behavior occurs, existing methods struggle to accurately reverse-locate and analyze the contribution of key action nodes based on temporal relationships, failing to quickly pinpoint the root cause of the anomaly and visualize the intent evolution path, leading to delays in supervisory decision-making. Therefore, there is an urgent need to develop a method capable of extracting structured, interpretable, and semantically meaningful behavioral feature data of security personnel, addressing the problems of feature fragmentation, weak intent reasoning capabilities, and poor adaptability to high-level applications in existing technologies. Summary of the Invention

[0004] This invention provides a method and system for extracting behavioral feature data of security personnel based on deep learning, in order to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides a method for extracting behavioral feature data of security personnel based on deep learning, comprising: S1: Perform target detection on the surveillance video stream to obtain a continuous image sequence of the security personnel; S2: Perform heatmap regression on the continuous image sequence to obtain the temporal characteristics of the security personnel's posture; S3: Based on a preset convolutional neural network, perform feature aggregation on the posture temporal features to obtain short-time action primitives in the posture temporal features; S4: Using the short-time action primitives as nodes, the adjacency relationships of the short-time action primitives as edges, and the time intervals of the short-time action primitives as weights, construct the behavioral semantic graph of the security personnel; S5: Extract the mainline manifold from the behavioral semantic graph to obtain the behavioral feature chain of the security personnel; S6: Perform dynamic Bayesian inference on the behavioral feature chain to obtain the behavioral intent label of the behavioral feature chain; S7: Perform joint feature encoding on the behavioral feature chain and the behavioral intent label to obtain the behavioral feature data of the security personnel.

[0006] In a preferred embodiment, the step of performing target detection on the surveillance video stream to obtain a continuous image sequence of the security personnel includes: Foreground separation is performed on the surveillance video stream to obtain a foreground motion region mask of the surveillance video stream; Morphological filtering is applied to the foreground motion region mask to obtain candidate connected regions of the surveillance video stream; Human body detection is performed on the candidate connected regions to obtain the human body connected regions of the surveillance video stream; Based on the preset identity characteristics of security personnel, the target connected area of ​​the security personnel is selected from the human body connected area; By performing trajectory association and identity identification on the target connected region, the target trajectory of the security personnel can be obtained; Based on the target trajectory, image blocks of the security personnel are extracted from the surveillance video stream, and the size of the image blocks is normalized to obtain a continuous image sequence of the security personnel.

[0007] In a preferred embodiment, performing heatmap regression on the continuous image sequence to obtain the temporal features of the security personnel's posture includes: Frame extraction is performed on the continuous image sequence to obtain a set of image frames of the continuous image sequence; By performing keypoint probability distribution mapping on the images in the image frame set, the multi-channel heatmap tensor of the image is obtained; Non-maximum suppression is applied to the multi-channel heatmap tensor to obtain the sub-pixel coordinates of the joints in the image; By performing temporal correlation on the subpixel coordinates, the motion trajectory of the joint points in the image is obtained; The motion trajectory is reconstructed in time and space to obtain the temporal characteristics of the security personnel's posture.

[0008] In a preferred embodiment, the step of performing feature aggregation on the pose temporal features based on a preset convolutional neural network to obtain short-time action primitives in the pose temporal features includes: Construct a training dataset for the aforementioned pose temporal features; A sliding time window is applied to the training dataset to obtain a local time-domain slice of the training dataset; Based on a preset initial convolutional neural network, temporal convolution is performed on the local temporal slice to obtain the initial feature embedding vector of the local temporal slice; Based on a preset contrastive learning loss function, the initial feature embedding vector is input into the initial convolutional neural network to obtain the iteratively optimized network parameters of the initial convolutional neural network. Based on the iteratively optimized network parameters, the parameters of the initial convolutional neural network are updated to obtain the updated convolutional neural network. A sliding time window is used to extract the attitude temporal features to obtain local temporal slices of the attitude temporal features; Based on the updated convolutional neural network, temporal convolution is performed on the local temporal slice to obtain the temporal feature tensor of the local temporal slice; The local temporal feature tensor is vectorized to obtain the feature embedding vector of the local temporal slice. Clustering and merging the feature embedding vectors yields the feature vector clusters of the local temporal slice; The cluster center of the feature vector cluster is defined as the short-term action primitive of the security personnel.

[0009] In a preferred embodiment, the step of clustering and merging the feature embedding vectors to obtain the feature vector clusters of the local temporal slice includes: Obtain the feature embedding vector set of the local temporal slice; Traverse the set of feature embedding vectors and the preset set of cluster centers, and calculate the affiliation metric between the set of feature embedding vectors and the set of cluster centers; The feature embedding vectors are assigned to the cluster center with the largest assignment metric among the cluster centers to obtain the feature vector cluster of the local temporal slice; The formula for calculating the attribution metric is as follows: ; in, Represents the feature embedding vector With the Cluster centers Attribution metric between them This represents the feature embedding vector currently being traversed. Indicates the first Cluster center vectors Represents spatial scale parameters. Representing the eigenvector With cluster center The Euclidean distance between them Indicates the first The strength of each cluster center.

[0010] In a preferred embodiment, constructing the behavioral semantic graph of the security personnel, using the short-time action primitives as nodes, the adjacency relationships of the short-time action primitives as edges, and the time intervals of the short-time action primitives as weights, includes: The short-time action primitives are sorted temporally and indexed by nodes to obtain a set of node identifiers for the short-time action primitives; Based on the temporal arrangement of the short-time action primitives, the directed adjacency relationships between the short-time action primitives are extracted; Based on the temporal interval of the short-time action primitive, the directed adjacency relationship is weighted and fused to obtain the directed adjacency relationship set of the short-time action primitive; Based on the set of node identifiers and the set of directed adjacency relationships, a semantic graph of the security personnel's behavior is constructed.

[0011] In a preferred embodiment, the step of extracting the mainline manifold from the behavioral semantic graph to obtain the behavioral feature chain of the security personnel includes: Perform topological sorting on the nodes in the behavioral semantic graph to obtain a temporal sorting sequence of the nodes; Based on the temporal sorting sequence, the directed edges in the behavioral semantic graph are backtracked by the maximum weight to obtain the main behavioral paths in the behavioral semantic graph. The path structure of the main behavior path is parsed to obtain the node sequence of the main behavior path; By connecting the nodes in the node sequence together in a behavioral chain, the behavioral feature chain of the security personnel is obtained.

[0012] In a preferred embodiment, the step of performing dynamic Bayesian inference on the behavioral feature chain to obtain the behavioral intent label of the behavioral feature chain includes: The hidden state space of the behavioral feature chain is initialized to obtain the hidden state candidate space of the behavioral feature chain; The temporal transition relationship between adjacent hidden states in the hidden state candidate space is parameterized by the transition probability to obtain the hidden state transition probability distribution of the behavioral feature chain. The observation probability parameterization is performed on the emission relationship between the behavioral feature chain and the hidden states in the hidden state candidate space to obtain the observation probability distribution of the behavioral feature chain. Based on the hidden state transition probability distribution and the observation probability distribution, Viterbi backtracking decoding is performed on the hidden state sequence of the behavioral feature chain to obtain the maximum posterior probability hidden state sequence of the behavioral feature chain. The terminal hidden state of the maximum a posteriori probability hidden state sequence is decoded and output to obtain the behavioral intent label of the security personnel.

[0013] In a preferred embodiment, the step of jointly encoding the behavioral feature chain and the behavioral intent label to obtain the behavioral feature data of the security personnel includes: By projecting the short-time action primitives in the behavioral feature chain into a vector space, the primitive embedding vectors of the short-time action primitives are obtained. The behavioral intent tag is semantically encoded and projected to obtain the tag embedding vector of the behavioral intent tag; The primitive embedding vector and the label embedding vector are fused to obtain the joint feature tensor of the short-time action primitive; The joint feature vector is normalized in its feature range and rearranged in its channels to obtain the behavioral feature data of the security personnel.

[0014] To address the aforementioned problems, this invention also provides a deep learning-based system for extracting behavioral characteristic data of security personnel, the system comprising: The target detection and tracking module is used to perform target detection on the surveillance video stream to obtain a continuous image sequence of the security personnel; The posture temporal feature extraction module is used to perform heatmap regression on the continuous image sequence to obtain the posture temporal features of the security personnel. The short-time action primitive aggregation module is used to perform feature aggregation on the posture temporal features based on a preset convolutional neural network to obtain short-time action primitives in the posture temporal features; The behavioral semantic graph construction module is used to construct the behavioral semantic graph of the security personnel using the short-term action primitives as nodes, the adjacency relationships of the short-term action primitives as edges, and the time intervals of the short-term action primitives as weights. The mainline manifold extraction module is used to extract the mainline manifold from the behavioral semantic graph to obtain the behavioral feature chain of the security personnel. The dynamic Bayesian inference module is used to perform dynamic Bayesian inference on the behavioral feature chain to obtain the behavioral intent label of the behavioral feature chain; The joint feature encoding module is used to perform joint feature encoding on the behavioral feature chain and the behavioral intent label to obtain the behavioral feature data of the security personnel.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention achieves refined modeling and structured representation of security personnel's behavioral features by constructing a complete extraction chain from surveillance video streams to structured behavioral feature data. Addressing the shortcomings of existing deep learning methods that highly couple feature extraction and classification tasks, resulting in outputs of only isolated action labels, this invention first performs target detection and heatmap regression on the surveillance video stream to accurately obtain the temporal features of security personnel's postures, overcoming the problems of inaccurate target detection and low posture estimation accuracy. Then, based on a convolutional neural network, feature aggregation of the temporal features of postures is performed. Through comparative learning and clustering, short-term action primitives with semantic interpretability are extracted, decomposing the continuous action stream into reusable basic action units. This solves the problems of insufficient ability to model the internal structure of behavior and difficulty in extracting fine-grained action primitives in existing technologies. Building upon this foundation, a behavioral semantic graph is constructed using short-term action primitives as nodes, temporal adjacency relationships as edges, and time intervals as weights. This organizes discrete action primitives into a structured graph model reflecting the evolutionary patterns of behavior. Behavioral feature chains are obtained through principal manifold extraction, accurately extracting key behavioral paths representing core behavioral intentions and avoiding interference from redundant actions in behavior analysis. This process transforms fragmented action label sequences from traditional methods into a semantic graph containing temporal structure, adjacency relationships, and interval weights, significantly improving the interpretability and structured representation of behavioral features. This provides high-quality input data for high-level applications such as security report generation and comprehensive behavioral situation assessment.

[0016] 2. This invention achieves probabilistic reasoning of behavioral intent and generation of behavioral features adapted to high-level applications through dynamic Bayesian inference and joint feature encoding. Addressing the shortcomings of existing temporal models, such as the lack of explicit extraction mechanisms for the main behavioral path and the neglect of joint representation learning of behavioral features and high-level intent, this invention performs dynamic Bayesian inference on the behavioral feature chain. Through hidden state space initialization, parameterization of transition and observation probabilities, and Viterbi backtracking decoding, it obtains the maximum a posteriori probability hidden state sequence, thereby outputting behavioral intent labels. This establishes a probabilistic association between behavioral features and high-level semantic intent, solving the problems of weak behavioral intent reasoning ability and lack of uncertainty modeling in existing technologies, and enabling the inference results to possess global optimality and probabilistic interpretability. Furthermore, it performs joint feature encoding on short-term action primitives and behavioral intent labels in the behavioral feature chain, fusing primitive embedding vectors and label embedding vectors to generate unified behavioral feature data. This ensures that the output features not only include the temporal structure of the action sequence but also incorporate high-level semantic intent, effectively avoiding high coupling between feature representation and specific recognition tasks. The generated behavioral feature data is universal and reusable, and can be directly used for high-level applications such as abnormal behavior early warning, personnel performance evaluation, and security data report generation, which greatly improves the automation level and data reuse efficiency of intelligent security systems. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a method for extracting behavioral characteristics of security personnel based on deep learning, as provided in an embodiment of the present invention. Figure 2 A functional block diagram of a security personnel behavior feature data extraction system based on deep learning provided in an embodiment of the present invention; The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0018] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0019] This application provides a method for extracting behavioral feature data of security personnel based on deep learning. The execution entity of this method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the method can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cluster of cloud servers. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0020] Reference Figure 1 The diagram shown is a flowchart illustrating a method for extracting behavioral features of security personnel based on deep learning, according to an embodiment of the present invention. In this embodiment, the method includes: S1: Perform target detection on the surveillance video stream to obtain a continuous image sequence of the security personnel; In this embodiment of the invention, the step of performing target detection on the surveillance video stream to obtain a continuous image sequence of the security personnel includes: Foreground separation is performed on the surveillance video stream to obtain a foreground motion region mask of the surveillance video stream; Morphological filtering is applied to the foreground motion region mask to obtain candidate connected regions of the surveillance video stream; Human body detection is performed on the candidate connected regions to obtain the human body connected regions of the surveillance video stream; Based on the preset identity characteristics of security personnel, the target connected area of ​​the security personnel is selected from the human body connected area; By performing trajectory association and identity identification on the target connected region, the target trajectory of the security personnel can be obtained; Based on the target trajectory, image blocks of the security personnel are extracted from the surveillance video stream, and the size of the image blocks is normalized to obtain a continuous image sequence of the security personnel.

[0021] Each frame of the video stream is compared pixel-by-pixel with a pre-built background model. The background model is obtained by statistically modeling surveillance footage with no moving targets for a long time. For each pixel in the current frame, the difference between its pixel value and the corresponding pixel value in the background model is calculated. When the difference exceeds a set threshold, the pixel is identified as a foreground point. The area formed by all foreground points is extracted to form a foreground motion region mask. This mask is a binary image with the same size as the original image, where the pixel value corresponding to the foreground point is 1 and the pixel value corresponding to the background point is 0, thereby accurately marking the area in the surveillance footage where there are moving targets.

[0022] The algorithm employs a dilation-erosion closing operation. The dilation operation uses a predefined structuring element to slide across the mask image. When a foreground point exists in the area covered by the structuring element, the output pixel corresponding to the center of the structuring element is set as the foreground point, thereby filling small holes inside the foreground region and connecting adjacent broken areas. The erosion operation uses the same structuring element to slide across the dilated image. Only when the entire area covered by the structuring element is a foreground point is the output pixel corresponding to the center of the structuring element set as the foreground point, thereby removing isolated noise points at the edges of the foreground region. After morphological filtering, candidate connected regions are obtained, which are continuous and complete sets of foreground pixel blocks.

[0023] The aspect ratio of the bounding rectangle of each candidate connected region and the pixel area of ​​the region are calculated respectively. The calculated aspect ratio is compared with the standard aspect ratio range of the human body, and the pixel area is compared with the reasonable area range occupied by the human body in the image. When the aspect ratio of a candidate connected region falls within the standard aspect ratio range of the human body and the pixel area falls within the reasonable area range, the region is determined to be a human body connected region. In this way, the part belonging to the human body is filtered out from all motion regions, and non-human motion interference is eliminated.

[0024] The preset security personnel identification features include the color distribution features of the security personnel's uniforms and the identification features of the work badges worn by the security personnel. For each human body connected region, the color histogram of the pixels in the region is extracted and matched with the preset uniform color distribution. At the same time, the presence of a specific shape or pattern of a work badge is detected in the region. When the color matching degree exceeds the set threshold and a work badge is detected, the human body connected region is determined as the target connected region of the security personnel, thereby accurately locating the location of the security personnel from multiple human targets.

[0025] The location distance between the target connected region detected in the current frame and the target trajectory marked in the previous frame is calculated. For each existing trajectory in the previous frame, the Euclidean distance between the predicted position of the trajectory at the current position and the center point of the target connected region in the current frame is calculated. The target connected region in the current frame is associated with the existing trajectory with the closest distance. When a target connected region cannot be associated with any existing trajectory in multiple consecutive frames, a new target trajectory is created for the target connected region and a unique identity is assigned. After frame-by-frame association, the target trajectory of each security personnel during the entire monitoring period is obtained.

[0026] Based on the coordinates of the target trajectory recorded at each time point, an image region bounded by the bounding rectangle of the target connected region is cropped from the corresponding video frame. This image region is then separated from the original video frame to form an image block. Subsequently, the image blocks of different sizes are uniformly scaled to a preset fixed height and fixed width using bilinear interpolation. During the scaling process, the gray values ​​of four adjacent pixels corresponding to the target pixel in the original image are weighted and averaged to obtain the gray value of the scaled pixel. Finally, a continuous image sequence with all image blocks of the same size is obtained.

[0027] The beneficial effects include: foreground separation of surveillance video streams to accurately extract moving target regions and eliminate static background interference; filling holes, connecting broken areas, and removing isolated noise through morphological filtering to obtain regular candidate connected regions, avoiding target misjudgment; filtering regions with aspect ratios and pixel areas conforming to human standards through human body detection to effectively eliminate non-human interference such as vehicles and animals, improving detection accuracy; filtering target connected regions based on uniform color and work badge identification to achieve accurate positioning and identification of security personnel; obtaining continuous target trajectories through trajectory association and identity calibration to ensure tracking stability and identity continuity, overcoming tracking interruptions caused by occlusion or temporary disappearance; and eliminating distance and angle differences through size normalization to provide standardized and well-aligned input data for subsequent attitude temporal feature extraction.

[0028] S2: Perform heatmap regression on the continuous image sequence to obtain the temporal characteristics of the security personnel's posture; In this embodiment of the invention, performing heatmap regression on the continuous image sequence to obtain the temporal features of the security personnel's posture includes: Frame extraction is performed on the continuous image sequence to obtain a set of image frames of the continuous image sequence; By performing keypoint probability distribution mapping on the images in the image frame set, the multi-channel heatmap tensor of the image is obtained; Non-maximum suppression is applied to the multi-channel heatmap tensor to obtain the sub-pixel coordinates of the joints in the image; By performing temporal correlation on the subpixel coordinates, the motion trajectory of the joint points in the image is obtained; The motion trajectory is reconstructed in time and space to obtain the temporal characteristics of the security personnel's posture.

[0029] When extracting frames from a continuous image sequence, each frame in the sequence is extracted sequentially according to its chronological order on the timeline. Each frame corresponds to a complete image at a specific point in time. All extracted frames are arranged in chronological order to form a set of image frames for the continuous image sequence. This set of image frames contains complete image information for security personnel at every moment during the entire monitoring period.

[0030] When mapping the probability distribution of joints in an image frame set, each frame in the image frame set is input into a pre-trained pose estimation network. This network extracts feature maps from the image through multi-layer convolution operations. For each preset human joint type, the network outputs a heatmap with the same size as the original image. The value of each pixel in the heatmap represents the probability value of the joint at that pixel. The heatmaps corresponding to all joint types are stacked together in the order of joint type to form a three-dimensional tensor structure. This tensor is the multi-channel heatmap tensor of the image, where each channel corresponds to the probability distribution map of a joint type.

[0031] When performing non-maximum suppression on a multi-channel heatmap tensor, for each channel of the heatmap tensor, all pixels are traversed on the heatmap of that channel. The probability value of each pixel is compared with the probability values ​​of other pixels in its neighborhood. When the probability value of a pixel is greater than the probability values ​​of all other pixels in its neighborhood, the pixel is retained as a candidate joint position. When the probability value of a pixel is less than or equal to the probability value of any pixel in its neighborhood, the pixel is discarded. After non-maximum suppression, only the position of the pixel with the highest probability value is retained on the heatmap of each channel. The integer coordinates of the pixel are then weighted and averaged with the probability values ​​of the pixels surrounding the pixel in the heatmap to obtain the sub-pixel level precise coordinates. These coordinates are the sub-pixel coordinates of the joints in the image.

[0032] When performing temporal correlation on subpixel coordinates, the distance between the subpixel coordinates of each joint point detected in the current frame image and the corresponding subpixel coordinates of the joint point in the previous frame image is calculated. For each joint point type, the Euclidean distance between the subpixel coordinates of the joint point in the current frame and the subpixel coordinates of the same joint point in the previous frame is calculated. When the distance is less than the preset maximum displacement threshold, the joint point coordinates of the current frame are correlated with the joint point coordinates of the previous frame to form a continuous motion path for the joint point. When the distance is greater than or equal to the preset maximum displacement threshold, a new motion path starting point is created for the joint point in the current frame. The subpixel coordinates of the same joint point type in all frames are connected sequentially in time order to obtain the motion trajectory of each joint point during the entire monitoring period.

[0033] When reconstructing the motion trajectory in time and space, the motion trajectories of all joints are arranged in a fixed order according to the joint type. For each time point, the sub-pixel coordinates of all joint types at that time point are extracted. These sub-pixel coordinates are combined into a coordinate vector according to the order of joint types. This vector describes the overall posture of the security personnel at that time point. The coordinate vectors of all time points are arranged in chronological order to form a two-dimensional matrix structure. The rows of this matrix correspond to the time points, and the columns correspond to the coordinate dimensions of the joints. This matrix is ​​the temporal feature of the security personnel's posture. This feature completely records the full-body posture information of the security personnel at every moment during the entire monitoring period.

[0034] The beneficial effects include: extracting frames from continuous image sequences to obtain image frame sets, splitting the video stream into single-frame images, providing temporally aligned processing units for frame-by-frame pose analysis, and ensuring that each frame is processed independently and accurately. Keypoint probability distribution mapping is performed on the image frame set to obtain a multi-channel heatmap tensor. A pose estimation network then converts each frame into a multi-channel heatmap, transforming keypoint localization into pixel-level probability prediction. This ensures that keypoint location determination has spatial continuity and probabilistic confidence expression, laying the foundation for accurate coordinate extraction. Non-maximum suppression is applied to the multi-channel heatmap tensor to obtain sub-pixel coordinates of the keypoints. By retaining local maximum probability points and discarding non-maximum points, the most probable location of the keypoints is accurately located. Weighted averaging further improves the coordinate accuracy to the sub-pixel level, eliminating localization errors caused by heatmap resolution. By temporally associating subpixel coordinates, the motion trajectories of key points are obtained. By calculating the distance between the same key point in adjacent frames and setting a displacement threshold for matching, discrete coordinates are connected into continuous paths, forming the complete motion trajectory of each key point within the monitoring period. This provides temporally continuous data for analyzing the motion patterns of key points. The motion trajectories are then spatiotemporally reconstructed to obtain the temporal characteristics of the security personnel's posture. All key point motion trajectories are combined into a two-dimensional matrix structure in a fixed order, unifying spatial coordinates and temporal changes within the same data structure. This comprehensively describes the temporal characteristics of the security personnel's full-body posture evolution over time, providing clearly structured and complete input data for subsequent short-term motion primitive extraction.

[0035] S3: Based on a preset convolutional neural network, perform feature aggregation on the posture temporal features to obtain short-time action primitives in the posture temporal features; In this embodiment of the invention, the step of performing feature aggregation on the pose temporal features based on a preset convolutional neural network to obtain short-time action primitives in the pose temporal features includes: Construct a training dataset for the aforementioned pose temporal features; A sliding time window is applied to the training dataset to obtain a local time-domain slice of the training dataset; Based on a preset initial convolutional neural network, temporal convolution is performed on the local temporal slice to obtain the initial feature embedding vector of the local temporal slice; Based on a preset contrastive learning loss function, the initial feature embedding vector is input into the initial convolutional neural network to obtain the iteratively optimized network parameters of the initial convolutional neural network. Based on the iteratively optimized network parameters, the parameters of the initial convolutional neural network are updated to obtain the updated convolutional neural network. A sliding time window is used to extract the attitude temporal features to obtain local temporal slices of the attitude temporal features; Based on the updated convolutional neural network, temporal convolution is performed on the local temporal slice to obtain the temporal feature tensor of the local temporal slice; The local temporal feature tensor is vectorized to obtain the feature embedding vector of the local temporal slice. Clustering and merging the feature embedding vectors yields the feature vector clusters of the local temporal slice; The cluster center of the feature vector cluster is defined as the short-term action primitive of the security personnel.

[0036] The step of clustering and merging the feature embedding vectors to obtain the feature vector clusters of the local temporal slice includes: Obtain the feature embedding vector set of the local temporal slice; Traverse the set of feature embedding vectors and the preset set of cluster centers, and calculate the affiliation metric between the set of feature embedding vectors and the set of cluster centers; The feature embedding vectors are assigned to the cluster center with the largest assignment metric among the cluster centers to obtain the feature vector cluster of the local temporal slice; The formula for calculating the attribution metric is as follows: ; in, Represents the feature embedding vector With the Cluster centers Attribution metric between them This represents the feature embedding vector currently being traversed. Indicates the first Cluster center vectors Represents spatial scale parameters. Representing the eigenvector With cluster center The Euclidean distance between them Indicates the first The strength of each cluster center.

[0037] A training dataset for the posture temporal features is constructed by collecting a large number of posture temporal feature samples of security personnel from historical surveillance videos. Each sample corresponds to a complete action sequence, and each sample is labeled with its action category. All samples and their category labels are aggregated to form a training dataset for posture temporal features. This dataset is used for subsequent contrastive learning training of convolutional neural networks.

[0038] A sliding time window is used to extract the training dataset. A time window of fixed length is set and slides gradually backward from the starting time on the time axis of each sample in the training dataset. Each slide is a fixed step size. Each time the time window is slid, a segment of the attitude temporal features within the time period covered by the time window is extracted from the sample. This segment is a local temporal slice of the training dataset. All local temporal slices together constitute a set of data units for training.

[0039] Each local temporal slice is input into an initial convolutional neural network, which contains multiple temporal convolutional layers. Each temporal convolutional layer uses multiple one-dimensional convolutional kernels to perform sliding convolution operations in the time dimension. The convolutional kernels are multiplied point by point with the input data on the time axis and then accumulated to extract the feature patterns of the local temporal slice at different time scales. After multiple layers of temporal convolution processing, the network outputs a fixed-length feature vector, which is the initial feature embedding vector of the local temporal slice.

[0040] Initial feature embedding vectors corresponding to local temporal slices belonging to the same action category are used as positive sample pairs, and initial feature embedding vectors corresponding to local temporal slices belonging to different action categories are used as negative sample pairs. The contrastive learning loss function is used to calculate the similarity between the two feature embedding vectors in each positive sample pair and the similarity between the two feature embedding vectors in each negative sample pair. The network parameters are adjusted by minimizing the distance between positive sample pairs and maximizing the distance between negative sample pairs. After multiple iterations, the loss value is calculated and backpropagated to obtain a set of network parameters that minimize the contrastive learning loss function. This set of parameters is the iterative optimization network parameters of the initial convolutional neural network.

[0041] The weight and bias parameters of each layer in the initial convolutional neural network are replaced with the corresponding values ​​in the network parameters of the iterative optimization. The network obtained after the replacement is the updated convolutional neural network, which has learned to have the ability to extract features that distinguish different action categories.

[0042] Using the same fixed-length time window and sliding step size as in the training phase, the time window is gradually slid backward from the starting moment on the time axis of the attitude temporal features to be processed. Each time it slides, a segment of the attitude temporal features within the time period covered by the time window is extracted. This segment is the local temporal slice of the attitude temporal features. All local temporal slices are arranged in chronological order, covering the entire time period of the entire attitude temporal features.

[0043] Each local temporal slice is input into the updated convolutional neural network. The temporal convolutional layers in the network use the trained convolutional kernels to perform sliding convolution operations in the time dimension. The convolutional kernels are multiplied point by point with the input data and then accumulated to extract deep feature patterns in the local temporal slices. After multiple layers of temporal convolution, the network outputs a three-dimensional tensor structure. This tensor contains the temporal feature representations of the local temporal slices in multiple channels. This tensor is the temporal feature tensor of the local temporal slice.

[0044] The values ​​of all channels in the temporal feature tensor are flattened into a one-dimensional vector in the time dimension. The flattening operation extracts the values ​​of each channel at all time points on the time axis in the order of channels. These values ​​are then concatenated to form a fixed-length feature embedding vector, which is the feature embedding vector of the local temporal slice. It compresses the temporal feature information in the local temporal slice.

[0045] Obtain the feature embedding vectors corresponding to all local temporal slices to form a feature embedding vector set. Simultaneously, pre-define an initial cluster center set, which contains several cluster center vectors. Each cluster center vector represents a feature template of a short-term action primitive. Iterate through each feature embedding vector in the feature embedding vector set. For the currently iterated feature embedding vector, calculate the affiliation metric between this vector and each cluster center vector in the cluster center set. The affiliation metric calculation consists of two steps: first, calculate the Euclidean distance between the vector and the cluster center vectors; then, use this distance... Divide by the spatial scale parameter, square, add one, and take the reciprocal to obtain the first factor. Then calculate the dot product of the difference between this vector and the cluster center vector, the difference between the cluster center vector and the feature embedding vector, divide the dot product by the product of the Euclidean distance and the strength of the cluster center vector, and add one to obtain the second factor. Multiply the first factor and the second factor to obtain the final assignment metric. Assign the feature embedding vector to the cluster center with the largest assignment metric. All feature embedding vectors belonging to the same cluster center constitute a feature vector cluster, which is the feature vector cluster of the local temporal slice.

[0046] For each feature vector cluster, calculate the average value of all feature embedding vectors within the cluster, and use the calculated average vector as the new cluster center of the cluster. The cluster center vector represents the action pattern corresponding to the cluster in the feature space. Use the cluster center vector as a short-term action primitive of the security personnel. Each short-term action primitive corresponds to a class of basic action units with distinctiveness.

[0047] The beneficial effect is that it constructs a training dataset for temporal pose features, providing sufficient data support for the contrastive learning training of convolutional neural networks and ensuring that the model learns discriminative feature representations. A sliding window is applied to the training dataset, dividing continuous temporal pose features into fixed-length local temporal slices. This allows the network to focus on action patterns within the local window, providing temporally aligned input units for subsequent short-term action primitive extraction. Temporal convolution is performed on the local temporal slices based on the initial convolutional neural network to extract temporal dependencies, compressing the high-dimensional pose data into initial feature embedding vectors. The network is iteratively optimized based on a contrastive learning loss function. By using positive and negative sample pairs to bring feature embedding vectors of the same action category closer together and those of different categories further apart, a semantically discriminative feature space is learned, resulting in an updated convolutional neural network.

[0048] A sliding time window is used to extract the temporal features of the posture, dividing them into local temporal slices of the same size as those used in the training phase to ensure consistency in feature extraction. Temporal convolution is then performed on these local temporal slices using the updated convolutional neural network, extracting deep temporal features using the trained network parameters to obtain a temporal feature tensor. This temporal feature tensor is then vectorized and flattened into fixed-length feature embedding vectors. The feature embedding vectors are then clustered and merged. Similar feature embedding vectors are aggregated to the same cluster center using a membership metric, forming feature vector clusters. The membership metric combines the distance similarity and directional consistency between the feature vectors and the cluster centers to ensure accurate clustering results. The cluster centers of the feature vector clusters are defined as short-term action primitives. Each cluster center represents a common basic action pattern. These short-term action primitives constitute the atomic-level components of security personnel behavior, providing semantically clear and structurally regular node elements for subsequent behavioral semantic graph construction.

[0049] S4: Using the short-time action primitives as nodes, the adjacency relationships of the short-time action primitives as edges, and the time intervals of the short-time action primitives as weights, construct the behavioral semantic graph of the security personnel; In this embodiment of the invention, the step of constructing the behavioral semantic graph of the security personnel, using the short-time action primitives as nodes, the adjacency relationships of the short-time action primitives as edges, and the time intervals of the short-time action primitives as weights, includes: The short-time action primitives are sorted temporally and indexed by nodes to obtain a set of node identifiers for the short-time action primitives; Based on the temporal arrangement of the short-time action primitives, the directed adjacency relationships between the short-time action primitives are extracted; Based on the temporal interval of the short-time action primitive, the directed adjacency relationship is weighted and fused to obtain the directed adjacency relationship set of the short-time action primitive; Based on the set of node identifiers and the set of directed adjacency relationships, a semantic graph of the security personnel's behavior is constructed.

[0050] The short-time action primitives are temporally sorted and indexed. All extracted short-time action primitives are arranged according to their chronological order of appearance in the original attitude temporal features. After the arrangement is completed, each short-time action primitive is assigned a unique number, which is used as the node identifier of the short-time action primitive. The node identifiers of all short-time action primitives are summarized into a set according to the temporal order. This set is the node identifier set of short-time action primitives.

[0051] Based on the temporal arrangement of the short-time action primitives, the directed adjacency relationships between the short-time action primitives are extracted. The sequence of short-time action primitives after temporal arrangement is traversed. Starting from the first short-time action primitive in the sequence, each short-time action primitive is sequentially paired with its immediately adjacent next short-time action primitive in the sequence. The first element of the paired pair represents the start point of the action, and the second element represents the end point of the action. The paired pair indicates that there is a temporal transition between the start action and the end action. All such extracted paired pairs are gathered together to form the directed adjacency relationships between the short-time action primitives.

[0052] The directed adjacency relationships are weighted and fused based on the temporal intervals of the short-time action primitives. For each extracted directed adjacency relationship, the time difference between the end time of the starting action and the start time of the ending action in the adjacency relationship is calculated. This time difference is used as the initial weight of the adjacency relationship. When the same directed adjacency relationship appears multiple times in the temporal sequence, the initial weights of each occurrence are accumulated to obtain the cumulative weight of the directed adjacency relationship. The cumulative weight is used as the final weight of the directed adjacency relationship. All directed adjacency relationships and their corresponding final weights are summarized together to form a set of directed adjacency relationships of short-time action primitives.

[0053] Based on the set of node identifiers and the set of directed adjacency relationships, a semantic graph of the security personnel's behavior is constructed. Each node identifier in the set of node identifiers is treated as a graph node, and the semantic name of the short-term action primitive corresponding to the node is marked on the graph node. Each directed adjacency relationship in the set of directed adjacency relationships is treated as a directed edge, and the starting node and ending node of the directed edge are connected between the graph nodes. The final weight corresponding to the directed edge is marked on the edge. All graph nodes and directed edges together constitute a weighted directed graph structure, which is the semantic graph of the security personnel's behavior.

[0054] The beneficial effects include temporal sorting and node indexing of short-term action primitives to obtain a set of node identifiers. This arranges the unordered short-term action primitives according to time sequence and assigns them unique identifiers, providing a regular node sequence and clear indexing basis for subsequent graph structure construction. Directed adjacency relationships are extracted based on the temporal arrangement of short-term action primitives. By traversing the temporal sequence, the transition relationships between adjacent primitives are obtained, clarifying the temporal dependencies between actions and linking discrete action primitives through temporal order. Weighted fusion of directed adjacency relationships is performed based on the temporal intervals of short-term action primitives. By calculating the time intervals between actions and accumulating the weights of multiple occurrences, the temporal interval information is quantified into edge weights, making the adjacency relationships not only reflect the order of actions but also the tightness of action connections. A behavioral semantic graph is constructed based on a set of node identifiers and a set of directed adjacency relationships. Nodes and weighted directed edges are integrated into a graph structure, forming a structured expression that can completely describe the behavioral patterns of security personnel. This graph contains both the semantic information of action primitives and the temporal sequence and time interval information between actions, providing a complete and well-structured graph data foundation for the subsequent extraction of behavioral feature chains.

[0055] S5: Perform mainline manifold extraction on the behavioral semantic graph to obtain the behavioral feature chain of the security personnel; In this embodiment of the invention, the step of extracting the main manifold from the behavioral semantic graph to obtain the behavioral feature chain of the security personnel includes: Perform topological sorting on the nodes in the behavioral semantic graph to obtain a temporal sorting sequence of the nodes; Based on the temporal sorting sequence, the directed edges in the behavioral semantic graph are backtracked by the maximum weight to obtain the main behavioral paths in the behavioral semantic graph. The path structure of the main behavior path is parsed to obtain the node sequence of the main behavior path; By concatenating the nodes in the node sequence into a behavior chain, the behavioral feature chain of the security personnel can be obtained. Perform topological sorting on the nodes in the behavioral semantic graph, using the directed edges in the behavioral semantic graph as the sequential constraints between nodes. Find all nodes with an in-degree of zero in the graph. A node with an in-degree of zero is a node to which no directed edge points. Output these nodes as starting nodes and remove them from the graph. At the same time, remove all directed edges emanating from these nodes. Repeat this process, finding and outputting nodes with an in-degree of zero in the current graph each time, until all nodes have been output. Arrange the output nodes in the order of removal to obtain the temporal sorting sequence of nodes.

[0056] Based on the temporal sorting sequence, the directed edges in the behavioral semantic graph are backtracked to their maximum weight. Starting from the last node of the temporal sorting sequence, this node is taken as the endpoint of the main behavioral path. The nodes in the temporal sorting sequence are traversed in reverse. For each node currently traversed, all directed edges pointing from this node to its subsequent nodes are found. The directed edge with the largest weight is selected as a segment of the main behavioral path. The endpoint node of this directed edge is added to the main behavioral path. Then, the starting node of this directed edge is taken as the new current node and the backtracking continues. This process is repeated until the first node of the temporal sorting sequence is backtracked. The nodes and edges collected during the backtracking process are rearranged in order from the starting point to the endpoint to obtain the main behavioral path in the behavioral semantic graph.

[0057] The main behavior path is parsed by expanding all directed edges in the main behavior path in sequence, extracting the starting and ending nodes connected by each directed edge, listing all nodes in the order in which they appear in the main behavior path, removing duplicate nodes, and retaining the position of the first appearance of each node in the path, forming a node list without duplicate nodes. This list is the node sequence of the main behavior path.

[0058] The nodes in the node sequence are linked together in a behavior chain. The short-time action primitives corresponding to each node in the node sequence are taken out in sequence according to the order of the nodes in the sequence. These short-time action primitives are linked together by a time sequence connector to form a linear structure. Each element in the linear structure represents a short-time action primitive, and the order between the elements represents the order in which the action primitives appear in the actual behavior. This linear structure is the behavior feature chain of security personnel.

[0059] The beneficial effects include topological sorting of nodes in the behavioral semantic graph to obtain a temporal sorting sequence of nodes, and determining the legal temporal order of nodes through the constraints of directed edges, providing a global temporal reference between nodes for subsequent extraction of the main path. Based on the temporal sorting sequence, the directed edges are backtracked to obtain the main behavioral path. The edge with the largest weight is selected from the endpoint and backtracked in reverse to extract the path with the largest cumulative weight in the behavioral graph while ensuring the correct temporal order. This path represents the most important and representative behavioral pattern of security personnel, eliminating side branches and redundant transition relationships. The path structure of the main behavioral path is parsed to obtain the node sequence of the main behavioral path. The nodes in the main path are extracted sequentially into a linear sequence, simplifying the graph structure into an ordered list of nodes, which facilitates subsequent processing. By linking the nodes in the node sequence into a behavioral chain, a behavioral feature chain of security personnel is obtained. Short-term action primitives are connected into a chain structure in chronological order, forming a chronological sequence that completely describes the core behavioral process of security personnel. This behavioral feature chain not only retains the semantic information of the action primitives, but also reflects the chronological rules of behavioral evolution, providing input data with a clear structure and complete information for subsequent behavioral intention inference.

[0060] S6: Perform dynamic Bayesian inference on the behavioral feature chain to obtain the behavioral intent label of the behavioral feature chain; In this embodiment of the invention, the step of performing dynamic Bayesian inference on the behavioral feature chain to obtain the behavioral intent label of the behavioral feature chain includes: The hidden state space of the behavioral feature chain is initialized to obtain the hidden state candidate space of the behavioral feature chain; The temporal transition relationship between adjacent hidden states in the hidden state candidate space is parameterized by the transition probability to obtain the hidden state transition probability distribution of the behavioral feature chain. The observation probability parameterization is performed on the emission relationship between the behavioral feature chain and the hidden states in the hidden state candidate space to obtain the observation probability distribution of the behavioral feature chain. Based on the hidden state transition probability distribution and the observation probability distribution, Viterbi backtracking decoding is performed on the hidden state sequence of the behavioral feature chain to obtain the maximum posterior probability hidden state sequence of the behavioral feature chain. The terminal hidden state of the maximum a posteriori probability hidden state sequence is decoded and output to obtain the behavioral intent label of the security personnel.

[0061] The hidden state space of the behavioral feature chain is initialized. According to the possible behavioral intentions of the security personnel, a hidden state is defined for each intention category. The hidden states corresponding to all intention categories are aggregated to form a set containing all possible hidden states. This set is the hidden state candidate space of the behavioral feature chain.

[0062] The temporal transition relationship between adjacent hidden states in the hidden state candidate space is parameterized by transition probability. The frequency of each hidden state transitioning to another hidden state in time is statistically analyzed from historical data. The frequency of transition between two hidden states is divided by the total frequency of all transitions starting from the starting hidden state to obtain the transition probability from the starting hidden state to the ending hidden state. The transition probabilities between all pairs of hidden states are organized into a matrix, which is the hidden state transition probability distribution of the behavioral feature chain.

[0063] The emission relationship between the behavioral feature chain and the hidden states in the hidden state candidate space is parameterized by observation probability. Sequence fragments of short-term action primitives in the behavioral feature chain and the corresponding hidden state labels are collected from historical data. For each hidden state, the frequency of observing each short-term action primitive in the hidden state is counted. The frequency is divided by the total frequency of all observed events in the hidden state to obtain the observation probability of observing the short-term action primitive in the hidden state. The observation probabilities of all short-term action primitives in all hidden states are organized into a matrix, which is the observation probability distribution of the behavioral feature chain.

[0064] Based on the hidden state transition probability distribution and the observation probability distribution, Viterbi backtracking decoding is performed on the hidden state sequence of the behavioral feature chain. The first short-term action primitive in the behavioral feature chain is taken as the initial observation. For each hidden state in the hidden state candidate space, the observation probability of observing the first short-term action primitive in that hidden state is calculated, and this probability is taken as the initial path probability of that hidden state. For each observation starting from the second short-term action primitive in the behavioral feature chain, for each hidden state in the hidden state candidate space, the transition probability from all possible previous hidden states to the current hidden state is calculated by multiplying it by the path probability of the previous hidden state, and the maximum value is taken as the path probability of the current hidden state. The previous hidden state that generated the maximum value is recorded. All short-term action primitives in the behavioral feature chain are processed in this way. After processing the last observation, the hidden state with the highest path probability among all hidden states corresponding to the last observation is found as the optimal hidden state of the endpoint. Starting from this hidden state, backtracking is performed along the recorded previous hidden state pointer, and the hidden states at each time point are extracted in turn. These hidden states are arranged in chronological order to obtain the maximum a posteriori probability hidden state sequence of the behavioral feature chain.

[0065] The terminal hidden state of the maximum a posteriori probability hidden state sequence is decoded and output. The hidden state corresponding to the last time point in the maximum a posteriori probability hidden state sequence is extracted. This hidden state represents the behavior intention category corresponding to the final behavior feature chain. The behavior intention label corresponding to this hidden state is used as the output to obtain the behavior intention label of the security personnel.

[0066] The beneficial effects include initializing the hidden state space of the behavioral feature chain to obtain a hidden state candidate space, transforming the possible behavioral intentions of security personnel into discrete hidden states, and establishing a clear semantic space for subsequent probabilistic inference. Transition probability parameterization is applied to the temporal transition relationships between adjacent hidden states in the hidden state candidate space to obtain a hidden state transition probability distribution. Historical data is used to statistically learn the temporal evolution patterns between hidden states, enabling the inference process to utilize natural transition patterns between behavioral intentions. Observation probability parameterization is applied to the emission relationship between the behavioral feature chain and hidden states to obtain an observation probability distribution. Historical data is used to learn the probability of observing specific short-term action primitive sequences under different behavioral intentions, establishing a correlation model between behavioral features and intentions. Viterbi backtracking decoding is performed based on the hidden state transition probability distribution and the observation probability distribution to obtain the maximum a posteriori probability hidden state sequence. A dynamic programming algorithm is used to find the most probable hidden state path globally, comprehensively considering the dual probabilities of the observation sequence and hidden state transitions, ensuring the global optimality of the inference result. The terminal hidden state of the maximum a posteriori probability hidden state sequence is decoded and output to obtain the behavioral intent label of the security personnel. The temporal reasoning results are focused on the final intent category. The output behavioral intent label accurately reflects the final intent of the security personnel in the whole behavior process, and provides high-level semantic guidance for the joint encoding of subsequent behavioral feature data.

[0067] S7: Perform joint feature encoding on the behavioral feature chain and the behavioral intent label to obtain the behavioral feature data of the security personnel.

[0068] In this embodiment of the invention, the step of jointly encoding the behavioral feature chain and the behavioral intent label to obtain the behavioral feature data of the security personnel includes: By projecting the short-time action primitives in the behavioral feature chain into a vector space, the primitive embedding vectors of the short-time action primitives are obtained. The behavioral intent tag is semantically encoded and projected to obtain the tag embedding vector of the behavioral intent tag; The primitive embedding vector and the label embedding vector are fused to obtain the joint feature tensor of the short-time action primitive; The joint feature vector is normalized in its feature range and rearranged in its channels to obtain the behavioral feature data of the security personnel.

[0069] The short-term action primitives in the behavioral feature chain are projected into a vector space. The cluster center vector corresponding to each short-term action primitive in the behavioral feature chain is input into a preset linear projection layer. The projection layer maps the cluster center vector from the original feature space to a low-dimensional embedding space through matrix multiplication. The mapped vector retains the semantic features of the short-term action primitive but reduces the dimension. This vector is the primitive embedding vector of the short-term action primitive. The primitive embedding vectors corresponding to all short-term action primitives are arranged in order of their position in the behavioral feature chain to form a primitive embedding vector sequence.

[0070] The behavioral intent label is semantically encoded and projected, converting it into a one-hot encoded vector. The dimension of this vector is equal to the total number of hidden states in the hidden state candidate space. The position corresponding to the behavioral intent label in the vector is set to 1, and the other positions are set to 0. This one-hot encoded vector is input into a preset semantic embedding layer. The embedding layer maps the one-hot encoded vector to a low-dimensional semantic embedding space through matrix multiplication. The mapped vector is the label embedding vector of the behavioral intent label.

[0071] Feature fusion is performed on the primitive embedding vector and the label embedding vector. Each primitive embedding vector in the primitive embedding vector sequence is concatenated with the label embedding vector. The two vectors are joined end to end to form a longer vector. This vector integrates the local action features of the short-term action primitive and the global semantic features of the overall behavioral intent. The vector obtained by concatenating all primitive embedding vectors with the label embedding vector is arranged in the order of the behavioral feature chain to form a two-dimensional tensor structure. Each row of this tensor corresponds to the fused feature of a short-term action primitive. This tensor is the joint feature tensor of the short-term action primitive.

[0072] The joint feature tensor is subjected to feature range normalization and channel rearrangement. Each value in the joint feature tensor is normalized, and the mean and standard deviation of all values ​​in the joint feature tensor are calculated. Each value is subtracted from the mean and divided by the standard deviation, so that the distribution of all values ​​is adjusted to a standard distribution with zero mean and unit variance, thus completing the feature range normalization. Then, the normalized joint feature tensor is reorganized according to the preset channel rearrangement rules, and the channels that were originally arranged in the order of action primitives are adjusted to the order of semantic category grouping. The final tensor structure is obtained after rearrangement, which is the behavioral feature data of security personnel.

[0073] The beneficial effects include: projecting short-term action primitives in the behavioral feature chain into a vector space to obtain primitive embedding vectors, compressing high-dimensional cluster center vectors into low-dimensional embedding representations, reducing feature dimensionality while preserving the semantic discriminativeness of action primitives, facilitating subsequent fusion processing; projecting semantic space encoding onto behavioral intent labels to obtain label embedding vectors, converting discrete intent labels into continuous low-dimensional vector representations, enabling the fusion of intent semantics and action primitive features in the same space; and fusing primitive embedding vectors with label embedding vectors to obtain a joint feature tensor, combining local action features with global intent semantics through concatenation, ensuring that the representation of each short-term action primitive includes contextual information of the overall behavioral intent, enhancing the semantic richness of feature expression. By performing eigenrangement and channel rearrangement on the joint feature tensor, behavioral feature data of security personnel is obtained. Normalization eliminates the dimensional differences between different feature dimensions, giving the feature data a uniform numerical range. Channel rearrangement optimizes the feature organization structure. The final output behavioral feature data includes both the temporal structure of action sequences and high-level intent semantics, and has a standardized data format. It can be directly used for high-level applications such as security report generation, behavior situation analysis, and abnormal behavior early warning, realizing a universal expression of the behavioral features of security personnel.

[0074] like Figure 2 The diagram shown is a functional block diagram of a security personnel behavior feature data extraction system based on deep learning, provided in an embodiment of the present invention.

[0075] The deep learning-based security personnel behavior feature data extraction system 100 described in this invention can be installed in an electronic device. Depending on the functions implemented, the deep learning-based security personnel behavior feature data extraction system 100 may include a target detection and tracking module 101, a posture temporal feature extraction module 102, a short-term action primitive aggregation module 103, a behavior semantic graph construction module 104, a mainline manifold extraction module 105, a dynamic Bayesian inference module 106, and a joint feature encoding module 107. The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, stored in the memory of the electronic device.

[0076] In this embodiment, the functions of each module / unit are as follows: The target detection and tracking module 101 is used to perform target detection on the surveillance video stream to obtain a continuous image sequence of the security personnel; The posture temporal feature extraction module 102 is used to perform heatmap regression on the continuous image sequence to obtain the posture temporal features of the security personnel. The short-time action primitive aggregation 103 is used to perform feature aggregation on the posture temporal features based on a preset convolutional neural network to obtain short-time action primitives in the posture temporal features; The behavior semantic graph construction module 104 is used to construct the behavior semantic graph of the security personnel using the short-time action primitives as nodes, the adjacency relationships of the short-time action primitives as edges, and the time intervals of the short-time action primitives as weights. The mainline manifold extraction module 105 is used to perform mainline manifold extraction on the behavioral semantic graph to obtain the behavioral feature chain of the security personnel. The dynamic Bayesian inference module 106 is used to perform dynamic Bayesian inference on the behavioral feature chain to obtain the behavioral intent label of the behavioral feature chain. The joint feature encoding module 107 is used to perform joint feature encoding on the behavior feature chain and the behavior intent label to obtain the behavior feature data of the security personnel.

[0077] In the several embodiments provided by this invention, it should be understood that the disclosed methods and systems can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0078] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0079] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0080] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0081] This application embodiment can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for extracting behavioral characteristics of security personnel based on deep learning, characterized in that, The method includes: S1: Perform target detection on the surveillance video stream to obtain a continuous image sequence of the security personnel; S2: Perform heatmap regression on the continuous image sequence to obtain the temporal characteristics of the security personnel's posture; S3: Based on a preset convolutional neural network, perform feature aggregation on the posture temporal features to obtain short-time action primitives in the posture temporal features; S4: Using the short-time action primitives as nodes, the adjacency relationships of the short-time action primitives as edges, and the time intervals of the short-time action primitives as weights, construct the behavioral semantic graph of the security personnel; S5: Perform mainline manifold extraction on the behavioral semantic graph to obtain the behavioral feature chain of the security personnel; S6: Perform dynamic Bayesian inference on the behavioral feature chain to obtain the behavioral intent label of the behavioral feature chain; S7: Perform joint feature encoding on the behavioral feature chain and the behavioral intent label to obtain the behavioral feature data of the security personnel.

2. The method for extracting behavioral feature data of security personnel based on deep learning as described in claim 1, characterized in that, The step of performing target detection on the surveillance video stream to obtain a continuous image sequence of the security personnel includes: Foreground separation is performed on the surveillance video stream to obtain a foreground motion region mask of the surveillance video stream; Morphological filtering is applied to the foreground motion region mask to obtain candidate connected regions of the surveillance video stream; Human body detection is performed on the candidate connected regions to obtain the human body connected regions of the surveillance video stream; Based on the preset identity characteristics of security personnel, the target connected area of ​​the security personnel is selected from the human body connected area; By performing trajectory association and identity identification on the target connected region, the target trajectory of the security personnel can be obtained; Based on the target trajectory, image blocks of the security personnel are extracted from the surveillance video stream, and the size of the image blocks is normalized to obtain a continuous image sequence of the security personnel.

3. The method for extracting behavioral feature data of security personnel based on deep learning as described in claim 1, characterized in that, The step of performing heatmap regression on the continuous image sequence to obtain the temporal features of the security personnel's posture includes: Frame extraction is performed on the continuous image sequence to obtain a set of image frames of the continuous image sequence; By performing keypoint probability distribution mapping on the images in the image frame set, the multi-channel heatmap tensor of the image is obtained; Non-maximum suppression is applied to the multi-channel heatmap tensor to obtain the sub-pixel coordinates of the joints in the image; By performing temporal correlation on the subpixel coordinates, the motion trajectory of the joint points in the image is obtained; The motion trajectory is reconstructed in time and space to obtain the temporal characteristics of the security personnel's posture.

4. The method for extracting behavioral feature data of security personnel based on deep learning as described in claim 1, characterized in that, The method based on a preset convolutional neural network performs feature aggregation on the pose temporal features to obtain short-time action primitives in the pose temporal features, including: Construct a training dataset for the aforementioned pose temporal features; A sliding time window is applied to the training dataset to obtain a local time-domain slice of the training dataset; Based on a preset initial convolutional neural network, temporal convolution is performed on the local temporal slice to obtain the initial feature embedding vector of the local temporal slice; Based on a preset contrastive learning loss function, the initial feature embedding vector is input into the initial convolutional neural network to obtain the iteratively optimized network parameters of the initial convolutional neural network. Based on the iteratively optimized network parameters, the parameters of the initial convolutional neural network are updated to obtain the updated convolutional neural network. A sliding time window is used to extract the attitude temporal features to obtain local temporal slices of the attitude temporal features; Based on the updated convolutional neural network, temporal convolution is performed on the local temporal slice to obtain the temporal feature tensor of the local temporal slice; The local temporal feature tensor is vectorized to obtain the feature embedding vector of the local temporal slice. Clustering and merging the feature embedding vectors yields the feature vector clusters of the local temporal slice; The cluster center of the feature vector cluster is defined as the short-term action primitive of the security personnel.

5. The method for extracting behavioral feature data of security personnel based on deep learning as described in claim 4, characterized in that, The step of clustering and merging the feature embedding vectors to obtain the feature vector clusters of the local temporal slice includes: Obtain the feature embedding vector set of the local temporal slice; Traverse the set of feature embedding vectors and the preset set of cluster centers, and calculate the affiliation metric between the set of feature embedding vectors and the set of cluster centers; The feature embedding vectors are assigned to the cluster center with the largest assignment metric among the cluster centers to obtain the feature vector cluster of the local temporal slice; The formula for calculating the attribution metric is as follows: ； in, Represents the feature embedding vector With the Cluster centers Attribution metric between them This represents the feature embedding vector currently being traversed. Indicates the first Cluster center vectors Represents spatial scale parameters. Representing the eigenvector With cluster center The Euclidean distance between them Indicates the first The strength of each cluster center.

6. The method for extracting behavioral feature data of security personnel based on deep learning as described in claim 1, characterized in that, The process of constructing a behavioral semantic graph of the security personnel, using the short-time action primitives as nodes, their adjacency relationships as edges, and their time intervals as weights, includes: The short-time action primitives are sorted temporally and indexed by nodes to obtain a set of node identifiers for the short-time action primitives; Based on the temporal arrangement of the short-time action primitives, the directed adjacency relationships between the short-time action primitives are extracted; Based on the temporal interval of the short-time action primitive, the directed adjacency relationship is weighted and fused to obtain the directed adjacency relationship set of the short-time action primitive; Based on the set of node identifiers and the set of directed adjacency relationships, a semantic graph of the security personnel's behavior is constructed.

7. The method for extracting behavioral feature data of security personnel based on deep learning as described in claim 1, characterized in that, The step of extracting the mainline manifold from the behavioral semantic graph to obtain the behavioral feature chain of the security personnel includes: Perform topological sorting on the nodes in the behavioral semantic graph to obtain a temporal sorting sequence of the nodes; Based on the temporal sorting sequence, the directed edges in the behavioral semantic graph are backtracked by the maximum weight to obtain the main behavioral paths in the behavioral semantic graph. The path structure of the main behavior path is parsed to obtain the node sequence of the main behavior path; By connecting the nodes in the node sequence together in a behavioral chain, the behavioral feature chain of the security personnel is obtained.

8. The method for extracting behavioral feature data of security personnel based on deep learning as described in claim 1, characterized in that, The step of performing dynamic Bayesian inference on the behavioral feature chain to obtain the behavioral intent label of the behavioral feature chain includes: The hidden state space of the behavioral feature chain is initialized to obtain the hidden state candidate space of the behavioral feature chain; The temporal transition relationship between adjacent hidden states in the hidden state candidate space is parameterized by the transition probability to obtain the hidden state transition probability distribution of the behavioral feature chain. The observation probability parameterization is performed on the emission relationship between the behavioral feature chain and the hidden states in the hidden state candidate space to obtain the observation probability distribution of the behavioral feature chain. Based on the hidden state transition probability distribution and the observation probability distribution, Viterbi backtracking decoding is performed on the hidden state sequence of the behavioral feature chain to obtain the maximum posterior probability hidden state sequence of the behavioral feature chain. The terminal hidden state of the maximum a posteriori probability hidden state sequence is decoded and output to obtain the behavioral intent label of the security personnel.

9. The method for extracting behavioral feature data of security personnel based on deep learning as described in claim 1, characterized in that, The step of jointly encoding the behavioral feature chain and the behavioral intent label to obtain the behavioral feature data of the security personnel includes: By projecting the short-time action primitives in the behavioral feature chain into a vector space, the primitive embedding vectors of the short-time action primitives are obtained. The behavioral intent tag is semantically encoded and projected to obtain the tag embedding vector of the behavioral intent tag; The primitive embedding vector and the label embedding vector are fused to obtain the joint feature tensor of the short-time action primitive; The joint feature vector is normalized in its feature range and rearranged in its channels to obtain the behavioral feature data of the security personnel.

10. A deep learning-based system for extracting behavioral characteristics of security personnel, characterized in that, The system for implementing the deep learning-based security personnel behavior feature data extraction method of claim 1 includes: The target detection and tracking module is used to perform target detection on the surveillance video stream to obtain a continuous image sequence of the security personnel; The posture temporal feature extraction module is used to perform feature aggregation on the posture temporal features based on a preset convolutional neural network to obtain short-term action primitives in the posture temporal features; The short-time action primitive aggregation module is used to construct the behavioral semantic graph of the security personnel using the short-time action primitive as nodes, the adjacency relationship of the short-time action primitive as edges, and the time interval of the short-time action primitive as weights. The behavioral semantic graph construction module is used to construct the behavioral semantic graph of the security personnel using the short-term action primitives as nodes, the adjacency relationships of the short-term action primitives as edges, and the time intervals of the short-term action primitives as weights. The mainline manifold extraction module is used to extract the mainline manifold from the behavioral semantic graph to obtain the behavioral feature chain of the security personnel. The dynamic Bayesian inference module is used to perform dynamic Bayesian inference on the behavioral feature chain to obtain the behavioral intent label of the behavioral feature chain; The joint feature encoding module is used to perform joint feature encoding on the behavioral feature chain and the behavioral intent label to obtain the behavioral feature data of the security personnel.

Images