A campus personnel abnormal behavior recognition method and device based on an improved PoseConv3D model and a medium

Abstract

The application provides a campus personnel abnormal behavior recognition method and device based on an improved PoseConv3D model and a medium. The method comprises the following steps: acquiring campus personnel behavior video data; performing human body detection and posture estimation on the video frames, generating key point heat maps and limb heat maps according to the bone key point coordinates and confidence, and stacking the heat maps along the time dimension to form a three-dimensional posture heat map body; performing behavior recognition by using a trained DRT-PoseConv3D model and outputting a behavior category. The DRT-PoseConv3D model is based on the PoseConv3D, introduces a DPTAM mechanism to jointly model the channel dimension and the time dimension, enhances the attention to key fine-grained action changes, introduces a RefConv module to establish the cross-channel parameter correlation of the convolution kernel through refocusing transformation, enhances the global feature capturing capability, and introduces a TCN network to capture long-time dependence by using a causal expansion convolution. Compared with the prior art, the application can improve the recognition accuracy of similar behaviors, complex behaviors and long-time action sequences, has good stability, anti-background interference capability and engineering application value.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and intelligent security technology, and in particular to a method, device and medium for identifying abnormal behavior of personnel on campus based on an improved PoseConv3D model. Background Technology

[0002] In recent years, with the continuous increase in the number of university students and the increasingly complex and changeable campus environment, campus safety accidents have also become more frequent. Investigations of numerous accident cases have revealed that many safety incidents are closely related to abnormal behavior by campus personnel. Currently, many campuses primarily rely on traditional manual patrols and video surveillance to monitor campus personnel. However, these methods suffer from high costs, low efficiency, and strong subjectivity, making it difficult to promptly and reliably detect hidden safety hazards within the campus environment.

[0003] With the rapid development of deep learning, deep learning-based behavior recognition methods are gradually replacing methods based on manual features. Existing behavior recognition methods typically rely on video data from surveillance cameras, using modalities such as skeletal keypoints, RGB frames, and optical flow to recognize human actions. Among these, RGB frames and optical flow data have high computational resource requirements and are easily affected by background environment, occlusion, and lighting changes. In contrast, behavior recognition methods based on skeletal keypoints have lower complexity and better robustness to background changes.

[0004] The PoseConv3D model represents skeletal key points or limb information as a 3D heatmap and uses a 3D convolutional network for human behavior recognition, showing some applicability in skeleton-based human behavior recognition. However, in the task of identifying abnormal behavior on campus, the following shortcomings still exist: First, campus personnel behaviors are diverse, and similar behaviors such as running and walking, pushing and punching have highly overlapping spatiotemporal features, making it difficult to fully capture key fine-grained movement changes using only ordinary 3D convolutions; Second, complex behaviors such as pushing, punching, kicking, jumping, falling, crawling through fences, and climbing over fences consist of multiple sub-actions, and the inter-channel convolutional kernels of ordinary 3D convolutions are relatively independent, making it difficult to effectively model the global correlation between different body parts; Third, behaviors such as crawling through fences, climbing over fences, and falling have long durations, and ordinary 3D convolutions are limited by fixed receptive fields, making it difficult to capture long-term dependencies in long-term sequences.

[0005] Therefore, it is necessary to provide a method for identifying abnormal behavior of personnel on campus that can take into account key fine-grained features, cross-channel global correlation features, and long-term time-series dependency features, so as to improve the accuracy and reliability of behavior recognition in campus scenarios. Summary of the Invention

[0006] The purpose of this invention is to propose a method, device, and medium for identifying abnormal behavior of personnel on campus based on an improved PoseConv3D model, in order to overcome the shortcomings of existing PoseConv3D models in identifying abnormal behavior on campus, such as insufficient capture of key fine-grained motion changes, insufficient modeling of cross-channel global correlations, and insufficient modeling of long-term action sequence dependencies. To achieve the above objectives, this invention proposes a method for identifying abnormal behavior of personnel on campus based on an improved PoseConv3D model, comprising the following steps: Obtain video data on the behavior of people on campus; Human targets are detected in video frames, and human pose is estimated for the detected human targets to obtain the coordinates of human skeleton key points and their corresponding confidence scores. Based on the coordinates of key points of the human skeleton and their corresponding confidence levels, key point heatmaps and limb heatmaps are generated, and the posture heatmaps corresponding to consecutive video frames are stacked along the time dimension to obtain a three-dimensional posture heatmap volume. The trained DRT-PoseConv3D model is tested using the aforementioned three-dimensional pose heatmap, and the behavior categories of campus personnel are output. The DRT-PoseConv3D model is based on the PoseConv3D model and introduces the DPTAM mechanism, RefConv module and TCN network into the three-dimensional convolutional backbone network. The overall structure of the DRT-PoseConv3D model includes an input convolutional layer, multiple 3D residual layers, the DPTAM mechanism, the RefConv module, the TCN network, a global average pooling layer, and a fully connected layer. After the 3D pose heatmap is input into the DRT-PoseConv3D model, it first passes through the input convolutional layer for initial feature mapping and channel dimensionality upscaling to obtain initial spatiotemporal features. Then, the initial spatiotemporal features are input into the 3D residual layer for stepwise feature extraction. Each 3D residual layer receives the features output from the previous layer. First, the channel dimension and temporal dimension of the features are jointly weighted by attention using the DPTAM mechanism to obtain attention-enhanced features. Then, the attention-enhanced features are processed by 3D convolution using the RefConv module to obtain cross-channel correlation enhancement features. Finally, the cross-channel correlation enhancement features are added element-wise with the residual branch features to obtain the output features of the current 3D residual layer. After passing through the three-dimensional residual layer, high-level spatiotemporal features are obtained. These high-level spatiotemporal features are then input into the TCN network to perform long-term dependency modeling along the time dimension, resulting in temporal enhancement features. These temporal enhancement features are then input into the global average pooling layer to compress the time dimension, yielding a behavior representation vector. Finally, the behavior representation vector is input into the fully connected layer and the Softmax function is used to obtain the predicted probability of each campus personnel behavior category.

[0007] Furthermore, the DPTAM mechanism is set in the three-dimensional residual layer to perform joint attention weighting on the channel and temporal dimensions of the input features. The DPTAM mechanism includes a dimension rearrangement module, a spatial global pooling module, a temporal enhancement branch, a channel attention branch, a feature cross-fusion module, and a temporal weight generation module. The processing steps of the DPTAM mechanism include: denoting the input features entering the DPTAM mechanism as:

[0008] in, Indicates batch size, Indicates the number of time frames. Indicates the number of channels. and These represent the height and width of the feature map, respectively. First, the input features Dimensional rearrangement is performed to make the temporal and channel dimensions the main dimensions for subsequent attention modeling. Then, the rearranged features are analyzed in terms of spatial dimension. and Perform global average pooling to obtain global descriptors for time and channels:

[0009] in, Indicates the first The sample, the first The first channel in the Global response value over each time frame; One-dimensional convolution is performed on the global descriptor along the time dimension to learn the action change relationship between different time frames and obtain temporal enhancement features. At the same time, the global descriptor is pooled along the time dimension to obtain channel description vectors. The channel description vectors are input into a one-dimensional convolutional layer and a sigmoid activation function to generate global channel attention weights. The global channel attention weights are multiplied element-wise with the temporal enhancement features and added element-wise with the time and channel global descriptors to obtain joint enhancement features. The joint enhancement features are input into a convolutional layer and an activation function to generate temporal importance weights. The temporal importance weights are applied to the original input features to obtain the output features of the DPTAM mechanism.

[0010] Furthermore, the temporal enhancement branch consists of a one-dimensional convolutional layer that performs one-dimensional convolution processing on the global descriptor along the temporal dimension; the channel attention branch includes pooling operations along the temporal dimension, a one-dimensional convolutional layer, and a sigmoid activation function; and the temporal weight generation module consists of a one-dimensional convolutional layer and a sigmoid activation function.

[0011] Furthermore, the RefConv module is set in the 3D residual layer to replace the ordinary 3D convolution and establish cross-channel parameter relationships between convolution kernels; the RefConv module first obtains the convolution kernel parameters of the corresponding ordinary 3D convolution layer in the pre-trained PoseConv3D model as the base convolution kernel W. b During training, the basic convolutional kernel remains frozen and does not participate in gradient updates; subsequently, trainable refocusing transformation parameters are set outside the basic convolutional kernel. The refocusing transformation parameters are used to perform cross-channel transformation on the basic convolutional kernel, establishing a correlation between the convolutional kernel parameters corresponding to different input channels and different output channels; the refocusing transformation process is represented as:

[0012] in, This represents the refocusing transformation function. This represents the refocusing convolution kernel obtained by transforming the basic convolution kernel; in order to make the refocusing convolution kernel... With basic convolution kernel The kernel sizes are consistent, and the convolution transformation operation uses padding to keep the kernel size unchanged; when the attention enhancement features output by the DPTAM mechanism are input into the RefConv module, the RefConv module uses the refocusing convolution kernel W. b The attention-enhanced features are subjected to 3D convolution operations to obtain the output features of the RefConv module.

[0013] Furthermore, the refocusing transformation function is specifically implemented using a convolution operation; the basic convolution kernel W... b As input, the trainable refocusing transformation parameters Perform convolution operations and output refocused convolution kernels .

[0014] Furthermore, the TCN network is positioned after multiple 3D residual layers to perform long-term dependency modeling on the high-level spatiotemporal features output by the 3D residual layers; the processing steps of the TCN network include: denoting the high-level spatiotemporal features output by the last 3D residual layer as: ; in, Indicates batch size, Indicates the number of channels. Indicates the number of time frames. and These represent the height and width of the feature map, respectively. First, the spatial dimensions of the high-level spatiotemporal features F are aggregated to obtain a temporal feature sequence dominated by the time dimension: ; in, This represents a temporal feature sequence obtained by aggregating high-level spatiotemporal features F through spatial dimensions; The temporal feature sequence is input into the temporal convolution module, which includes causal dilated convolution, batch normalization, ReLU activation function, and Dropout operation. When calculating features at time t, the causal dilated convolution only uses features from the current time and previous historical time steps. The calculation process is as follows:

[0015] in, Indicates the first The convolution output at time step, Indicates the kernel size. Represents the kernel index. Indicates the convolution kernel weights, Indicates the expansion factor. This represents the input features at the current or historical time; by setting an expansion factor. This expands the effective receptive field of the temporal convolution module in the temporal dimension; The TCN network sets up a residual branch to pass the original temporal feature sequence obtained after spatial dimension aggregation to the output. When the number of channels of the original temporal feature and the temporal enhancement feature are inconsistent, the channels of the original temporal feature are adjusted by convolution. The temporal enhancement feature and the residual branch feature are added element by element to obtain the output feature of the TCN network.

[0016] Furthermore, the keypoint heatmap generation process uses a two-dimensional Gaussian mapping function to generate the heatmap centered on the coordinates of key points in the human skeleton; for the k-th keypoint, the calculation process of its corresponding two-dimensional keypoint heatmap is as follows:

[0017] ; in, The variance of the two-dimensional Gaussian kernel function; a limb heatmap was also generated through Gaussian mapping. , No. The limbs are located at two key points. and Between; functions Used to calculate points to line segment distance, Key points and Line segments between; superimpose all generated two-dimensional heatmaps along the time dimension to obtain a three-dimensional posture heatmap volume with dimensions of K×T×H×W; the limb heatmap generation process connects adjacent human skeletal key points to form limb line segments according to the human skeletal topology, and generates a limb heatmap based on the distance from any pixel position in the limb heatmap to the limb line segment.

[0018] This invention also proposes a campus personnel abnormal behavior recognition device based on an improved PoseConv3D model, including a memory, a processor, and a program stored in the memory and capable of running on the processor. When the processor executes the program, it implements the campus personnel abnormal behavior recognition method based on the improved PoseConv3D model. The device includes a video acquisition module, a pose estimation module, a heatmap generation module, a model recognition module, and an alarm output module. The video acquisition module is used to receive campus surveillance videos or video clips uploaded by acquisition devices. The pose estimation module is used to detect human targets and locate key points in video frames. The heatmap generation module is used to generate a three-dimensional pose heatmap based on key point coordinates and confidence levels. The model recognition module is used to call the trained DRT-PoseConv3D model to obtain the behavior category. The alarm output module is used to output alarm information when the recognition result is abnormal behavior.

[0019] The present invention also proposes a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the campus personnel abnormal behavior identification method based on the improved PoseConv3D model; the storage medium includes a read-only memory, a random access memory, a disk, an optical disk, a solid-state drive, or flash memory.

[0020] Compared with the prior art, the advantages of the present invention are: 1) This invention uses the DPTAM mechanism to jointly model the channel dimension and the time dimension, which can pay more attention to the key fine-grained information changes in the action sequence and improve the ability to distinguish similar behaviors such as running and walking, pushing and boxing.

[0021] 2) This invention uses the RefConv module to establish cross-channel parameter correlation of convolution kernels through refocusing transformation, which enhances the model's ability to capture global correlation features between different body parts and improves the recognition ability of complex behaviors such as pushing, punching, kicking, jumping, falling, crawling through fences and climbing over fences.

[0022] 3) This invention expands the temporal receptive field by using causal dilated convolution through a TCN network, which can effectively capture long-term dependencies in long-term action sequences such as crawling through fences, climbing over fences, and falling down.

[0023] 4) This invention uses human posture heatmaps as input, which has better lightweight and anti-background interference capabilities compared to directly using RGB frames or optical flow, making it suitable for real-time or near-real-time behavior recognition applications in campus smart security scenarios. Attached Figure Description

[0024] Figure 1 This is a flowchart of the skeleton behavior recognition method based on the improved PoseConv3D model of the present invention; Figure 2 This is a schematic diagram of human skeletal joints in one embodiment of the present invention; Figure 3 This is a schematic diagram of the DRT-PoseConv3D network architecture in one embodiment of the present invention; Figure 4 This is a network structure diagram of the DPTAM mechanism in one embodiment of the present invention; Figure 5 This is a network structure diagram of the RefConv module in one embodiment of the present invention; Figure 6 This is a flowchart of the TCN temporal convolution process in one embodiment of the present invention; Figure 7 This is a diagram illustrating the effect of campus personnel behavior recognition in one embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be further described below.

[0026] like Figures 1 to 7 As shown, this invention provides a method for identifying abnormal behavior of people on campus based on an improved PoseConv3D model. This method takes a video of campus personnel behavior as input, converts the coordinates of two-dimensional human skeleton key points in the video frame into a three-dimensional pose heatmap using a human pose estimation module, and then uses the DRT-PoseConv3D model to complete behavior recognition and classification output. The overall processing flow of this invention includes three core steps: a video data acquisition module acquires the video of campus personnel behavior to be identified; a pose estimation module detects human targets and locates key points in the video frame; a heatmap generation module generates a three-dimensional pose heatmap based on key point coordinates and confidence levels; and the DRT-PoseConv3D model performs feature extraction and behavior classification on the three-dimensional pose heatmap, ultimately outputting the campus personnel behavior category.

[0027] S100: During the video data acquisition stage, surveillance cameras or video capture devices in the campus setting transmit video streams from monitored areas requiring security management, such as campus roads, playgrounds, dormitories, and fenced areas, to the behavior recognition system of this embodiment. After receiving the video data, the video acquisition module first performs preprocessing operations on the video, including frame rate unification (standardizing the frame rates of videos from different sources to a uniform value), resolution adjustment (unifying the video screen size to a preset resolution to ensure the stability of subsequent processing), human target cropping (extracting regions of interest containing human targets from complete video frames using target detection algorithms), invalid segment removal (filtering out invalid frames without human targets or video segments of too low quality to perform pose estimation), and time segment truncation (dividing long videos into short segments of fixed duration for easier model processing). After the above preprocessing, the video segments to be identified are output as a frame sequence to the pose estimation module for further processing.

[0028] S200: In the pose estimation stage, the pose estimation module first uses a human target detection algorithm to detect human targets in the video frames, obtaining the bounding boxes of the human targets. The human target detection algorithm can employ mature target detection networks such as the YOLO series and Faster R-CNN. This embodiment of the invention selects YOLOv5 as the human detection algorithm, which has a good balance between detection accuracy and speed, suitable for real-time or near-real-time processing requirements in campus monitoring scenarios. The human target detection algorithm outputs the bounding box coordinates of each detected human target, represented by the pixel coordinates of the upper left and lower right corners. Next, based on the human target bounding boxes, the pose estimation module performs cropping, scale normalization, and spatial alignment processing on the human target region. The cropping operation extracts the image region containing the human target from the original video frame according to the bounding box coordinates; scale normalization scales the cropped image to a preset standard size to adapt to the input requirements of the pose estimation network; spatial alignment ensures that the spatial position of the same human target in the image remains consistent across different frames, facilitating the subsequent construction of pose sequences along the time dimension.

[0029] After the above processing, the pose estimation module uses a top-down human pose estimation network to estimate the pose of the target area of ​​the person, outputting the coordinates of the human skeleton keypoints and their corresponding confidence scores. This embodiment of the invention uses HRNet as the top-down human pose estimation network. This network maintains high-resolution representation throughout the entire processing and can output relatively accurate human skeleton keypoint localization results. The HRNet network outputs the two-dimensional coordinate information of 17 human skeleton keypoints conforming to the COCO standard, as well as the confidence score corresponding to each keypoint, such as... Figure 2As shown, the 17 key points include the nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle. The key point coordinates are represented in pixel coordinates, and the confidence score reflects the network's confidence in the accuracy of each key point's localization, ranging from 0 to 1.

[0030] Step S300: In the heatmap generation stage, the heatmap generation module generates keypoint heatmaps and limb heatmaps based on the coordinates of key points in the human skeleton and their corresponding confidence levels. The keypoint heatmap generation process uses the coordinates of key points in the human skeleton as the center and generates keypoint heatmaps through a two-dimensional Gaussian mapping function. For the k-th keypoint, its corresponding two-dimensional keypoint heatmap... The calculation process is as follows:

[0031] in, Let be the variance of the two-dimensional Gaussian kernel function.

[0032] The limb heatmap generation process, based on the human skeletal topology, connects adjacent key points of the human skeleton to form limb segments, and generates the limb heatmap based on the distance from any pixel position in the limb heatmap to the limb segment. The human skeletal topology defines the connection relationships between 17 key points, such as the left shoulder connecting to the left elbow to form the left upper arm segment, the left elbow connecting to the left wrist to form the left forearm segment, and the left shoulder connecting to the right shoulder to form the shoulder segment, etc. For the l-th limb, it is located between two adjacent key points. and The calculation process for the limb thermometer is as follows: ; function Used to calculate points to line segment distance, Key points and The line segments between them. and These represent the confidence scores corresponding to the two key points. The smaller of the confidence scores of the two key points is taken as the intensity modulation factor of the limb heatmap. Through the above calculation process, a high-response region is formed along the line segment connecting adjacent key points in the limb heatmap; the closer the pixel position is to the line segment, the higher the response value.

[0033] The keypoint heatmaps or limb heatmaps corresponding to consecutive video frames are stacked along the time dimension to obtain a 3D pose heatmap volume. The dimensions of the 3D pose heatmap volume are represented as K×T×H×W, where K represents the number of channels in the pose heatmap. When using keypoint heatmaps as input, K equals the number of human keypoints (17), and when using limb heatmaps as input, K equals the number of limb channels; T represents the number of time frames, i.e., the number of stacked video frames; H and W represent the height and width of the heatmap, respectively. To reduce redundant computation and ensure the temporal consistency of the input data, the heatmap generation module performs uniform frame sampling on the video segment, that is, uniformly extracts T frames from the complete video segment for pose estimation and heatmap generation. Then, the pose heatmap region is cropped based on the center of the human target bounding box to ensure that the 3D pose heatmap volume remains consistent in both the time and spatial dimensions. The cropping operation uses the center of the human target as the origin and extracts a square region of fixed size to ensure that the spatial position of the human target in different frames remains aligned in the heatmap.

[0034] Step S400: The overall architecture of the DRT-PoseConv3D model is as follows Figure 3 As shown, this model is based on the PoseConv3D model and introduces three improved modules into the 3D convolutional backbone network: the DPTAM mechanism, the RefConv module, and the TCN network. The overall structure of the DRT-PoseConv3D model includes an input convolutional layer (ConvBlock), multiple 3D residual layers, the DPTAM mechanism, the RefConv module, the TCN network, a global average pooling layer, and a fully connected layer. After the 3D pose heatmap volume is input into the DRT-PoseConv3D model, it first passes through the input convolutional layer for preliminary feature mapping and channel dimensionality upscaling, mapping the input K-channel heatmap data into 32-channel initial spatiotemporal features. The input convolutional layer uses 3D convolution operations, performing convolution operations simultaneously in the time dimension, spatial height dimension, and spatial width dimension to achieve preliminary extraction of spatiotemporal features and channel expansion of the heatmap data.

[0035] After processing by the input convolutional layers, the initial spatiotemporal features are sequentially input into the first, second, and third level 3D residual layers for progressive feature extraction. Each level of the 3D residual layer employs a residual connection structure, containing two paths: a main branch and a residual branch. The main branch is processed sequentially through the DPTAM mechanism and the RefConv module, while the residual branch directly performs identity mapping. The outputs of the main branch and the residual branch are fused element-wise, and the fused result serves as the output feature of the current 3D residual layer, which is then input into the next level of the 3D residual layer. The number of output channels for the three levels of the 3D residual layer is set to 128, 256, and 512, respectively. The number of feature channels is gradually increased with increasing network depth, while the spatial size of the feature maps is gradually compressed. This allows lower-level features to focus on extracting local pose change information, while higher-level features focus on extracting action category discrimination information.

[0036] The DPTAM mechanism is set in the main branch of the 3D residual layer to perform joint attention weighting on the channel and temporal dimensions of the input features, enhancing the model's attention to key fine-grained action changes. For example... Figure 4 As shown, the DPTAM mechanism consists of a dimensionality rearrangement module, a spatial global pooling module, a temporal enhancement branch, a channel attention branch, a feature cross-fusion module, and a temporal weight generation module. The DPTAM mechanism receives input features from the previous layer. ,in Indicates batch size, Indicates the number of time frames. Indicates the number of channels. and These represent the height and width of the feature map, respectively.

[0037] The DPTAM mechanism first processes the input features. Dimensional rearrangement is performed, making the temporal and channel dimensions the primary dimensions for subsequent attention modeling. The rearranged feature input space global pooling module performs global average pooling operations on spatial dimensions H and W, compressing the spatial information of each channel in each time frame into a single value, resulting in global descriptors for time and channels.

[0038] in, Indicates the first The sample, the first The first channel in the Global response value over each time frame; After obtaining the temporal and channel global descriptors, the DPTAM mechanism processes the global descriptors along the temporal dimension to generate temporal enhancement features and global channel attention weights. The temporal enhancement branch 403 consists of a one-dimensional convolutional layer that performs one-dimensional convolution processing on the global descriptor along the temporal dimension, learning the action change relationships between different time frames and outputting temporal enhancement features. The channel attention branch 404 includes pooling operations along the temporal dimension, a one-dimensional convolutional layer, and a sigmoid activation function: first, the global descriptor is pooled along the temporal dimension to compress the temporal dimension and obtain channel description vectors; then, the channel description vectors are input into a one-dimensional convolutional layer for inter-channel information interaction; finally, the sigmoid activation function generates global channel attention weights.

[0039] The feature cross-fusion module receives temporal enhancement features and global channel attention weights. It performs element-wise multiplication of the global channel attention weights with the temporal enhancement features, selectively enhancing the temporal enhancement features through channel attention weights. Simultaneously, it adds the global channel attention weights element-wise with both temporal and channel global descriptors, achieving cross-dimensional feature fusion and outputting joint enhancement features. These joint enhancement features are input to the temporal weight generation module, which consists of a one-dimensional convolutional layer and a sigmoid activation function, generating temporal importance weights. These temporal importance weights are then element-wise multiplied with the original input features, enabling the model to adaptively highlight the responses of key time frames and key channels, resulting in the output features of the DPTAM mechanism—the attention enhancement features. The attention enhancement features maintain the same dimension as the input features and are input to the RefConv module in the same three-dimensional residual layer.

[0040] The RefConv module is located in the main branch of the 3D residual layer, after the DPTAM mechanism. It replaces ordinary 3D convolutions and establishes cross-channel parameter relationships between convolution kernels. Figure 5 As shown, the RefConv module consists of a basic convolutional kernel, trainable refocusing transformation parameters, a refocusing transformation function, and a processing flow for performing 3D convolution operations using the transformed convolutional kernel. The RefConv module first obtains the convolutional kernel parameters corresponding to the ordinary 3D convolutional layer in the pre-trained PoseConv3D model as the basic convolutional kernel. The basic convolutional kernel remains frozen during training and does not participate in gradient updates, thus preserving the effective feature extraction capabilities learned by the pre-trained model. The trainable refocusing transformation parameters are a matrix of learnable parameters added in addition to the basic convolutional kernel, denoted as... The refocusing transformation parameters are used to perform cross-channel transformation on the basic convolution kernel, establishing a correlation between the convolution kernel parameters corresponding to different input channels and different output channels.

[0041] The refocusing transformation process is represented as follows:

[0042] in, This represents the refocusing transformation function. This represents the refocusing convolution kernel obtained by transforming the basic convolution kernel. To make the refocusing convolution kernel... With basic convolution kernel The kernels are of the same size, and the convolution transformation operation uses a padding method to keep the kernel size unchanged.

[0043] When the attention-enhanced features output by the DPTAM mechanism are input into the RefConv module, the RefConv module uses a refocusing convolution kernel. The attention-enhancing features are subjected to 3D convolution to obtain the output features of the RefConv module. The RefConv output features are then element-wise added to the identity mapping features of the residual branches in the 3D residual layer to obtain the final output features of the 3D residual layer. The RefConv module enhances the parameter correlation between convolutional kernels of different channels through a refocusing transformation mechanism, thereby improving the model's ability to express the coordinated changes of different body parts in complex behaviors.

[0044] After processing through three levels of 3D residual layers, the resulting high-level spatiotemporal features are input into the TCN network for long-term dependency modeling. The TCN network expands the model's receptive field in the temporal dimension, enabling the model to fully utilize action evolution information over a longer time range and improve its ability to recognize long-term action sequences. Figure 6 As shown, the TCN network includes a spatial dimension aggregation module, a causal dilated convolution module, a batch normalization module, and a residual connection module. The TCN network receives the high-level spatiotemporal features output from the last 3D residual layer: ,in, Indicates batch size, Indicates the number of channels. Indicates the number of time frames. and These represent the height and width of the feature map, respectively.

[0045] The spatial dimension aggregation module aggregates the spatial dimensions of the high-level spatiotemporal features F, compressing the spatial information in the height dimension H and the width dimension W to obtain a temporal feature sequence dominated by the time dimension:

[0046] in, This represents a temporal feature sequence obtained by aggregating high-level spatiotemporal features F through spatial dimensions; The temporal convolution module includes causal dilated convolution, batch normalization, ReLU activation function, and Dropout operation. When calculating features at time t, the causal dilated convolution module only uses features from the current time and previous historical times, without incorporating information from future times. Its calculation process is represented as follows:

[0047] in, Indicates the first The convolution output at time step, Indicates the kernel size. Represents the kernel index. Indicates the convolution kernel weights, Indicates the expansion factor. This represents the input features at the current or historical time. An expansion factor can be set. This expands the effective perception of the temporal convolution module in the time dimension, enabling the model to capture dependencies spanning dozens of frames or even longer periods.

[0048] Following causal dilated convolution, batch normalization is performed sequentially to normalize the convolution output and stabilize the training process. Then, ReLU activation is applied to introduce a nonlinear transformation, and Dropout is performed to randomly discard some neurons to prevent overfitting. After these processes, the temporal augmentation features are obtained. When the number of channels in the original temporal features differs from that in the temporal augmentation features, the residual connection module adjusts the number of channels in the original temporal features passed through the residual branch using convolution operations, matching the feature dimensions of the main branch and the residual branch. Subsequently, the temporal augmentation features are added element-wise to the adjusted residual branch features to obtain the output features of the TCN network.

[0049] The output features of the TCN network are input into a global average pooling layer to compress the temporal features, transforming the temporal features at different time points into a fixed-length behavior representation vector. The global average pooling operation averages the temporal features. Finally, the behavior representation vector is input into a fully connected layer, where the Softmax function calculates the predicted probability distribution for each category, and the category with the highest probability value is taken as the behavior recognition result for the current video segment.

[0050] like Figure 7 As shown in the embodiments of the present invention, the campus personnel behavior categories defined include nine categories: climbing over fences, boxing, running, falling, kicking, jumping, pushing, walking, and crawling through fences. In practical application scenarios, walking is set as the normal behavior category, and the other eight categories are set as abnormal behavior categories. When the recognition result output by the DRT-PoseConv3D model is an abnormal behavior, the system automatically triggers an alarm mechanism and outputs alarm information to the campus security management platform.

[0051] The training process for the DRT-PoseConv3D model first requires constructing a video sample dataset containing multiple categories of campus human behavior. The dataset is obtained through real-world campus scenes or simulated recordings, and the video content covers the nine behavior categories mentioned above. The dataset is randomly divided into a training set and a test set in an 8:2 ratio, with the training set containing 3554 video samples and the test set containing 888 video samples. After the dataset is constructed, human detection, human pose estimation, and pose heatmap generation are performed on each video in the dataset to obtain a 3D pose heatmap for training.

[0052] The model training process was implemented using the PyTorch deep learning framework. The batch size was set to 32, and the training ran for 100 epochs. The optimizer employed stochastic gradient descent with momentum, with an initial learning rate of 0.1, momentum of 0.9, and weight decay coefficient of 0.0003. During training, the learning rate was dynamically adjusted using cosine annealing, and the model parameters were iteratively updated using backpropagation and gradient descent. The model was evaluated using recognition accuracy as the primary metric. The recognition accuracy on the test set met expectations, validating the effectiveness of the proposed method for identifying abnormal behavior of personnel on campus.

[0053] In a specific practical application scenario, the campus personnel abnormal behavior recognition system of this invention is deployed on a smart security platform on a university campus. Surveillance cameras on campus collect video streams from various areas in real time. The video streams are first transmitted to a video acquisition module for preprocessing. The preprocessed video frames are input to a pose estimation module, which outputs the coordinates and confidence scores of key points on the human skeleton through an HRNet network. A heatmap generation module converts the key point coordinates into a three-dimensional pose heatmap, which is then input into a trained DRT-PoseConv3D model for behavior recognition. When the model identifies abnormal behaviors such as pushing, punching, kicking, falling, climbing over fences, crawling under fences, jumping, or running in the video, the system automatically generates an alarm message and pushes it to the terminal devices of campus security personnel. The alarm message includes information such as the type of abnormal behavior, the time of occurrence, and the location of the monitored area, facilitating timely response and handling by security personnel. When the model identifies only normal behaviors such as walking (708) in the video, the system continues monitoring without triggering an alarm.

[0054] By jointly modeling the channel and temporal dimensions through the DPTAM mechanism, the DRT-PoseConv3D model of this embodiment can adaptively enhance the response of discriminative temporal frames and channels when handling similar behaviors such as running and walking, pushing and boxing, significantly improving the ability to distinguish between similar behaviors. By using the RefConv module to establish cross-channel parameter correlations of convolutional kernels through refocusing transformation, the model of this embodiment can effectively model the global collaborative changes between different body parts when handling complex behaviors such as pushing, boxing, kicking, jumping, falling, crawling through fences, and climbing over fences, improving the recognition accuracy of complex behaviors. By using causal dilated convolution to expand the temporal receptive field through the TCN network, the model of this embodiment can capture dependencies spanning longer time domains when handling action sequences with longer durations such as crawling through fences, climbing over fences, and falling, avoiding the problem of losing key temporal information due to the limitation of a fixed receptive field.

[0055] In another embodiment, the present invention also provides a campus personnel abnormal behavior recognition device based on an improved PoseConv3D model. The device includes a memory, a processor, and a program stored in the memory and executable by the processor. When the processor executes the program, it implements the aforementioned campus personnel abnormal behavior recognition method. The specific structure of the device includes a video acquisition module, a pose estimation module, a heatmap generation module, a model recognition module, and an alarm output module. The video acquisition module is used to receive campus surveillance video or video clips uploaded by acquisition devices; the pose estimation module is used to detect human targets and locate key points in the video frames; the heatmap generation module is used to generate a three-dimensional pose heatmap based on key point coordinates and confidence levels; the model recognition module is used to call the trained DRT-PoseConv3D model to obtain the behavior category; and the alarm output module is used to output alarm information when the recognition result is abnormal behavior. Each module can be implemented as a software program, hardware circuit, or a combination of both, and data transmission methods include local memory, message queues, databases, or network interfaces.

[0056] In yet another embodiment, the present invention also provides a computer-readable storage medium having program instructions stored thereon. When the program instructions are executed by a processor, they implement the above-described method for identifying abnormal behavior of personnel on campus. The storage medium may be a read-only memory, random access memory, disk, optical disk, solid-state drive, flash memory, or other computer-readable medium capable of storing program instructions.

[0057] The above are merely preferred embodiments of the present invention and do not constitute any limitation on the present invention. Any equivalent substitutions or modifications made by those skilled in the art to the technical solutions and content disclosed in the present invention without departing from the scope of the present invention shall be deemed to have remained within the protection scope of the present invention.

Claims

1. A method for identifying abnormal behavior of personnel on campus based on an improved PoseConv3D model, characterized in that, Includes the following steps: Obtain video data on the behavior of people on campus; Human targets are detected in video frames, and human pose is estimated for the detected human targets to obtain the coordinates of human skeleton key points and their corresponding confidence scores. Based on the coordinates of key points of the human skeleton and their corresponding confidence levels, key point heatmaps and limb heatmaps are generated, and the posture heatmaps corresponding to consecutive video frames are stacked along the time dimension to obtain a three-dimensional posture heatmap volume. The trained DRT-PoseConv3D model is tested using the aforementioned three-dimensional pose heatmap, and the behavior categories of campus personnel are output. The DRT-PoseConv3D model is based on the PoseConv3D model and introduces the DPTAM mechanism, RefConv module and TCN network into the three-dimensional convolutional backbone network. The overall structure of the DRT-PoseConv3D model includes an input convolutional layer, multiple 3D residual layers, the DPTAM mechanism, the RefConv module, the TCN network, a global average pooling layer, and a fully connected layer. After the 3D pose heatmap is input into the DRT-PoseConv3D model, it first passes through the input convolutional layer for initial feature mapping and channel dimensionality upscaling to obtain initial spatiotemporal features. Then, the initial spatiotemporal features are input into the 3D residual layer for stepwise feature extraction. Each 3D residual layer receives the features output from the previous layer. First, the channel dimension and temporal dimension of the features are jointly weighted by attention using the DPTAM mechanism to obtain attention-enhanced features. Then, the attention-enhanced features are processed by 3D convolution using the RefConv module to obtain cross-channel correlation enhancement features. Finally, the cross-channel correlation enhancement features are added element-wise with the residual branch features to obtain the output features of the current 3D residual layer. After passing through the three-dimensional residual layer, high-level spatiotemporal features are obtained. These high-level spatiotemporal features are then input into the TCN network to perform long-term dependency modeling along the time dimension, resulting in temporal enhancement features. These temporal enhancement features are then input into the global average pooling layer to compress the time dimension, yielding a behavior representation vector. Finally, the behavior representation vector is input into the fully connected layer and the Softmax function is used to obtain the predicted probability of each campus personnel behavior category.

2. The campus personnel abnormal behavior identification method based on the improved PoseConv3D model as described in claim 1, characterized in that, The DPTAM mechanism is set in the three-dimensional residual layer and is used to perform joint attention weighting on the channel dimension and temporal dimension of the input features. The DPTAM mechanism includes a dimension rearrangement module, a spatial global pooling module, a temporal enhancement branch, a channel attention branch, a feature cross-fusion module, and a temporal weight generation module. The processing steps of the DPTAM mechanism include: denoting the input features entering the DPTAM mechanism as: ； in, Indicates batch size, Indicates the number of time frames. Indicates the number of channels. and These represent the height and width of the feature map, respectively. First, the input features Dimensional rearrangement is performed to make the temporal and channel dimensions the main dimensions for subsequent attention modeling. Then, the rearranged features are analyzed in terms of spatial dimension. and Perform global average pooling to obtain global descriptors for time and channels: ； in, Indicates the first The sample, the first The first channel in the Global response value over a time frame; One-dimensional convolution is performed on the global descriptor along the time dimension to learn the action change relationship between different time frames and obtain temporal enhancement features. At the same time, the global descriptor is pooled along the time dimension to obtain channel description vectors. The channel description vectors are input into a one-dimensional convolutional layer and a sigmoid activation function to generate global channel attention weights. The global channel attention weights are multiplied element-wise with the temporal enhancement features and added element-wise with the time and channel global descriptors to obtain joint enhancement features. The joint enhancement features are input into a convolutional layer and an activation function to generate temporal importance weights. The temporal importance weights are applied to the original input features to obtain the output features of the DPTAM mechanism.

3. The method for identifying abnormal campus behavior based on the improved PoseConv3D model according to claim 2, characterized in that, The temporal enhancement branch consists of a one-dimensional convolutional layer that performs one-dimensional convolution processing on the global descriptor along the temporal dimension; the channel attention branch includes pooling operations along the temporal dimension, a one-dimensional convolutional layer, and a sigmoid activation function; the temporal weight generation module consists of a one-dimensional convolutional layer and a sigmoid activation function.

4. The method for identifying abnormal campus personnel behavior based on the improved PoseConv3D model according to claim 1, characterized in that, The RefConv module is located in the 3D residual layer and is used to replace the ordinary 3D convolution and establish cross-channel parameter relationships between convolution kernels. The RefConv module first obtains the convolution kernel parameters of the corresponding ordinary 3D convolution layer in the pre-trained PoseConv3D model as the base convolution kernel W. b During training, the basic convolutional kernel remains frozen and does not participate in gradient updates; subsequently, trainable refocusing transformation parameters are set outside the basic convolutional kernel. The refocusing transformation parameters are used to perform cross-channel transformation on the basic convolutional kernel, establishing a correlation between the convolutional kernel parameters corresponding to different input channels and different output channels; the refocusing transformation process is represented as: ； in, This represents the refocusing transformation function. This represents the refocusing convolution kernel obtained by transforming the basic convolution kernel; in order to make the refocusing convolution kernel... With basic convolution kernel The kernel sizes are consistent, and the convolution transformation operation uses padding to keep the kernel size unchanged; when the attention enhancement features output by the DPTAM mechanism are input into the RefConv module, the RefConv module uses the refocusing convolution kernel W. b The attention-enhanced features are subjected to 3D convolution operations to obtain the output features of the RefConv module.

5. The method for identifying abnormal campus behavior based on the improved PoseConv3D model according to claim 4, characterized in that, The refocusing transformation function is specifically implemented using a convolution operation; the basic convolution kernel W is... b As input, the trainable refocusing transformation parameters Perform convolution operations and output refocused convolution kernels .

6. The method for identifying abnormal campus behavior based on the improved PoseConv3D model according to claim 1, characterized in that, The TCN network is set after multiple three-dimensional residual layers and is used to perform long-term dependency modeling on the high-level spatiotemporal features output by the three-dimensional residual layers. The processing steps of the TCN network include: denoting the high-level spatiotemporal features output from the last 3D residual layer as: ； in, Indicates batch size, Indicates the number of channels. Indicates the number of time frames. and These represent the height and width of the feature map, respectively. First, the spatial dimensions of the high-level spatiotemporal features F are aggregated to obtain a temporal feature sequence dominated by the time dimension: ； in, This represents a temporal feature sequence obtained by aggregating high-level spatiotemporal features F through spatial dimensions; The temporal feature sequence is input into the temporal convolution module, which includes causal dilated convolution, batch normalization, ReLU activation function, and Dropout operation. When calculating features at time t, the causal dilated convolution only uses features from the current time and previous historical time steps. The calculation process is as follows: ； in, Indicates the first The convolution output at time step, Indicates the kernel size. Represents the kernel index. Indicates the convolution kernel weights, Indicates the expansion factor. This represents the input features at the current or historical time point; by setting an expansion factor... This expands the effective receptive field of the temporal convolution module in the temporal dimension; The TCN network sets up a residual branch to pass the original temporal feature sequence obtained after spatial dimension aggregation to the output. When the number of channels of the original temporal feature and the temporal enhancement feature are inconsistent, the channels of the original temporal feature are adjusted by convolution. The temporal enhancement feature and the residual branch feature are added element by element to obtain the output feature of the TCN network.

7. The method for identifying abnormal campus behavior based on the improved PoseConv3D model according to claim 1, characterized in that, The keypoint heatmap generation process uses the coordinates of key points on the human skeleton as the center and generates a two-dimensional Gaussian mapping function; for the k-th keypoint, the calculation process of its corresponding two-dimensional keypoint heatmap is as follows: ； ； in, The variance of the two-dimensional Gaussian kernel function; a limb heatmap was also generated through Gaussian mapping. , No. The limbs are located at two key points. and Between; functions Used to calculate points to line segment distance, Key points and Line segments between; superimpose all generated two-dimensional heatmaps along the time dimension to obtain a three-dimensional posture heatmap volume with dimensions of K×T×H×W; the limb heatmap generation process connects adjacent human skeletal key points to form limb line segments according to the human skeletal topology, and generates a limb heatmap based on the distance from any pixel position in the limb heatmap to the limb line segment.

8. A campus personnel abnormal behavior recognition device based on an improved PoseConv3D model, characterized in that, The device includes a memory, a processor, and a program stored in the memory and capable of running on the processor. When the processor executes the program, it implements the campus personnel abnormal behavior recognition method based on the improved PoseConv3D model as described in any one of claims 1 to 7. The device includes a video acquisition module, a pose estimation module, a heatmap generation module, a model recognition module, and an alarm output module. The video acquisition module is used to receive campus surveillance videos or video clips uploaded by acquisition devices. The pose estimation module is used to detect human targets and locate key points in video frames. The heatmap generation module is used to generate a three-dimensional pose heatmap based on key point coordinates and confidence levels. The model recognition module is used to call the trained DRT-PoseConv3D model to obtain the behavior category. The alarm output module is used to output alarm information when the recognition result is abnormal behavior.

9. A computer-readable storage medium, characterized in that, The device stores a program that, when executed by a processor, implements the campus personnel abnormal behavior identification method based on the improved PoseConv3D model as described in any one of claims 1 to 7; the storage medium includes a read-only memory, a random access memory, a disk, an optical disk, a solid-state drive, or flash memory.

Images