Abnormal turn behavior detection method based on feature fusion and weighted maximum margin neural network

Abstract

An abnormal fence-climbing behavior detection method based on feature fusion and a weighted maximum margin neural network includes: 1) constructing normal and abnormal behavior datasets for training and testing the model; 2) designing an improved pose feature fusion strategy to enhance the expressive power and discriminative power of the features; 3) inputting the feature-fused dataset into the neural network to capture nonlinear features; 4) introducing a maximum margin loss function to improve the model's generalization ability, and constructing a maximum margin loss function to train the model; 5) introducing an imbalance factor to weight the loss values ​​of different categories of samples, balancing the influence of categories during training; 6) predicting whether the behavior is abnormal, and classifying and discriminating new behavior data based on the trained model. This invention can extract more accurate and expressive features and has a strong ability to identify abnormal behaviors.

Description

Technical Field

[0001] This invention relates to a method for detecting abnormal fence-climbing behavior in key areas. It utilizes the fusion of human posture features to extract behavioral information and constructs a Weighted Maximum Margin Neural Network (WMMNN) to address the learning problem of imbalanced abnormal behavior categories. This method is applicable to detecting abnormal fence-climbing behavior in key areas, such as guardrails on urban roads, bridge and tunnel railings, and public places like exhibition halls. It aims to protect the safety of people and property and prevent safety accidents. Background Technology

[0002] In real life, critical areas refer to areas of significant importance for the protection of people, property, and the environment, such as roads, riverbank railings, and exhibition halls. Safety accidents in these areas often have substantial impacts and losses. For example, guardrails on urban roads prevent traffic accidents caused by people climbing over them while vehicles are in motion; railings on bridges and tunnels prevent pedestrians from climbing over or scaling the railings when passing through these structures, thus avoiding potential falls; railings on dams, rivers, and lakes prevent people from crossing them when approaching these bodies of water, thus preventing drowning accidents; and safety barriers in public places such as exhibition halls, museums, and stadiums detect whether people are climbing over railings, scaling barriers, or crossing exhibition areas when entering and exiting these places.

[0003] By deploying an abnormal fence-climbing behavior detection system, abnormal behavior can be identified in advance, allowing for timely warnings and preventing accidents. This system improves the safety of personnel and property. Therefore, abnormal fence-climbing behavior detection in critical areas is essential. Traditional security measures in critical areas rely primarily on manual patrols and inspections, which are inefficient and fail to cover all areas. Deploying an abnormal fence-climbing behavior detection system enables automated monitoring and management of critical areas, improving efficiency, reducing manpower burden, and ensuring safety.

[0004] Currently, traditional methods for detecting abnormal behavior mainly employ two stages: feature extraction and classifier behavior recognition. The feature extraction stage aims to extract discriminative and expressive features from the raw data for subsequent behavior classification tasks. The classifier behavior recognition stage trains a classifier model to match and classify the extracted features against predefined behavior categories, thereby achieving the detection and recognition of abnormal behavior. However, the abnormal behavior recognition problem is characterized by nonlinearity and imbalance, which introduces the following technical limitations to traditional models:

[0005] In terms of nonlinear feature learning: Abnormal behavior often exhibits highly complex nonlinear patterns, which are difficult to capture and describe by traditional linear models. Furthermore, for kernel-based methods, selecting appropriate kernel functions and parameters to capture the potential discriminative information for anomalous behavior discrimination is also a challenging problem.

[0006] Regarding the class imbalance issue: Because data on abnormal behavior is often relatively scarce, far less than data on normal behavior, in actual anomaly detection tasks, there will be situations where the proportion of abnormal data categories is small, i.e., the training data suffers from class imbalance. This class imbalance problem causes the abnormal behavior detection model to tend to capture normal behavior categories, which can easily lead to misjudgment or missed detection of abnormal behavior data.

[0007] Therefore, traditional models have certain limitations in anomaly detection, necessitating the search for new methods and technologies to capture and represent nonlinear behavior patterns and address the issue of data imbalance. Only in this way can the accuracy and robustness of anomaly detection be effectively improved, meeting the needs of reliable identification of anomalies in practical applications. Summary of the Invention

[0008] To overcome the shortcomings of the prior art, this invention proposes an abnormal fence-climbing behavior detection method based on feature fusion and weighted maximum interval neural network, which is suitable for application scenarios that identify abnormal fence-climbing behavior in key areas.

[0009] The abnormal fence-climbing behavior detection method based on feature fusion and weighted maximum margin neural network includes the following steps:

[0010] Step 1: Capture human behavior data in key areas using surveillance cameras, and extract key points for human pose estimation from key frames.

[0011] (1.1) Human pose feature estimation. Video stream data is read using the OpenCV computer vision module, and keyframe image data is extracted.

[0012] (1.2) For the k-th keyframe image data, the position information of 8 human skeleton key points is extracted by analyzing the pose estimation module, such as... Figure 3 As shown. Specific features are described below: nose (N), left shoulder (LS), right shoulder (RS), left hip (LH), right hip (RH), left knee (LK), right knee (RK), left ankle (LA), and right ankle (RA). Additionally, record... or This represents the horizontal or vertical coordinates of the left shoulder in the k-th keyframe image. The superscript k indicates the k-th keyframe image, and the subscripts h and v represent the horizontal or vertical coordinates, respectively.

[0013] Step Two: To improve the detection performance of abnormal fence-climbing behavior, this invention will further fuse pose features of the key point location information of the skeletal contour of the image. For the k-th keyframe image, the feature fusion process is described as follows:

[0014] (2.1) Construct the shoulder-to-ankle distance feature SA (k) First, calculate the shoulders. and The absolute difference in vertical height; then, calculate the difference in vertical height between the two ankles. and The absolute difference in vertical height. Based on this, calculate their absolute vertical height value SA. (k) :

[0015]

[0016] (2.2) Constructing small distance features from the left and right hips to the ankles First, obtain the vertical coordinates of the key points of the two buttocks. and Then, obtain the vertical coordinates of the key points on the left and right ankles. and Finally, calculate the smaller vertical distance from the hip to the ankle on both the left and right sides.

[0017]

[0018] (2.3) Constructing small distance features from the nose to the left and right ankles First, obtain the vertical coordinates of the key points of the nose. Then, obtain the vertical coordinates of the key points on the left and right ankles. and Finally, calculate the smaller of the vertical distances from the nose to the left and right ankles.

[0019]

[0020] (2.4) Construct the feature of the ratio of the distance from the nose to the left and right ankles. First, obtain the vertical coordinates of the nose keypoint in the current frame. Then, obtain the vertical coordinates of the key points on the left and right ankles. and Next, calculate the larger of the vertical distances from the nose to the left and right ankles.

[0021]

[0022] Finally, calculate the ratio of the larger to the smaller vertical distance from the nose to the left and right ankles.

[0023]

[0024] (2.5) Construct the angle Ang between the left and right ankles and the midpoint of the hip. (k) Obtain the vertical coordinates of the key points of the two buttocks in the current frame. and And calculate the vertical coordinates of the key points at the midpoint of the two buttocks.

[0025]

[0026] Then, obtain the horizontal coordinates of the key points of the two buttocks. and And calculate the horizontal coordinates of the key points at the midpoint of the two buttocks.

[0027]

[0028] Next, obtain the horizontal coordinates of key points on the left and right ankles. and Vertical coordinates and Calculate the distances HAl from the left ankle to the midpoint of the hip, from the left ankle to the right ankle, and from the right ankle to the midpoint of the hip. (k) Al r (k) and HAr (k) :

[0029]

[0030] Finally, using the law of cosines, calculate the angle Ang between the left and right ankles and the midpoint of the hip. (k) :

[0031]

[0032] Step 3: Based on the features of pose fusion, construct a dataset for recognizing abnormal human behaviors, which will be used to train the weighted maximum margin neural network model. The specific process is described below:

[0033] (3.1) For the k-th keyframe image, the fusion features from step two are integrated to form a new fused pose feature.

[0034]

[0035] Where k is the k-th keyframe image, SA (k) Features of the distance from shoulder to ankle, The characteristic of a shorter distance from the left and right hips to the ankles. Features representing the shorter distance from the nose to the left and right ankles. The ratio of the distance from the nose to the left and right ankles is a characteristic feature. g (k) It is the angle between the left and right ankles and the midpoint of the hip.

[0036] (3.2) Label sample x according to whether the behavior in the k-th keyframe image is abnormal. k Output label y k Here, -1 represents normal behavior, and +1 represents abnormal page-jumping behavior.

[0037] (3.3) Extract key point features from all keyframe images in the scene, label and output them, and construct a dataset. Normal Behavior Dataset and abnormal behavior dataset Where x k Let y be the feature vector of the sample. k The output is the sample size, where n is the total sample size, n1 is the sample size of normal behavior, n2 is the sample size of abnormal behavior, and k represents the kth keyframe image.

[0038] Step Four: Construct a Weighted Maximum Margin Neural Network (WMMNN) to address the class imbalance problem of abnormal behavior. Since the abnormal behavior recognition problem is non-linear, traditional linear models are inadequate. Therefore, this invention uses a neural network to capture the non-linear features in the data and then introduces a maximum margin loss function to improve the model's generalization ability. Furthermore, there is class imbalance in abnormal behavior, meaning the number of abnormal behavior samples is much smaller than the number of normal behavior samples. Therefore, this invention introduces an imbalance factor to weight the loss values ​​of samples from different classes. For example... Figure 4 As shown, the specific network structure is described below:

[0039] (4.1) Constructing the network nonlinear mapping of the WMMNN model. To capture data nonlinearity, this invention will use a neural network to implement feature mapping. The network structure includes an input layer, a hidden layer, and an output layer.

[0040] The input layer is designed with 5 nodes, and the sample features are x. k It is a 5-dimensional vector.

[0041] The hidden layers are designed as 3 fully connected layers, each with 10 nodes, and the ReLU activation function is chosen between each layer. For the h-th... i Hidden layer, forward propagation function is

[0042]

[0043] in, For the hth i The output of layer -1 is the h-th layer.i Input to the hidden layer; For the hth i The weights of neurons in the hidden layer; i is the input feature index of the hidden layer; j is the index of the j-th neuron in the hidden layer; f(·) is the activation function, and the ReLU function f(z) = max(0, z) is selected in this invention.

[0044] The output layer is designed with two nodes, representing the prediction confidence outputs for normal and abnormal behaviors, respectively. The larger the output of a node, the greater the probability that it is predicted to be that behavior.

[0045] (4.2) Calculate the imbalance factor as the class weights. Let q be the within-class weights of the minority and majority classes. s , representing the intra-class weight in class s, where s=1 represents normal behavior type and s=2 represents abnormal behavior type. q s The specific calculations are as follows:

[0046]

[0047] Where n is the total number of samples, n cls n is the number of categories. s Let be the number of samples in class s.

[0048] (4.3) Construct the weighted maximum margin loss function. For sample feature x k and the actual output y k The output layer nodes of the WMMNN model are for predicting normal behavior. and abnormal behavior prediction The maximum margin loss function is then constructed as follows:

[0049]

[0050] The loss function is explained as follows: when y k When = -1, that is, sample x k If the true label is normal behavior, then the normal behavior node in the WMMNN model... The output is at least higher than the abnormal behavior node. Output a value greater than 1; otherwise, a non-zero loss value will be generated. k When =+1, that is, sample x k If the true label is abnormal behavior, then the model's... Output at least more than Output a value greater than 1; otherwise, a non-zero loss value will be generated.

[0051] (4.4) Train the WMMNN model using the SGD stochastic gradient descent method, with the optimization objective being:

[0052]

[0053] Where W represents the parameters of the WMMNN model, and this invention uses the PyTorch deep learning framework to train the WMMNN model.

[0054] Step 5: Abnormal Fence-Climbing Behavior Stage (Model Prediction). After the WMMNN model is trained, the optimal model parameters are obtained, and an abnormal fence-climbing behavior prediction decision function is constructed. For a given video image to be detected, the video stream is first captured in real time using OpenCV, and then the MediaPipe open-source module is used to extract key point information of human posture. Then, the key point information from Step 2 is fused to obtain sample features x. Finally, the WMMNN model is used to predict sample x, obtaining the output of normal behavior nodes. Output of abnormal behavior nodes The decision function for behavior detection is constructed as follows:

[0055]

[0056] Here, sgn(·) is the sign function, which outputs 1 when the value is greater than 0 and -1 when the value is less than 0. Represents normal behavior node prediction Prediction of abnormal behavior nodes The difference. When At this point, the confidence level for normal behavior is higher than that for abnormal behavior, therefore the predicted label for sample x is -1; when At this point, the confidence level of abnormal behavior is greater than that of normal behavior, so the label prediction for sample x is +1.

[0057] The advantages of this invention are:

[0058] To overcome the problems of information redundancy in keypoint features and the inability of raw keypoint location information to accurately represent the fence-climbing behavior, an improved pose feature fusion strategy is proposed. In this strategy, the keypoint location information of the skeletal contour of the image is further processed to extract more accurate and expressive features.

[0059] Because the problem of abnormal behavior recognition is non-linear, traditional linear models are inadequate. Therefore, this invention employs neural networks to capture the non-linear features in the data and introduces a maximum margin loss function to improve the model's generalization ability. Specifically, the non-linear features of the feature-fused dataset are input into the neural network, and the model is trained by constructing a maximum margin loss function.

[0060] The categories of abnormal behavior exhibit imbalance, meaning that data on abnormal behavior is far less than samples of normal behavior. To address this issue, this invention introduces an imbalance factor, which balances the dataset by weighting the loss values ​​of samples from different categories, thereby improving the ability to identify abnormal behavior. Attached Figure Description

[0061] Figure 1 This is a flowchart of the method of the present invention.

[0062] Figure 2 This is a diagram of the overall system framework of the present invention.

[0063] Figure 3 This is a schematic diagram of the skeletal structure characteristic of the present invention.

[0064] Figure 4 This is a network structure diagram of the present invention.

[0065] Figure 5 This is a visual prediction of the method of the present invention. Detailed Implementation

[0066] The preferred embodiments of the present invention are given below with reference to the figures to illustrate the technical solution of the present invention in detail.

[0067] A method for detecting abnormal fence-climbing behavior in key regions based on pose feature fusion and imbalanced learning is described in detail below:

[0068] Step 1: Capture human behavior data in key areas using surveillance cameras, and extract key points for human pose estimation from key frames.

[0069] (1.1) Human pose feature estimation. Video stream data is read using the OpenCV computer vision module, and keyframe image data is extracted.

[0070] (1.2) For the k-th keyframe image data, the position information of 8 human skeleton key points is extracted by analyzing the pose estimation module, such as... Figure 3 As shown. Specific features are described below: nose (N), left shoulder (LS), right shoulder (RS), left hip (LH), right hip (RH), left knee (LK), right knee (RK), left ankle (LA), and right ankle (RA). Additionally, record... or This represents the horizontal or vertical coordinates of the left shoulder in the k-th keyframe image. The superscript k indicates the k-th keyframe image, and the subscripts h and υ represent the horizontal or vertical coordinates, respectively.

[0071] Step Two: To improve the detection performance of abnormal fence-climbing behavior, this invention will further fuse pose features of the key point location information of the skeletal contour of the image. For the k-th keyframe image, the feature fusion process is described as follows:

[0072] (2.1) Construct the shoulder-to-ankle distance feature SA (k) First, calculate the shoulders. and The absolute difference in vertical height; then, calculate the difference in vertical height between the two ankles. and The absolute difference in vertical height. Based on this, calculate their absolute vertical height value SA. (k) :

[0073]

[0074] (2.2) Constructing small distance features from the left and right hips to the ankles First, obtain the vertical coordinates of the key points of the two buttocks. and Then, obtain the vertical coordinates of the key points on the left and right ankles. and Finally, calculate the smaller vertical distance from the hip to the ankle on both the left and right sides.

[0075]

[0076] (2.3) Constructing small distance features from the nose to the left and right ankles First, obtain the vertical coordinates of the key points of the nose. Then, obtain the vertical coordinates of the key points on the left and right ankles. and Finally, calculate the smaller of the vertical distances from the nose to the left and right ankles.

[0077]

[0078] (2.4) Construct the feature of the ratio of the distance from the nose to the left and right ankles. First, obtain the vertical coordinates of the nose keypoint in the current frame. Then, obtain the vertical coordinates of the key points on the left and right ankles. and Next, calculate the larger of the vertical distances from the nose to the left and right ankles.

[0079]

[0080] Finally, calculate the ratio of the larger to the smaller vertical distance from the nose to the left and right ankles.

[0081]

[0082] (2.5) Construct the angle Ang between the left and right ankles and the midpoint of the hip. (k) Obtain the vertical coordinates of the key points of the two buttocks in the current frame. and And calculate the vertical coordinates of the key points at the midpoint of the two buttocks.

[0083]

[0084] Then, obtain the horizontal coordinates of the key points of the two buttocks. and And calculate the horizontal coordinates of the key points at the midpoint of the two buttocks.

[0085]

[0086] Next, obtain the horizontal coordinates of key points on the left and right ankles. and Vertical coordinates and Calculate the distances HAl from the left ankle to the midpoint of the hip, from the left ankle to the right ankle, and from the right ankle to the midpoint of the hip. (k) Al r (k) and HAr (k) :

[0087]

[0088] Finally, using the law of cosines, calculate the angle Ang between the left and right ankles and the midpoint of the hip. (k) :

[0089]

[0090] Step 3: Based on the features of pose fusion, construct a dataset for recognizing abnormal human behaviors, which will be used to train the weighted maximum margin neural network model. The specific process is described below:

[0091] (3.1) For the k-th keyframe image, the fusion features from step two are integrated to form a new fused pose feature.

[0092]

[0093] Where k is the kth keyframe image, SA (k) Features of the distance from shoulder to ankle, The characteristic of a shorter distance from the left and right hips to the ankles. Features representing the shorter distance from the nose to the left and right ankles. The ratio of the distance from the nose to the left and right ankles is a characteristic feature, Ang (k) It is the angle between the left and right ankles and the midpoint of the hip.

[0094] (3.2) Label sample x according to whether the behavior in the k-th keyframe image is abnormal. k Output label y k Here, -1 represents normal behavior, and +1 represents abnormal page-jumping behavior.

[0095] (3.3) Extract key point features from all keyframe images in the scene, label and output them, and construct a dataset. Normal Behavior Dataset and abnormal behavior dataset Where x k Let y be the feature vector of the sample. k The output is the sample size, where n is the total sample size, n1 is the sample size of normal behavior, n2 is the sample size of abnormal behavior, and k represents the kth keyframe image.

[0096] Step Four: Construct a Weighted Maximum Margin Neural Network (WMMNN) to address the class imbalance problem of abnormal behavior. Since the abnormal behavior recognition problem is non-linear, traditional linear models are inadequate. Therefore, this invention uses a neural network to capture the non-linear features in the data and then introduces a maximum margin loss function to improve the model's generalization ability. Furthermore, there is class imbalance in abnormal behavior, meaning the number of abnormal behavior samples is much smaller than the number of normal behavior samples. Therefore, this invention introduces an imbalance factor to weight the loss values ​​of samples from different classes. For example... Figure 4 As shown, the specific network structure is described below:

[0097] (4.1) Constructing the network nonlinear mapping of the WMMNN model. To capture data nonlinearity, this invention will use a neural network to implement feature mapping. The network structure includes an input layer, a hidden layer, and an output layer.

[0098] The input layer is designed with 5 nodes, and the sample features are x. k It is a 5-dimensional vector.

[0099] The hidden layers are designed as 3 fully connected layers, each with 10 nodes, and the ReLU activation function is chosen between each layer. For the h-th... i Hidden layer, forward propagation function is

[0100]

[0101] in, For the hth i The output of layer -1 is the h-th layer. i Input to the hidden layer; For the hth i The weights of neurons in the hidden layer; i is the input feature index of the hidden layer; j is the index of the j-th neuron in the hidden layer; f(·) is the activation function, and the ReLU function f(z) = max(0, z) is selected in this invention.

[0102] The output layer is designed with two nodes, representing the prediction confidence outputs for normal and abnormal behaviors, respectively. The larger the output of a node, the greater the probability that it is predicted to be that behavior.

[0103] (4.2) Calculate the imbalance factor as the class weights. Let q be the within-class weights of the minority and majority classes. s , representing the intra-class weight in class s, where s=1 represents normal behavior type and s=2 represents abnormal behavior type. q s The specific calculations are as follows:

[0104]

[0105] Where n is the total number of samples, n cls n is the number of categories. s Let be the number of samples in class s.

[0106] (4.3) Construct the weighted maximum margin loss function. For sample feature x k and the actual output y k The output layer nodes of the WMMNN model are for predicting normal behavior. and abnormal behavior prediction The maximum margin loss function is then constructed as follows:

[0107]

[0108] The loss function is explained as follows: when y k When = -1, that is, sample x k If the true label is normal behavior, then the normal behavior node in the WMMNN model... The output is at least higher than the abnormal behavior node. Output a value greater than 1; otherwise, a non-zero loss value will be generated. k When =+1, that is, sample x k If the true label is abnormal behavior, then the model's... Output at least more than Output a value greater than 1; otherwise, a non-zero loss value will be generated.

[0109] (4.4) Train the WMMNN model using the SGD stochastic gradient descent method, with the optimization objective being:

[0110]

[0111] Where W represents the parameters of the WMMNN model, and this invention uses the PyTorch deep learning framework to train the WMMNN model.

[0112] Step 5: Abnormal Fence-Climbing Behavior Stage (Model Prediction). After the WMMNN model is trained, the optimal model parameters are obtained, and an abnormal fence-climbing behavior prediction decision function is constructed. For a given video image to be detected, the video stream is first captured in real time using OpenCV, and then the MediaPipe open-source module is used to extract key point information of human posture. Then, the key point information from Step 2 is fused to obtain sample features x. Finally, the WMMNN model is used to predict sample x, obtaining the output of normal behavior nodes. Output of abnormal behavior nodes The decision function for behavior detection is constructed as follows:

[0113]

[0114] Here, sgn(·) is the sign function, which outputs 1 when the value is greater than 0 and -1 when the value is less than 0. Represents normal behavior node prediction Prediction of abnormal behavior nodes The difference. When At this point, the confidence level for normal behavior is higher than that for abnormal behavior, therefore the predicted label for sample x is -1; when At this point, the confidence level of abnormal behavior is greater than that of normal behavior, so the label prediction for sample x is +1.

[0115] The dataset used in this invention consists of keyframe image data collected from key areas for abnormal fence-climbing monitoring. The output of the samples is a subjective calibration, where -1 represents normal behavior and +1 represents abnormal fence-climbing behavior. Since this recognition task is an imbalanced class problem, to verify the effectiveness of the proposed WMMNN model, this invention samples the dataset to obtain three imbalanced datasets with imbalance ratios of 0.2, 0.25, and 0.3. Furthermore, this invention uses the geometric mean G. mean As a metric for evaluating the imbalanced classification performance of a model.

[0116]

[0117] Where TP represents the number of samples correctly predicted as fence-climbing behavior, FP represents the number of samples correctly predicted as normal behavior, TN represents the number of samples correctly predicted as normal behavior, and FN represents the number of samples correctly predicted as fence-climbing behavior. To avoid bias caused by uneven dataset distribution, this invention employs a ten-fold cross-validation method and takes the average as the result. In this method, the dataset is divided into ten equal subsets. Each time, one subset is selected as the test set, and the remaining nine subsets are used as the training set. Then, the experiment is repeated ten times, each time selecting a different test set, and the corresponding evaluation metric is calculated. Finally, the results of these ten experiments are averaged to obtain the final result.

[0118] To verify the effectiveness of the proposed method, this invention compared and verified the cross-validation performance of ADASYN and SMOTE before feature fusion, ADASYN and SMOTE after feature fusion, and the proposed WMMNN model. The results are shown in Table 1. The results show that, except when the class ratio is 0.25, where the SMOTE algorithm has a slightly higher accuracy than the WMMNN method, the WMMNN method outperforms both the ADASYN and SMOTE algorithms in accuracy for the other two class ratios. Therefore, the above experiments demonstrate that the WMMNN method has certain advantages in handling class imbalance. Furthermore, five keyframe images were selected from the prediction set, and the WMMNN method was used for prediction. The results are shown in Table 1. Figure 5 As shown in the figure. The results demonstrate that the WMMNN method proposed in this invention can accurately predict behavior.

[0119] Table 1 Performance comparison between the method of the present invention and existing methods

[0120]

[0121] The above are preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions conceived by those skilled in the art within the scope of the technology disclosed in the present invention without inventive effort are included within the scope of protection of the present invention. The scope of protection of the present invention is determined by the scope defined in the claims.

Claims

1. An abnormal fence-climbing behavior detection method based on feature fusion and weighted maximum margin neural network, comprising the following steps: Step 1: Capture human behavior data in key areas using surveillance cameras, and extract key points for human pose estimation from key frames; Step 2: To improve the detection performance of abnormal fence-climbing behavior, further pose feature fusion will be performed on the key point location information of the skeletal contour of the image; for the k-th keyframe image, the feature fusion process is described as follows: (2.1) Construct the shoulder-to-ankle distance feature SA (k) First, calculate the shoulders. and The absolute difference in vertical height; then, calculate the difference in vertical height between the two ankles. and The absolute difference in vertical height; based on this, calculate their absolute vertical height value SA. (k) : (2.2) Constructing small distance features from the left and right hips to the ankles First, obtain the vertical coordinates of the key points of the two buttocks. and Then, obtain the vertical coordinates of the key points on the left and right ankles. and Finally, calculate the smaller vertical distance from the hip to the ankle on both the left and right sides. (2.3) Constructing small distance features from the nose to the left and right ankles First, obtain the vertical coordinates of the key points of the nose. Then, obtain the vertical coordinates of the key points on the left and right ankles. and Finally, calculate the smaller of the vertical distances from the nose to the left and right ankles. (2.4) Construct the feature of the ratio of the distance from the nose to the left and right ankles. First, obtain the vertical coordinates of the nose keypoint in the current frame. Then, obtain the vertical coordinates of the key points on the left and right ankles. and Next, calculate the larger of the vertical distances from the nose to the left and right ankles. Finally, calculate the ratio of the larger to the smaller vertical distance from the nose to the left and right ankles. (2.5) Construct the angle Ang between the left and right ankles and the midpoint of the hip. (k) Obtain the vertical coordinates of the key points of the two buttocks in the current frame. and And calculate the vertical coordinates of the key points at the midpoint of the two buttocks. Then, obtain the horizontal coordinates of the key points of the two buttocks. and And calculate the horizontal coordinates of the key points at the midpoint of the two buttocks. Next, obtain the horizontal coordinates of key points on the left and right ankles. and Vertical coordinates and Calculate the distances HAl from the left ankle to the midpoint of the hip, from the left ankle to the right ankle, and from the right ankle to the midpoint of the hip. (k) Alr (k) and HAr (k) : Finally, using the law of cosines, calculate the angle Ang between the left and right ankles and the midpoint of the hip. (k) : Step 3: Based on the features of pose fusion, construct a dataset for recognizing abnormal human behavior, which will be used to train the weighted maximum margin neural network model; Step 4: Construct a Weighted Maximum Margin Neural Network (WMMNN) to address the class imbalance problem of abnormal behavior; the specific network structure of the WMMNN is as follows: (4.1) Construct the network nonlinear mapping of the WMMNN model; In order to capture the nonlinearity of the data, a neural network is used to implement feature mapping. The network structure includes an input layer, a hidden layer, and an output layer. The input layer is designed with 5 nodes, and the sample features are x. k It is a 5-dimensional vector; The hidden layer is designed as a 3-layer fully connected layer, with 10 nodes in each layer. The activation function between each layer is the ReLU function; for the h-th layer... i Hidden layer, forward propagation function is in, For the hth i The output of layer -1 is the h-th layer. i Input to the hidden layer; For the hth i The weights of neurons in the hidden layer; i is the input feature index of the hidden layer; j is the index of the j-th neuron in the hidden layer; f(·) is the activation function, and the ReLU function f(z) = max(0, z) is selected; The output layer is designed with two nodes, representing the prediction confidence outputs for normal and abnormal behaviors, respectively; the larger the output of a node, the greater the probability that it is predicted to be that behavior. (4.2) Calculate the imbalance factor as the class weights; let the intra-class weights of the label minority class and the majority class be denoted as q. s , represents the intra-class weight in class s, where s=1 represents normal behavior type and s=2 represents abnormal behavior type; q s The specific calculations are as follows: Where n is the total number of samples, n cls n is the number of categories. s Let be the number of samples in class s; (4.3) Construct the weighted maximum margin loss function; for sample feature x k and the actual output y k The output layer nodes of the WMMNN model are for predicting normal behavior. and abnormal behavior prediction The maximum margin loss function is then constructed as follows: The loss function is explained as follows: when y k When = -1, that is, sample x k If the true label is normal behavior, then the normal behavior node in the WMMNN model... The output is at least higher than the abnormal behavior node. Output a value greater than 1; otherwise, a non-zero loss value will be generated. k When =+1, that is, sample x k If the true label is abnormal behavior, then the model's... Output at least more than Output a value greater than 1; otherwise, a non-zero loss value will be generated. (4.4) Train the WMMNN model using the SGD stochastic gradient descent method, with the optimization objective being: Where W represents the parameters of the WMMNN model, and this invention uses the PyTorch deep learning framework to train the WMMNN model. Step 5: After the WMMNN model is trained, the optimal model parameters are obtained, and the abnormal flipping behavior prediction decision function is constructed.

2. The abnormal fence-climbing behavior detection method based on feature fusion and weighted maximum margin neural network as described in claim 1, characterized in that: Step one specifically includes: (1.1) Human pose feature estimation; The video stream data was read using the OpenCV computer vision module, and keyframe image data was extracted; (1.2) For the k-th keyframe image data, the position information of 8 human skeleton key points is extracted by analyzing the pose estimation module; the specific features are described as follows: nose N, left shoulder LS, right shoulder RS, left hip LH, right hip RH, left knee LK, right knee RK, left ankle LA, and right ankle RA; in addition, let... or Let h and v be the horizontal or vertical coordinates of the left shoulder of the k-th keyframe image; where the superscript k represents the k-th keyframe image, and the subscripts h and v represent the horizontal or vertical directions, respectively.

3. The abnormal fence-climbing behavior detection method based on feature fusion and weighted maximum margin neural network as described in claim 1, characterized in that: Step three specifically includes: (3.1) For the k-th keyframe image, the fusion features from step two are integrated to form a new fused pose feature. Where k represents the k-th keyframe image, SA (k) Features of the distance from shoulder to ankle, The characteristic of a shorter distance from the left and right hips to the ankles. Features representing the shorter distance from the nose to the left and right ankles. The ratio of the distance from the nose to the left and right ankles is a characteristic feature, Ang (k) The angle between the left and right ankles and the midpoint of the hip; (3.2) Label sample x according to whether the behavior in the k-th keyframe image is abnormal. k Output label y k Where -1 represents normal behavior and +1 represents abnormal folding behavior; (3.3) Extract key point features from all keyframe images in the scene, label and output them, and construct a dataset. Normal Behavior Dataset and abnormal behavior dataset Where x k Let y be the feature vector of the sample. k The output is the sample size, where n is the total sample size, n1 is the sample size of normal behavior, n2 is the sample size of abnormal behavior, and k represents the kth keyframe image.

Images