Pulling and pushing behavior identification method based on improved TSM video classification algorithm

Abstract

A pulling and pushing behavior identification method based on an improved TSM video classification algorithm. The method comprises the following steps: S1, generating pulling and pushing detection datasets, performing classification and marking, and then dividing the datasets; S2, improving a TSM network; S3, training the improved TSM network to obtain an algorithm model weight, and obtaining a pulling and pushing behavior detection model; S4, collecting data of images, and acquiring data of each image frame; S5, inputting the data of each image frame obtained in step S4 into the pulling and pushing behavior detection model for inference, and obtaining an inference result; and S6, analyzing the inference result, determining whether a pulling and pushing behavior has occurred, and if so, storing the pulling and pushing behavior, and determining whether an alarm needs to be raised. The identification method uses an improved TSM video classification algorithm to determine a pulling and pushing behavior in a surveillance video, reducing the dependence on abnormal behavior samples and improving the accuracy of the determination of abnormal behaviors.

Description

Pulling behavior recognition method based on improved TSM video classification algorithm

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on September 28, 2023, with application number 202311272743X and invention name “Method for identifying pulling behavior based on improved TSM video classification algorithm”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application belongs to the field of novel artificial intelligence, and more specifically, relates to a method for identifying pulling behavior based on an improved TSM video classification algorithm. Background Art

[0003] Currently, traditional deep learning object detection algorithms are combined with target tracking algorithms to determine whether a person is exhibiting abnormal behavior. The effectiveness of this approach is highly dependent on the accuracy of the target detection algorithm used to detect people. If the target detection algorithm performs poorly in complex scenes, occlusion, and lighting changes, the accuracy of abnormal behavior detection will be affected.

[0004] Traditional methods struggle to accurately identify abnormal behavior among multiple people. This is because they primarily focus on the movements and actions of a single target. In multi-person scenarios, the interactions and coordinated movements of different individuals complicate the identification process. Furthermore, traditional target detection and tracking algorithms may perform poorly in complex and crowded scenes. They also struggle to handle complex abnormal behaviors. Some abnormal behaviors may not be based solely on the movements or actions of a single person but may involve collaboration among multiple individuals or complex action sequences. Traditional methods may struggle to effectively capture and identify this type of abnormal behavior.

[0005] Due to the above reasons, this method is highly dependent on the accuracy of the target detection model, and it is difficult to judge complex abnormal behaviors, which reduces the accuracy of abnormal behavior recognition and leads to problems of misjudgment and omission of abnormal behaviors.

[0006] Existing methods for abnormal behavior recognition based on skeleton sequences are common in the fields of human motion analysis and behavior recognition. This method may produce changes at different angles and postures of the human body, thereby affecting the model's representation ability and recognition performance. This may make it difficult for the model to accurately capture and understand the changing human body movements in complex real-world scenarios. At the same time, this method may have poor generalization capabilities in new environments or unseen scenes. Because these methods mainly learn specific skeleton patterns from training data, recognition performance may degrade when encountering environmental factors that differ from the training data. In addition, this method usually only focuses on the trajectory of the human body's movements, while ignoring the surrounding environment and contextual information. This may make it difficult to understand and distinguish between normal and abnormal behaviors in some cases, because the environment and context may affect the interpretation and judgment of behavior.

[0007] A Chinese patent document (CN115346266A) discloses a skeleton-based method for extracting key nodes of human behavior in surveillance videos. This application has a certain degree of versatility in the field of behavior detection and can be used as a module of a behavior detection framework. The patent uses fighting and falling behaviors as illustrative cases to demonstrate a method for extracting key nodes of behavior. In response to the irrationality of the existing behavior detection framework extracting video frames at a fixed frame rate, this application proposes to dynamically adjust the position of the extracted video frames and the extraction frame rate based on the motion information of the human skeleton key points, and extract more video frames for the behavior node segments with faster movement. Adding a skeleton-based human behavior key node extraction module to the existing behavior detection framework can more accurately determine the spatiotemporal boundaries of specific behaviors and achieve better behavior detection results. For this method of abnormal behavior recognition, the target detection algorithm in the field of deep learning computer vision is used. As can be seen from the process, this method uses frame extraction technology for video processing, performs target detection on the extracted image frames, and judges fighting behavior based on the results obtained by target detection. The main steps of this method use the Harris3D spatiotemporal interest point detector to detect areas in the image that have obvious changes in both spatial and temporal domains, count the number of Harris3D spatiotemporal interest points in each candidate key frame in a video sequence, and take the top 50% candidate frames with the largest number of interest points as key frames.

[0008] For these reasons, abnormal behavior recognition methods based on skeleton sequences may be limited in some cases by posture variations, difficulty in annotation, and poor generalization. These issues lead to poor generalization of the model, causing such algorithms to produce false positives and missed positives in a variety of scenarios.

[0009] Traditional TSM detection algorithms embed a TSM module within a ResNet50 network to determine whether a person in a video exhibits abnormal behavior. This method uses a ResNet50-based network as its foundation, adding the TSM module after the convolutional layers to improve image classification accuracy. However, this method cannot accurately capture the motion characteristics of objects in the video and the relationships between context. Therefore, it cannot quickly and accurately determine whether abnormal behavior occurs in a video, and can result in false positives and missed detections in a variety of scenarios. This method also suffers from low model generalization and is not applicable to complex detection scenarios.

[0010] Therefore, it is necessary to propose a pulling behavior recognition method based on an improved TSM video classification algorithm. The improved TSM video classification algorithm can accurately identify pulling behaviors in all complex scenes without misidentification or missed identification, thereby improving the recognition accuracy.

[0011] Summary of the Invention

[0012] In view of this, the purpose of this application is to provide a method for identifying pulling behavior based on an improved TSM video classification algorithm. This method uses a video classification algorithm to detect pulling behavior, reducing the serious false positives of traditional abnormal behavior detection. By combining multiple technologies, it solves the problem of difficult detection and judgment of pulling behavior in complex scenarios. The technical solution is as follows:

[0013] The pulling behavior recognition method based on the improved TSM video classification algorithm specifically includes the following steps:

[0014] S1: Generate a pull detection dataset, classify and label the content of the dataset, and then divide it into a training set and a validation set;

[0015] S2: Improve the TSM network by adding a channel-level motion feature encoding module (CME module) after obtaining the feature map in the TSM network, and adding an interactive aggregation update module (IAU-Net module) after the second two-dimensional convolution module;

[0016] S3: Input the pulling detection dataset generated in step S1 into the improved TSM algorithm in step S2 to obtain the algorithm model weights trained by the improved TSM and obtain a pulling behavior detection model;

[0017] S4: Image data acquisition, obtaining image data of each frame;

[0018] S5: Inputting each frame of image data obtained in step S4 into the pulling behavior detection model trained in step S3 for inference to obtain an inference result;

[0019] S6: Analyze the inference result in step S5 to determine whether pulling behavior occurs. If so, save the pulling behavior and determine whether an alarm is required.

[0020] By adopting the above technical solution, in order to address the shortcomings of traditional deep learning algorithms in detection, a channel-level motion feature encoding module and an interactive fusion and update IAU module are added to the TSM video classification model to detect and judge pulling behavior. The TSM video classification algorithm is used to detect pulling behavior. Based on TSM, this application adds a channel-level motion feature encoding module and introduces an interactive fusion and update IAU module, avoiding the problem of traditional deep learning algorithms and skeleton sequence-based abnormal behavior detection algorithms lacking contextual dependencies between different frames and strong dependence on samples and scenes. In the monitoring area, when there are more than two people, continuous frames are extracted from the video stream and input into the improved TSM model to detect and judge pulling behavior. If pulling behavior occurs, an early warning of the pulling behavior is output; that is, a video classification model is adopted in combination with a channel-level motion feature encoding module and an interactive fusion update IAU module, so that the pulling behavior detection model can better obtain the contextual relationship of the target in different frames, so that the pulling behavior detection model can effectively detect and judge the pulling behavior between people, improve the accuracy of model detection, and solve the problems of traditional abnormal behavior recognition methods that are difficult to detect abnormal behavior in multiple people and diverse scenarios, and have high false alarm and missed detection rates.

[0021] Preferably, the pulling detection dataset in step S1 is in video format. Pulling and non-pulling behaviors in different scenarios are simulated and recorded to generate the pulling dataset. The videos are then labeled differently based on their content, with videos showing pulling behaviors being labeled as one category and videos without pulling behaviors being labeled as another category. The video dataset showing pulling behaviors is labeled as pulling behaviors, while the video dataset without pulling behaviors is labeled as normal behaviors.

[0022] Preferably, a channel-level motion feature encoding module is added in step S2 to extract motion features and excite motion channels. The specific steps of extracting motion features by the channel-level motion feature encoding module are as follows:

[0023] S21: First, the input video frame is subjected to a 1×1 2D spatial convolution to reduce the feature channel dimension. Assuming the scaling factor is r and the input dimension is [N, T, C, H, W], the convolution results in a feature map with an output dimension of [N, T, C / r, H, W].

[0024] S22: Split the obtained feature map into separate frames along the time dimension T, then perform differential analysis on adjacent frames and use the image feature X of the next frame t+1 Subtract the previous frame Xt Due to the movement of the foreground target in the video, the target position changes between the two frames. Direct subtraction may cause feature confusion. Therefore, in order to eliminate feature confusion, the spatial feature transformation of the latter frame is first performed using a 3x3 2D convolution, and then the obtained transformation feature is subtracted from the image feature of the previous frame to obtain the motion representation between the two adjacent frames. Assume that the motion feature from the t-th frame to the t+1-th frame is M t , then the formula is: M t =Conv 3×3 (X t+1 )-X t , M t ∈R N×C / r×H×W ;

[0025] Among them, X t represents the image features of the t-th frame, X t+1 is the image feature of the t+1th frame, Conv 3×3 (·) represents a 3×3 2D spatial convolution; R represents a real number;

[0026] S23: Concatenate the obtained motion features between adjacent frames to obtain the motion features of the input video, denoted as M, that is: M=Concat(M1, M2, ..., M T ), M∈R N×T×C / r×H×W ;

[0027] Among them, Concat(·) represents the connection of motion features; M1,M2,·…·,M T Both refer to movement characteristics;

[0028] S24: Since the input feature X has undergone dimensionality reduction by the convolution channel, the dimension of the obtained video motion feature M is [N, T, C / r, H, W]. The obtained video motion feature M is input into two parallel branches, namely, the input motion coding branch and the motion channel excitation branch. Thus, the motion feature is obtained according to the motion coding branch and the motion channel excitation branch.

[0029] Preferably, in the motion coding branch in step S24, a 1×1 2D convolution is designed to encode the motion feature M, and the channel dimension is restored from C / r to C, so as to ensure that the feature information is not lost and facilitate fusion with the input feature; assuming that the feature obtained after motion coding is M encode , then: M encode =Conv 1×1 (M), M encode ∈R N×T×C×H×W ;

[0030] Among them, Conv 1×1(·) represents a 1×1 2D convolution operation; the motion coding branch obtains the motion features between adjacent frames after encoding.

[0031] Preferably, because the feature channel contains the motion features of the foreground target and the background features, the attention mechanism is adopted in step S24 to learn the attention weight of the channel of the motion feature obtained after splicing, to stimulate the motion feature channel and suppress the background feature channel; specifically, considering that the channel feature weight is independent of the spatial position, the motion feature is firstly subjected to global average pooling in space to obtain the space-independent motion feature, which is recorded as M pooling , that is, the formula is: M pooling =Pooling(M), M pooling ∈R N×T×C / r×1× 1;

[0032] Where Pooling(·) represents the spatial pooling operation, and the obtained M pooling The dimensions are [N, T, C / r, W, H];

[0033] Then M pooling After convolution and dimensionality increase, and input into the sigmoid activation function, the attention weight matrix is ​​recorded as A, that is, the formula is: A=sigmoid(Conv 1×1 (M pooling )), A∈R N×T×C×1×1 ;

[0034] Where sigmoid(·) represents the sigmoid activation function, Conv 1×1 (·) represents a 1×1 2D convolution operation, which combines the obtained channel attention weight A with the motion coding feature M encode Multiply them together to stimulate the motion feature channel, and add residual connections to keep the original input features from being lost, that is, the formula is: Y = X + A·M encosde , Y∈R N×T×C×H×W ;

[0035] Among them, X is the input feature, and Y is the final output feature of the module; and X and Y have the same dimension, that is, the channel-level motion feature encoding module (CME module) does not change the output dimension of the input feature.

[0036] Preferably, the Interaction-Aggregation-Update (IAU-Net) module, added after the second two-dimensional convolutional module in step S2, comprises an interaction module, an aggregation module, and an update module. The interaction module is used to pull the behavior detection model to process interactions between nodes and their neighboring nodes in the image data; the aggregation module is used to pull the behavior detection model to aggregate information propagated from neighboring nodes to the central node; and the update module is used to update the features or representation of the central node after calculating the aggregated information. The Interaction-Aggregation-Update (IAU) module can better capture the contextual dependencies of the same object across different frames, improving the accuracy of abnormal behavior detection. The Interaction-Aggregation-Update (IAU) module is a concept used to describe the general process or pipeline in graph neural networks (GNNs). This module is particularly suitable for modeling relationships and dependencies between entities in graph-structured data. Graph neural networks have become popular for tasks such as node classification, link prediction, and graph classification. The "interaction-aggregation-update" process enables these networks to effectively extract information from the structure of the graph and perform predictions or classification based on this information. Interaction: During the interaction phase, the model processes the interactions between nodes in the graph and their neighbors. Nodes in the graph can represent entities, objects, users, etc., while edges represent relationships or connections between these entities. In this phase, information flows from neighboring nodes to the central node through the edges of the graph; Aggregation: After the interaction phase, the model aggregates the information propagated from neighboring nodes to the central node. Aggregation can involve various mathematical operations such as summation, averaging, or more complex aggregation functions. The goal is to combine information from different neighbors in a meaningful way to capture relevant contextual information about the central node; Update: Once the aggregated information is calculated, it is used to update the features or representation of the central node. This update step typically involves some form of transformation or neural network operation to integrate the aggregated information with the existing features of the node. The updated features become the new representation of the node, fusing its original features and contextual information from its neighbors.

[0037] The above technical solution circumvents the strong dependence of traditional deep learning algorithms and skeleton sequence algorithms on samples, as well as the low generalization ability of the unimproved TSM detection algorithm model that cannot be applied to complex scenes. An improved TSM video classification algorithm is used to determine pulling behavior in surveillance videos, reducing the dependence on abnormal behavior samples. At the same time, the idea of ​​using the frame difference method to detect moving targets is used. A channel-wise motion feature encoding module CME (Channel-wise Motion Encoding) is added to the original TSM network to extract motion features and excite motion channels, suppressing background channels, thereby obtaining more effective short-term motion features and improving the accuracy of pulling behavior recognition. In order to better obtain the abnormal behavior characteristics of people in continuous frames, this application introduces the interactive fusion update IAU (Interaction-Aggregation-Update) module, which can better capture the contextual dependency of the same target between different frames and improve the accuracy of abnormal behavior judgment.

[0038] Preferably, the step S5 first performs frame extraction processing on the video stream, specifically, according to each frame of image data obtained in step S4, one frame of image is saved every 10 frames of image data. When the number of saved image frames reaches 5 frames, the pulling behavior detection model obtained in step S3 is input for inference.

[0039] Preferably, the specific steps of step S6 are:

[0040] S61: Based on the inference result obtained by inputting the image frame extracted in step S5 into the pulling behavior detection model trained in step S3, determine whether abnormal pulling behavior occurs in the video stream. If so, go to step S61; if not, return to step S4.

[0041] S62: If abnormal pulling behavior occurs in the video, the video is saved for later inspection, and an alarm message is issued for abnormal pulling behavior. If no abnormal pulling behavior occurs, no alarm message is issued.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] (1) Using a video format dataset as a training dataset and the TSM video classification algorithm to detect pulling behavior; TSM is used to train the pulling behavior detection model. TSM uses a video format dataset as a training dataset, which reduces the difficulty of dataset labeling; it greatly saves manpower in labeling and improves the training efficiency of the model;

[0044] (2) The improved TSM network is used for model training. The CME module and IAU-Net module are added to the original TSM network to better obtain the relationship between the target's motion features and context, thereby improving the detection accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0046] FIG1 is a flow chart of a method for identifying pulling behavior based on an improved TSM video classification algorithm of the present application;

[0047] FIG2 is a diagram of a channel-level motion feature encoding (CME) module added in step S2 of the pulling behavior recognition method based on the improved TSM video classification algorithm of the present application;

[0048] FIG3 is a diagram of the interactive aggregation update IAU-Net module added in step S2 of the pulling behavior recognition method based on the improved TSM video classification algorithm of the present application;

[0049] FIG4 is a diagram of an improved TSM module in the method for identifying pulling behavior based on the improved TSM video classification algorithm of the present application;

[0050] FIG5 is a network structure diagram of a traditional TSM video classification algorithm in the pulling behavior recognition method based on the improved TSM video classification algorithm of the present application;

[0051] FIG6 is a comparison chart of the accuracy of the traditional TSM video classification algorithm and the improved TSM video classification algorithm in the pulling behavior recognition method based on the improved TSM video classification algorithm of the present application. DETAILED DESCRIPTION

[0052] The embodiments of the present application are described in detail below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present application and are not intended to limit the scope of protection of the present application.

[0053] The common knowledge in this technical solution specifically includes: TSM video classification algorithm.

[0054] Specifically: Video Action Classification Network: TSM (Temporal Shift Module for Efficient Video Understanding), which is a general and effective temporal shift module (TSM). It promotes information exchange between adjacent frames by shifting some channels along the temporal dimension. At the same time, it can be inserted into 2D convolutional neural networks to achieve zero-computation and zero-parameter temporal modeling, thereby combining the efficiency of 2D convolution with the high performance of 3D convolution.

[0055] TSM processes the temporal features of video frames to improve the efficiency of video understanding. TSM is widely used in many video analysis and recognition tasks, such as face recognition and motion recognition. The core idea of ​​TSM is to improve the accuracy of video analysis by segmenting video frames into different time segments and then analyzing the features of each time segment independently.

[0056] For abnormal behavior video recognition, the Temporal Transformation Module (TSM) deep learning model effectively describes the characteristics of event-domain information. This model inserts the TSM module into the ResNet50 network. This module improves network video comprehension capabilities through temporal shifting, shifting some channels in the temporal dimension, thereby facilitating information exchange between adjacent frames. Inserting this module into a 2D convolutional neural network enables modeling of timeline information with zero computational effort.

[0057] TSM video classification model detailed introduction:

[0058] Motivation and Principle: In video classification tasks, understanding temporal information is crucial for correct classification. However, standard convolutional neural network (CNN) architectures often fail to adequately capture temporal information when processing videos. TSM is motivated by introducing a shift operation in the temporal dimension of feature maps to enhance the model's ability to perceive temporal changes.

[0059] Shift Operation: The key component of TSM is the shift operation. By shifting input features across different channels, it effectively rearranges the input, allowing the model to observe the input at different time steps. This allows the model to perceive motion and changes in the video sequence from different temporal perspectives, thereby improving classification performance.

[0060] Network Architecture: TSM can be used in conjunction with different network architectures. It is usually embedded somewhere in these architectures to enhance the perception of the temporal dimension. TSM is usually introduced after certain convolutional layers in the network to ensure sufficient feature extraction before performing the shift operation.

[0061] Training strategy: During training, the TSM module introduces shift operations, but care should be taken to control the complexity of the model to prevent overfitting. Generally, TSM increases the perception capability of the model with a smaller computational cost and parameters.

[0062] Effects and Advantages: The introduction of TSM technology can significantly improve the performance of video classification tasks. By enhancing the model's understanding of time series, TSM can better capture the actions, changes, and key information in the video, thereby improving classification accuracy.

[0063] Example: As shown in FIG1 , the method for identifying pulling behavior based on the improved TSM video classification algorithm specifically includes the following steps:

[0064] S1: Generate a pull detection dataset, classify and label the content of the dataset, and then divide it into a training set and a validation set;

[0065] The pulling detection dataset in step S1 is in video format. A pulling dataset is generated by collecting a large number of videos of abnormal pulling behaviors and simulating videos of pulling and non-pulling behaviors in different scenarios. The videos are marked differently according to their content, with videos with pulling behaviors being marked as one category and videos without pulling behaviors being marked as another category. The video dataset with pulling behaviors is marked as pulling behaviors, and the dataset without pulling behaviors is marked as normal behaviors. The dataset is divided into a training set and a validation set.

[0066] S2: Improve the TSM network by adding a channel-level motion feature encoding module (CME module) after obtaining the feature map in the TSM network, and adding an interactive aggregation update module (IAU-Net module) after the second two-dimensional convolution module;

[0067] As shown in Figures 2 to 4, a channel-level motion feature encoding module is added in step S2 to extract motion features and excite motion channels. The specific steps of extracting motion features by the channel-level motion feature encoding module are as follows:

[0068] S21: First, the input video frame is subjected to a 1×1 2D spatial convolution to reduce the feature channel dimension. Assuming the scaling factor is r and the input dimension is [N, T, C, H, W], the convolution results in a feature map with an output dimension of [N, T, C / r, H, W].

[0069] S22: Split the obtained feature map into separate frames along the time dimension T, then perform differential analysis on adjacent frames and use the image feature X of the next frame t+1 Subtract the previous frame X tDue to the movement of the foreground target in the video, the target position changes between the two frames. Direct subtraction may cause feature confusion. Therefore, in order to eliminate feature confusion, the spatial feature transformation of the latter frame is first performed using a 3x3 2D convolution, and then the obtained transformation feature is subtracted from the image feature of the previous frame to obtain the motion representation between the two adjacent frames. Assume that the motion feature from the t-th frame to the t+1-th frame is M t , then the formula is: M t =Conv 3×3 (X t+1 )-X t , M t ∈R N×C / r×H×W ;

[0070] Among them, X t represents the image features of the t-th frame, X t+1 is the image feature of the t+1th frame, Conv 3×3 (·) represents a 3×3 2D spatial convolution; R represents a real number;

[0071] S23: Concatenate the obtained motion features between adjacent frames to obtain the motion features of the input video, denoted as M, that is: M=Concat(M1, M2, ..., M T ), M∈R N×T×C / r×H×W ;

[0072] Among them, Concat(·) represents the connection of motion features; M1,M2,·…·,M T Both refer to movement characteristics;

[0073] S24: Since the input feature X has undergone dimensionality reduction through the convolution channel, the dimension of the obtained video motion feature M is [N, T, C / r, H, W]. The obtained video motion feature M is input into two parallel branches, namely, the input motion coding branch and the motion channel excitation branch. Thus, motion features are obtained according to the motion coding branch and the motion channel excitation branch.

[0074] In the motion coding branch in step S24, a 1×1 2D convolution is designed to encode the motion feature M and restore the channel dimension from C / r to C to ensure that the feature information is not lost and is easy to merge with the input feature. Assume that the feature obtained after motion coding is M encode , then: M encode =Conv 1×1 (M), M encode ∈R N×T×C×H×W ;

[0075] Among them, Conv 1×1 (·) represents a 1×1 2D convolution operation; the motion coding branch obtains the motion features between adjacent frames after encoding;

[0076] Because the feature channel contains the motion features of the foreground target and the background features, the attention mechanism is used in step S24 to learn the attention weight of the channel of the motion feature obtained after splicing, to stimulate the motion feature channel and suppress the background feature channel; specifically, considering that the channel feature weight is independent of the spatial position, the motion feature is firstly subjected to global average pooling in space to obtain the space-independent motion feature, which is denoted as M pooling , that is, the formula is: M pooling =Pooling(M), M pooling ∈R N×T×C / r×1×1 ;

[0077] Where Pooling(·) represents the spatial pooling operation, and the obtained M pooling The dimensions are [N, T, C / r, W, H];

[0078] Then M pooling After convolution and dimensionality increase, and input into the sigmoid activation function, the attention weight matrix is ​​recorded as A, that is, the formula is: A=sigmoid(Conv 1×1 (M pooling )), A∈R N×T×C×1×1 ;

[0079] Where sigmoid(·) represents the sigmoid activation function, Conv 1×1 (·) represents a 1×1 2D convolution operation, which combines the obtained channel attention weight A with the motion coding feature M encode Multiply them together to stimulate the motion feature channel, and add residual connections to keep the original input features from being lost, that is, the formula is: Y = X + A·M encosde , Y∈R N×T×C×H×W ;

[0080] Where X is the input feature and Y is the final output feature of the module; X and Y have the same dimension, that is, the channel-level motion feature encoding module (CME module) does not change the output dimension of the input feature;

[0081] The Interaction-Aggregation-Update (IAU-Net) module, added after the second two-dimensional convolutional module in step S2, comprises an interaction module, an aggregation module, and an update module. The interaction module is used to pull the behavior detection model to process interactions between nodes and their neighbors in the image data; the aggregation module is used to pull the behavior detection model to aggregate information propagated from neighboring nodes to the central node; and the update module is used to update the features or representation of the central node after calculating the aggregated information. The Interaction-Aggregation-Update (IAU) module can better capture the contextual dependencies of the same object across different frames, improving the accuracy of abnormal behavior detection. The Interaction-Aggregation-Update (IAU) module is a concept used to describe the general process or pipeline in graph neural networks (GNNs). This module is particularly suitable for modeling relationships and dependencies between entities in graph-structured data. Graph neural networks have become popular for tasks such as node classification, link prediction, and graph classification. The "interaction-aggregation-update" process enables these networks to effectively extract information from the structure of the graph and perform predictions or classification based on this information. Interaction: During the interaction phase, the model processes the interactions between nodes in the graph and their neighbors. Nodes in the graph can represent entities, objects, users, etc., while edges represent relationships or connections between these entities. In this phase, information flows from neighboring nodes to the central node through the edges of the graph; Aggregation: After the interaction phase, the model aggregates the information propagated from neighboring nodes to the central node. Aggregation can involve various mathematical operations such as summation, averaging, or more complex aggregation functions. The goal is to combine information from different neighbors in a meaningful way to capture relevant contextual information about the central node; Update: Once the aggregated information is calculated, it is used to update the features or representation of the central node. This update step typically involves some form of transformation or neural network operation to integrate the aggregated information with the existing features of the node. The updated features become the new representation of the node, fusing its original features and contextual information from its neighbors;

[0082] S3: Input the pulling detection dataset generated in step S1 into the improved TSM algorithm in step S2 to obtain the algorithm model weights trained by the improved TSM and obtain a pulling behavior detection model;

[0083] S4: Image data acquisition, obtaining each frame of image data through video or RTSP stream;

[0084] S5: Inputting each frame of image data obtained in step S4 into the pulling behavior detection model trained in step S3 for inference to obtain an inference result;

[0085] In step S5, the video stream is first subjected to frame extraction processing. Specifically, based on each frame of image data obtained in step S4, one frame of image data is saved every 10 frames of image data. When the number of saved image frames reaches 5, the pulling behavior detection model obtained in step S3 is input for inference.

[0086] S6: Analyze the inference result in step S5 to determine whether pulling behavior occurs. If so, save the pulling behavior and determine whether an alarm needs to be issued.

[0087] The specific steps of step S6 are:

[0088] S61: Based on the inference result obtained by inputting the image frame extracted in step S5 into the pulling behavior detection model trained in step S3, determine whether abnormal pulling behavior occurs in the video stream. If so, go to step S61; if not, return to step S4.

[0089] S62: If abnormal pulling behavior occurs in the video, the video is saved for later inspection, and an alarm message is issued for abnormal pulling behavior. If no abnormal pulling behavior occurs, no alarm message is issued.

[0090] As shown in Figure 5, traditional TSM video classification algorithms generally use a ResNet50 model as their base network structure. These algorithms improve image classification accuracy by embedding the TSM module after the convolutional layers of the ResNet50 network. While this method achieves satisfactory classification results in some scenarios, it cannot accurately capture the motion characteristics of the target and the relationship between the context in the video. This results in an inability to accurately determine whether abnormal behavior occurs in complex scenes, and may even result in false detections and missed detections. Therefore, this algorithm has low generalization ability and is not suitable for complex scenarios. To address the low applicability of traditional TSM video classification algorithms in complex scenarios, this application proposes an improved TSM video classification algorithm. To better capture the relationship between the target's motion characteristics and context, thereby improving the model's accuracy in abnormal behavior identification, this application proposes a CME module and an IAU-Net module. In the original TSM network, we add the CME module after the feature map, enabling the model to better capture the target's motion characteristics. Furthermore, to better capture the relationship between the target's context, we place the IAU-Net module after the second convolutional layer of the TSM network. The improved TSM network structure is shown in Figure 4.

[0091] Compared with the traditional TSM video classification algorithm, the improved TSM video classification algorithm can be better applied to complex scenes. The accuracy of pulling behavior recognition in complex scenes is higher than that of the traditional TSM video classification algorithm. In addition, the recognition effect of pulling behavior in multi-person scenes and scenes with occlusion is better than that of the traditional TSM video classification algorithm. The accuracy comparison of the traditional TSM video classification algorithm and the improved TSM video classification algorithm is shown in Figure 6. Using 10 pulling behavior videos in complex scenes as a test, it can be seen from Figure 6 that the improved TSM video classification algorithm in this application can accurately identify pulling behaviors in all complex scenes without misidentification or missed identification, while the traditional TSM video classification algorithm has misidentification and missed identification in pulling behavior recognition in complex scenes, and has a low accuracy rate for pulling behavior detection. The improved TSM video classification algorithm can better obtain the relationship between the target's motion characteristics and its context, making the algorithm better applicable to the recognition of pulling behavior in complex scenes and improving the accuracy of the algorithm in pulling behavior detection.

[0092] For ordinary technicians in this field, the specific embodiments are only illustrative descriptions of the present application. It is obvious that the specific implementation of the present application is not limited to the above-mentioned methods. As long as various non-substantial improvements are made using the method concepts and technical solutions of the present application, or the concepts and technical solutions of the present application are directly applied to other occasions without improvement, they are all within the scope of protection of the present application.

Claims

1. A method for identifying pulling behavior based on an improved TSM video classification algorithm, characterized in that: The specific steps include: S1: Generate a pull detection dataset, classify and mark the content of the dataset, and then divide it into a training set and a validation set; S2: Improve the TSM network by adding a channel-level motion feature encoding module after obtaining the feature map in the TSM network, and adding an interactive aggregation update module after the second two-dimensional convolution module; S3: input the pulling detection data set generated in step S1 into the improved TSM algorithm in step S2, obtain the algorithm model weight of the improved TSM training, and obtain the pulling behavior detection model; S4: image data acquisition, obtaining image data of each frame; S5: inputting each frame of image data obtained in step S4 into the pulling behavior detection model trained in step S3 for inference to obtain an inference result; S6: Analyze the inference result in step S5 to determine whether pulling behavior occurs. If so, save the pulling behavior and determine whether an alarm is required.

2. The method for identifying pulling behavior based on the improved TSM video classification algorithm according to claim 1 is characterized in that: The pulling detection data set in step S1 is in video format. The pulling data set is generated by simulating the videos of pulling and non-pulling behaviors in different scenes, and is marked differently according to the content of the video. The videos with pulling behaviors are marked as one category, and the videos without pulling behaviors are marked as another category.

3. The method for identifying pulling behavior based on the improved TSM video classification algorithm according to claim 1, characterized in that: In step S2, a channel-level motion feature encoding module is added to extract motion features and excite motion channels. The specific steps of extracting motion features by the channel-level motion feature encoding module are as follows: S21: First, the input video frame is subjected to a 1×1 2D spatial convolution to reduce the feature channel dimension. Assuming the scaling factor is r and the input dimension is [N, T, C, H, W], the feature map with an output dimension of [N, T, C / r, H, W] is obtained after the convolution. S22: Split the obtained feature map into separate frames along the time dimension T, and then perform differential analysis on adjacent frames, i.e., use the image feature X of the next frame t+1 Subtract the previous frame X t , and then use 3x3 2D convolution to transform the spatial features of the next frame, and then use the transformed features to do the previous frame image features Difference, thus obtaining the motion representation between two adjacent frames. Assuming that the motion feature from the tth frame to the t+1th frame is M t , then the formula is: M t =Conv 3×3 (X t+1 )-X t ,M t ∈R N×C / r×H×W ; Among them, X t represents the image features of the tth frame, X t+1 is the image feature of the t+1th frame, Conv 3×3 (·) represents a 3×3 2D spatial convolution; R represents a real number; S23: The obtained motion features between adjacent frames are concatenated to obtain the motion features of the input video, which is represented as M, that is: M=Concat(M1,M2...,M T ),M∈R N×T×C / r×H×W ; Among them, Concat(·) represents the connection of motion features; M1,M2,···,M T Both refer to movement characteristics; S24: Since the input feature X has undergone dimensionality reduction by the convolution channel, the dimension of the obtained video motion feature M is [N, T, C / r, H, W]. The obtained video motion feature M is input into two parallel branches, namely, the input motion coding branch and the motion channel excitation branch. Thus, the motion feature is obtained according to the motion coding branch and the motion channel excitation branch.

4. The method for identifying pulling behavior based on the improved TSM video classification algorithm according to claim 3 is characterized in that: In the step S24, in the motion coding branch, a 1×1 2D convolution is designed to encode the motion feature M, and the channel dimension is restored from C / r to C, so as to ensure that the feature information is not lost and facilitate fusion with the input feature; Assume that the feature obtained after motion coding is M encode , then: M encode =Conv 1×1 (M),M encode ∈R N×T×C×H×W ; Among them, Conv 1×1 (·) represents a 1×1 2D convolution operation; the motion coding branch obtains the motion features between adjacent frames after encoding.

5. The method for identifying pulling behavior based on the improved TSM video classification algorithm according to claim 4 is characterized in that: In step S24, an attention mechanism is used to learn the attention weight of the channel of the motion feature obtained after splicing, so as to stimulate the motion feature channel and suppress the background feature channel; Specifically, the motion features are first pooled globally in space to obtain the space-independent motion features, denoted as M pooling , that is, the formula is: M pooling =Pooling(M),M pooling ∈R N×T×C / r×1×1 ; Where Pooling(·) represents the spatial pooling operation, and the obtained M pooling The dimensions are [N, T, C / r, W, H]; Then M pooling After convolution and dimensionality increase, and input into the sigmoid activation function, the attention weight matrix is ​​recorded as A, that is, the formula is: A=sigmoid(Conv 1×1 (M poolingg )),A∈R N×T×C×1×1 ; Where sigmoid(·) represents the sigmoid activation function, Conv 1×1 (·) represents a 1×1 2D convolution operation, which combines the obtained channel attention weight A with the motion coding feature M encode Multiply them to stimulate the motion feature channel, and add residual connection to keep the original input features from being lost, that is, the formula is: Y=X+A·M encosde ,Y∈R N×T×C×H×W ; Among them, X is the input feature, Y is the final output feature of the module; and X and Y have the same dimension, that is, the channel-level motion feature encoding module does not change the output dimension of the input feature.

6. The method for identifying pulling behavior based on the improved TSM video classification algorithm according to claim 4 is characterized in that: In the step S2, an interactive aggregation update module is added after the second two-dimensional convolution module, including an interactive module, an aggregation module and an update module, wherein the interactive module is used for the pulling behavior detection model to process the interaction between a node and its adjacent nodes in the image data; the aggregation module is used for the pulling behavior detection model to aggregate the information propagated from the adjacent nodes to the central node; The updating module is used to update the features or representation of the central node after calculating the aggregation information.

7. The method for identifying pulling behavior based on the improved TSM video classification algorithm according to claim 4 is characterized in that: The step S5 first performs frame extraction processing on the video stream, specifically, according to each frame of image data obtained in step S4, one frame of image is saved every 10 frames of image data. When the number of saved image frames reaches 5 frames, the pulling behavior detection model obtained in step S3 is input for inference.

8. The method for identifying pulling behavior based on the improved TSM video classification algorithm according to claim 7 is characterized in that: The specific steps of step S6 are: S61: according to the image frame extracted in step S5, input the inference result obtained in the pulling behavior detection model trained in step S3, determine whether there is abnormal pulling behavior in the video stream, and if so, go to step S61; If not, return to step S4; S62: If an abnormal pulling behavior occurs in the video, the video is saved and an alarm message is issued for the abnormal pulling behavior.

Images